KDC Authentication problems with 2003 to 2008 domain functional level

Recently I have had problems connecting to the console on a number of 2008 R2 Hyper-v guest virtual machines.  The error was “An Authentication Error Has Occurred.  The Encryption Type Requested Is not supported by the KDC” while I have also had a single Exchange 2010 server fail with the following event IDs: 2102, 2103, 2114, 9106 all reporting LDAP problems, non-responding domain controllers and global catalogs:

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1696). Topology discovery failed, error 0×80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message)). Look up the Lightweight Directory Access Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, “Microsoft LDAP Error Codes.” Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.

Process STORE.EXE (PID=4084). All Global Catalog Servers in forest DC=xxx,DC=xx,DC=xx are not responding:

Process STORE.EXE (PID=4084). All Domain Controller Servers in use are not responding:

Attempting to open the Exchange management console on the local server console ended with a  HTTP server error status 500 and “Kerberos” authentication failed.

The Exchange server was able to ping and resolve all DNS names correctly and the problem went away on restarting only to re-occur in 24 hours or so.

The rather simple resolution in the end turned out to be restarting the “KERBEROS DISTRIBUTION KEY (KDC) service” on all Domain controllers.  While Restarting all Domain controllers in their entirety is also a good idea it isn’t always possible (or desirable) on a live production environment.

Installing Update Rollup for Exchange 2010

When attempting to install an update on an Exchange 2010 server (or anything with the management tools on) it is possible to get the error below or one similar to it.

Windows Installer reconfigured the product. Product Name: Microsoft Exchange Server. Product Version: 14.1.218.15. Product Language: 1033. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 1603.

For me the resolution was to start a command window with elevated permissions (right click, run cmd as administrator) and then run the update from there.

E.G.

Then simply follow the on screen wizard as before and the update should install correctly.

Unable to change private key size when generating custom certificate request on windows

It is becoming the norm to use larger private key sizes with certificates and while trying to generate a new request on a windows 2003 box I found my self unable to change the key size at all, it was greyed out.  After a bit of head scratching I noticed all the cryptographic service providers were ticked.

After changing the tick boxes so that only the “RSA,Microsoft Software Key Storage Provider” is ticked the option becomes available and the key size can be customized.

If (like me) you are generating some CSRs to be used on a forefront threat management gateway or similar don’t forget to make the private key exportable so later on you can export it to the TMG server.

Update: This is still true for modern server versions such as 2012 R2 and 2016.

Lync Client on Linux

I have been wondering lately if it is possible to use Lync from a linux workstation.  As far as I am aware there is no official support or even a chance that the client will run outside of a virtual windows box or similar. (if somone has managed to get the Lync client running under wine or similar i’d love to hear about it) however it seems it is possible to at least have basic chat functionality using pidgin.

For my ubuntu laptop it was simply a case of running:

sudo apt-get install pidgin
sudo apt-get install pidgin-SIPE

Then running pidgin and adding my account details in as a “Office Communicator” account type.

For example:

Protocol: Office Communicator
Username: sipaddress
login: domain\user
Password: password

If all your services are published in DNS you wont need to mess about in the advanced tab but be warned you may get a certificate warning if you used self-signed certs.

 

System Centre Operations Manager Release Candidate Setup

Following on from the unified installer for the Microsoft private cloud, the System centre operations manager installation failed and I decided to attempt the installation myself.

Once you have a suitable server setup with windows 2008 R2 the first step is to install the .NET Framework 4, report viewer and all the required IIS roles:

IIS6 Metabase Compatibility role service.
ASP.NET role service.
Windows Authentication role service.
Static Content role service.
Default Document role service.
Directory Browsing role service.
HTTP Errors role service.
HTTP Logging role service.
Request Monitor role service.
Request Filtering role service.
Static Content Compression role service
IIS Management Console role service.

Even with all the correct roles and pre-requisit software installed the validator will still throw up a couple of problems.

 

You will need to load the IIS Manager and open the ISAPI and CGI Restrictions properties and change the deny to an allow for the ASP.NET v4.0.30319 line.

 

In addition to the above if you installed the .NET 4 framework before the roles you will get an error message: “The ISAPI and CGI Restrictions are disabled or missing” on validating the pre-requisites and will need to run the following command in a cmd window to resolve the problem.

%WINDIR%\Microsoft.NET\Framework64\v4.0.30319\aspnet_regiis.exe –r

If all is well the command window should look like this:

 

Now setup should allow you to proceed with setup and select weather to add a management server to an existing management group or create the first management server in a new management group.  I like most people playing with the release candidate chose the latter.

Next you are prompted to supply the SQL server details.  In my environment the failed unified installer attempt had left a default SQL instance installed which appeared to be unsuitable as the installer complained I was missing the “Full text indexing” features required.  I had used the SQL 2008 R2 express media which does not include this option (and according to tech-net not on the list of supported SQL versions) so my next steps were to remove the currently installed SQL instance and install a full fat version of SQL 2008 R2 Standard with the full text indexing and reporting services options.  In a production setup you may well choose to create a new database on an existing server but for the purposes of evaluating this I selected to have a local instance of SQL installed as it makes cleaning up this install later a lot easier as I can just destroy the VM.

With the advanced services options installed the operations manager setup wizard will allow you to continue and adjust the configuration of the operational database (although I just left these at the defaults)

 

On clicking next you should see a similar screen for the datawarehouse database.  Again I accepted the defaults and continued on to choose the reporting services instance I installed earlier with SQL 2008 R2 Standard (don’t forget to run the Reporting Services configuration manager and start the SQL Server agent service)

 

Next it is time to choose which IIS site to use for the web console.  In a production environment it would be prudent to configure a new site and setup SSL however I am sticking with the default website for now.

 

Next you are prompted to select an authentication mode for use with the web console.  I selected Mixed Authentication as it will be an entirely private deployment for evaluation/testing purposes only.

After selecting the authentication mode you are prompted to supply some domain account/s for the various roles to use.  I created a single user for this purpose however it would be advisable to separate the data and management accounts so you can fine tune the permissions they are granted.

 

After this step you can choose to opt-in (or not) to the various customer experience improvement programs.  I chose not to as this machine has no Internet access anyway.

Finally you are presented with a page full of the various configuration options set in the previous steps.  If all looks well click install, go grab a tea/coffee and you should be able to return to an installed SCOM 2012.

 

Next I will work on getting a few servers monitored and start evaulating what is avaliable in SCOM 2012.

Initialization Failed: The Operation Couldn’t Be Completed Because a Change Occurred in the Remote Forest

When attempting to start the exchange 2010 management console the error “Initialization Failed: The Operation Couldn’t Be Completed Because a Change Occurred in the Remote Forest” is displayed. This error doesn’t appear to be related to the cause of the problem which can make it especially confusing when trying to resolve.  The problem is usually caused by the server/s the management console is connecting to being a higher update roll-up or service pack than the management console itself. Simply make sure the management tools are patched to the same version as the server and you should then be able to connect successfully.

Configuring normalization rules for the Lync 2010 address book

Lync by default automatically pulls telephone information from active directory and publishes it to clients each morning at 1.30AM however often the telephone numbers entered in active directory have various problems, inconsistencies or are otherwise not formatted correctly for Lync to use.  If this is the case you may notice contact cards in the Lync client contain no phone numbers (or have some numbers missing) while if you examine the user in outlook or directly in active directory you can clearly see telephone numbers are configured for them.  Lync will only function with E.164 style numbers and as a result unless all of your numbers are stored correctly in this format, many will be missing.

A full list of numbers which fail to normalize can be found in the Lync Server event log of your front end server.  (normally logged shortly after 1:30AM) with event ID 21034.

 

As you can see there is a handy link to a file which contains a full list of the failed numbers.  The ideal way to fix this would be to edit all the numbers so they are stored in the right format.  For example changing a DDI and extension from 01372 112233 and 2233 to be stored as +441122112233;ext=2233 often this isn’t possible or desirable if other applications use these numbers and instead it maybe preferable (and quicker) to follow what the text in the event log suggests and setup normalization rules in the optional “Company_Phone_Number_Normalization_Rules.txt” file.

This was the case for me recently and I ended up creating the following rule to normalize standard UK national numbers into international E.164 style numbers:

## match National to UK E.164
\+?0([1-9]\d{7,9})
+44$1

(where +44 is your country code)

In an ideal world all your extension numbers will match your DDI numbers as well so you can normalize them with a rule similar to this:

## 9xxx extn into ddi
\+?(9\d\d\d)
+44112233$1;ext=$1

(where +44112233 is the appropriate area and regional code)

For my environment these two rules cut the number of failed numbers to 87.  If you cant wait until 1:30AM the next morning to see the results of your changes run ” Update-CSAddressBook” in the Lync shell window and wait approximately 5 minutes for the process to complete and the messages to appear in the event log.

In addition to the above if you wish to clear the cashed address book in a Lync client first close the client and navigate to %profilepath%\AppData\Local\Microsoft\Communicator and delete the appropriate SIP folder and then restart the client.  This will force it to re-download the address book and you can instantly enjoy the results of your changes.

A great reference on how to create these rules can be found here.

Evaluating the Microsoft Private Cloud with the System Centre 2012 Unified Installer

After reading a lot about Hyper-V and attending an IT Camp at Microsoft I really wanted to give the new SCCM 2012 a closer look and at the same time get better acquainted with Hyper-V and other related upcoming Microsoft releases such as Data protection Manager and Service Manager.

Getting all the installation files and pre-requisite software downloaded

Sign up and download all the installation files here: http://technet.microsoft.com/en-us/evalcenter/hh505660 (6.6GB) and in addition to this you will need all of the pre-requisite software which is listed here: http://technet.microsoft.com/en-us/library/hh751268.aspx  I went through a fair bit of trial and error to get the set-up utility to detect all of the installation files and I recommend making sure to extract each of the products into their own folders.  Don’t put all of the products in one folder or share folders as if the installer doesn’t recognize one of the paths/files you wont be able to tell which one (plus its messy and you might end up over writing files) I would also suggest you do the same for all of the pre-requisite software. (also don’t forget to extract all of the zip/exe/iso files as the unified installer wont read them otherwise.  I found winrar invaluable for this)

 

Preparing the installation environment and servers

To get all the products installed you need at least 8 servers (physical or virtual) with a minimum of 2GB of ram each.  I set all mine up on a single Hyper-V host as its only for testing purposes and I don’t have loads of servers spare for development/testing work.  I would also suggest using a sensible naming convention or it can get pretty confusing quite quickly and bare in mind that the server you choose to initially run the set-up on will become the Orchestrator server.

Aside from the base windows 2008 R2 operating system all the machines need to have a few things configured before they are ready for deployment.  To avoid duplication I made sure all of the servers were in a single OU and created a policy to apply the customizations for me rather than individually configuring the local policy on each host.

Computer Config \ Administrative Templates \ System \ Credentials Delegation \ Allow Delegating Fresh Credentials
Set to = Enabled
Server = WSMAN/*

Computer Config \ Administrative Templates \ System \ Credentials Delegation \ Allow Delegating Fresh Credentials with NTLM only server Authentication
Set to = Enabled
Server = WSMAN/*

Computer Config \ Administrative Templates \ Windows Components \ Windows Remote Management (WinRM) \ WinRM Client \ Allow CredSSP authentication
Set to = Enabled

Computer Config \ Administrative Templates \ Windows Components \ Windows Remote Management (WinRM) \ WinRM Client \ Trusted Hosts
Set to = Enabled
TrustedHostList = *

Computer Config \ Administrative Templates \ Windows Components \ Windows Remote Management (WinRM) \ WinRM Service \ Allow Automatic Configuration of listeners
Set to = Enabled
IPv4 filter = *
IPv6 filter = *

Computer Config \ Administrative Templates \ Windows Components \ Windows Remote Management (WinRM) \ WinRM Service \ Allow CredSSP authentication
Set to = Enabled 

Computer config \Administrative Templates \ Network \ Network Connection \ Windows firewall \ Standard Profile \ Windows firewall Protect all network connection
Set to =  disabled

I then went round each server and ran a gpupdate to ensure they all applied the new policy before I attempted to run the unified set-up.

 

Running the unified installer

On running the unified set-up and selecting the products you are evaluating (I wanted to try all of them) you are prompted to provide a path to each of the installation files you downloaded earlier.  (I bet you are glad you downloaded and extracted each of the products/pre-requisites into their own folders now) if all is well it should be a simple exercise of browsing and selecting each of the folder paths created earlier.  Dont be surprised if it doesnt recognize one of the paths or files just make sure you have the right product/version and its extracted, even the iso file for the windows automated installation kit needs to be extracted so its just a normal folder full of files.  (UNC or local path names are both OK.)

 

Once you have completed both this screen and the pre-requisites page that follows it you can select what account you wish to use as the installer account.  I created my own domain user for this but you can use any user account which has the required permissions.  Following this you can configure other options such as site name etc. and finally you are presented with an install button.  Sit back and watch the progress bars.

 

Access denied when managing group membership

After moving to Exchange 2010 from 2003 I experienced an issue design feature where a user who is configured as a group manager and allowed to update the membership list is unable to in outlook and instead receives an access denied message.

A bit of research revealed that this is the expected behaviour and is down to changes in how users can manage their own accounts and information. There is a really good explanation and work around on the exchange team blog here:

http://blogs.technet.com/b/exchange/archive/2009/11/18/3408844.aspx

Cannot login to Lync 2010 Control Panel – Unauthorized: Authorization Failed

After restarting our Lync front end servers it became impossible to log in to the control panel.

 

In our environment nothing was recorded in the application or system event logs and there were no audit failures recorded in the security log either. The only place anything showed up was in the “Lync Server” logs where a number of “LS Remote Powershell” warnings and errors with event IDs 35005, 35007 were being recorded on each failed login attempt. (see below for event details)

The resolution for me was the restart the SQL instances hosting RTC and RTCLOCAL. It would appear any problem preventing access to the SQL server or active directory can produce these symptoms and errors so check your network connections and user/computer accounts if SQL looks OK.

 

Log Name:      Lync Server Source:        LS Remote PowerShell Date:          04/01/2012 09:33:22 Event ID:      35005 Task Category: (3500) Level:         Error Keywords:      Classic User:          N/A Computer:      ***** Description: Remote PowerShell cannot read the RBAC Roles information from the store.

Remote PowerShell encountered problem when trying to read the RBAC Roles information for the user. Cause of failure: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 – Error Locating Server/Instance Specified) Cause: The failure may have happened due to some permissions issue in reading the management store. Resolution: Make sure that the server is domain joined machine and able to query the active directory.

Log Name:      Lync Server Source:        LS Remote PowerShell Date:          04/01/2012 09:40:05 Event ID:      35007 Task Category: (3500) Level:         Warning Keywords:      Classic User:          N/A Computer:      ***** Description: Remote PowerShell cannot create InitialSessionState.

Remote PowerShell cannot create InitialSessionState for user: S-1-5-21-*********-1589796742-927750060-43130. Cause of failure: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 – Error Locating Server/Instance Specified) Cause: Remote PowerShell can fail to create InitialSessionState for varied number of reasons. Please look for other events that can give some specific information. Resolution: Follow the resolution on the corresponding failure events.