WinRM Connection limits

Sometimes in various Microsoft products (Exchange/VMM mostly) you might reach some of the WinRM connection limits.  Personally I see it most in Virtual Machine Manager when you have many admins who are making lots of changes and deploying large numbers of services.

The symptom is sometimes an error like this:  Error Connecting to remote server failed with the following error message: The WS-Management service cannot process the request. This user is allowed a maximum number of 5 concurrent shells, which has been exceeded. Close existing shells or raise the quota for this user.

It can also just show as a generic Failed/timeout job which then works when you re-try later.

1. On the offending server open a command prompt or Powershell window with administrative privileges.
2. Type in winrm get winrm/config/winrs to view the current configuration.

winrmmax

These values will need to be increased, don’t just add a load of zeros to the end as having limits configured can stop unwanted or malicious connections from brining a server to its knees.

To adjust the values use the commands below where 20 and 100 are appropriate numbers for your environment.

3. winrm set winrm/config/winrs @{MaxConcurrentUsers=”20″}
4. winrm set winrm/config/winrs @{MaxShellsPerUser=”100}

Message tracking in Exchange 2010 Error: “String was not recognized as a valid DateTime.”

When attempting to run message tracking on an Exchange 2010 server you get an error similar to this: Cannot process argument transformation on parameter ‘Start’. Cannot convert value to type “System.DateTime”. Error: “String was not recognized as a valid DateTime.”

Message Tracking Error

Can you spot the difference?  Even if the server has a non US regional setting you have to specify the date in US format. e.g. 08/22/2013 instead of 22/08/2013.

KDC Authentication problems with 2003 to 2008 domain functional level

Recently I have had problems connecting to the console on a number of 2008 R2 Hyper-v guest virtual machines.  The error was “An Authentication Error Has Occurred.  The Encryption Type Requested Is not supported by the KDC” while I have also had a single Exchange 2010 server fail with the following event IDs: 2102, 2103, 2114, 9106 all reporting LDAP problems, non-responding domain controllers and global catalogs:

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1696). Topology discovery failed, error 0×80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message)). Look up the Lightweight Directory Access Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, “Microsoft LDAP Error Codes.” Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.

Process STORE.EXE (PID=4084). All Global Catalog Servers in forest DC=xxx,DC=xx,DC=xx are not responding:

Process STORE.EXE (PID=4084). All Domain Controller Servers in use are not responding:

Attempting to open the Exchange management console on the local server console ended with a  HTTP server error status 500 and “Kerberos” authentication failed.

The Exchange server was able to ping and resolve all DNS names correctly and the problem went away on restarting only to re-occur in 24 hours or so.

The rather simple resolution in the end turned out to be restarting the “KERBEROS DISTRIBUTION KEY (KDC) service” on all Domain controllers.  While Restarting all Domain controllers in their entirety is also a good idea it isn’t always possible (or desirable) on a live production environment.

Installing Update Rollup for Exchange 2010

When attempting to install an update on an Exchange 2010 server (or anything with the management tools on) it is possible to get the error below or one similar to it.

Windows Installer reconfigured the product. Product Name: Microsoft Exchange Server. Product Version: 14.1.218.15. Product Language: 1033. Manufacturer: Microsoft Corporation. Reconfiguration success or error status: 1603.

For me the resolution was to start a command window with elevated permissions (right click, run cmd as administrator) and then run the update from there.

E.G.

Then simply follow the on screen wizard as before and the update should install correctly.

Initialization Failed: The Operation Couldn’t Be Completed Because a Change Occurred in the Remote Forest

When attempting to start the exchange 2010 management console the error “Initialization Failed: The Operation Couldn’t Be Completed Because a Change Occurred in the Remote Forest” is displayed. This error doesn’t appear to be related to the cause of the problem which can make it especially confusing when trying to resolve.  The problem is usually caused by the server/s the management console is connecting to being a higher update roll-up or service pack than the management console itself. Simply make sure the management tools are patched to the same version as the server and you should then be able to connect successfully.

Access denied when managing group membership

After moving to Exchange 2010 from 2003 I experienced an issue design feature where a user who is configured as a group manager and allowed to update the membership list is unable to in outlook and instead receives an access denied message.

A bit of research revealed that this is the expected behaviour and is down to changes in how users can manage their own accounts and information. There is a really good explanation and work around on the exchange team blog here:

http://blogs.technet.com/b/exchange/archive/2009/11/18/3408844.aspx

Exchange 2010 Certificate Wizard request wont complete with Certificate Services

Exchange 2010 includes a certificate wizard to help you generate a certificate request without having to manually type each of the fields and make your own requests in the certificates MMC. Unfortunately you then cannot complete the request using the Microsoft certification authority as you get an error message. Normally this error is along the likes of “Bad TAG” or “Certificate not issued (Incomplete)”

Credits to Laurance at Dell for the solution to this particular error which turned out to be that the CSR generated by Exchange 2010 is in Unicode format and certificate services is expecting it to be in ANSI. Simply open the request file in notepad and save it in ANSI file format. Make sure to surround the entire file name in quotes to preserve the file extension and change the file name from the original.

 

Now you will get a new error because exchange hasn’t included a certificate template along with the request and certificate services needs to know which template to use. Open a CMD window (as admin or you will get another error) and type in the following: certreq -attrib “CertificateTemplate:WebServer” you should then be prompted to browse for your request file and if all is well for a place to save the completed certificate. Then you can use this certificate file to complete the pending request with the exchange 2010 certificate wizard.

Exchange 2010 Messages stuck in queue to remote AD site

I have recently been working on implementing a number of new exchange 2010 servers and had an issue where some mail was getting stuck in a queue. For example messages from mailboxes on Server B to mailboxes on Server A end up in one of Server B’s queues called “SMTP Relay to Remote Active Directory Site.” and the last error is 451 4.4.0 Primary Target IP address responded with: “421 4.4.2 Connection dropped.

Now I knew it wasn’t a firewall or network issue as I could telnet into port 25 from each of the servers to each other and sending email in this way also worked. Curiously I couldn’t see anything in the event logs either. The Mail Flow Troubleshooter gives the message ”It appears that the SMTP service and SMTP instance(s) on server A are started but the port did not respond. Check if there are any network errors or hung services.” Also gives “Error submitting mail.” The issue remains after a service and server restart.

The resolution for me was to re-assign the original self signed certificate to the SMTP service instead of the 3rd party CA signed certificate we were using for OWA. (even though it had a valid SAN for the internal FQDN) and then restart the transport service on server A. Once the SMTP service was using the self signed cert all the queued mail was delivered. I hope this helps someone else out there struggling with this issue.