vCenter and PSC and the 4096 bits

So, I’ve been meaning to do some tests around VMware VCSA and PSC high availability and a major prerequisites to achieve this is to change the SSL certificates on all participating appliances.

Going back to the documentation in the before you begin section, one of prerequisite items that is mentioned (Key size: 2048 bits or more (PEM encoded)) and going back to my lab configuration my RSA public key RSA is 4096 bits so I suppose we’re good right.

I issued the certificates and combined the chain into one file and actually imported the certificates successfully without any issues, now when I tried ti connect to the vCenter server web-client I received this error:

[500] SSO error: Could not parse certificate: java.io.IOException: java.lang.IllegalArgumentException: Input byte array has incorrect ending byte at 2380

This was the first time I saw this error and I thought I most probably did something wrong with the certificate request or import process, so I reverted everything back to a snapshot (pre-SSL) and after going through the process once more I also ended up with the same error.

After some research I found a comment by Féidhlim O’Leary on his blog and I quote:

Based on “[500] SSO error: Could not parse certificate: java.io.IOException: java.lang.IllegalArgumentException: Input byte array has incorrect ending byte at 2628” one guess is that your are placing the VMCA with a certificate greater than 2048 bit key length? I’ve heard reports that 4096 and greater are causing issues.

Now the setup which I am currently playing with is at 6.5e so I even tried using different updateLSEndpoint.py from different releases and still the same issue, so I decided to regenerate my CA root certificate and make it 2048.

I have a Windows Server based CA, so long story short here how is down (quite simple and effective by the way):

1. Create a CAPolicy.inf in C:\Windows
2. In the CAPolicy.inf put:
   1. [Certsrv_Server]
   2. RenewalKeyLength=2048
   3. RenewalValidityPeriod=Years
   4. RenewalValidityPeriodUnits=10
3. Renew the CA root certificate and accept regenerating the key and you’re done.

Here I re-did the SSL process from scratch and imported the SSL certificates successfully and when I tried to access the vCenter server web-client it was successfully accessible without any issues.

Hopefully either this gets fixed soon, or the KBs gets edited :-).

(Abdullah)^2