When vSphere PSC HA looks at you with a HAHA

When it comes to tasks that you’ve done a plethora of times in the lab and at customer sites, always expect to be surprised! I can’t tell you how many times I have done so many things that have the same results but in totally different approaches because simply not all approches work.

In a recent engagement, I had this little teeny-weeny task left to be done which is to configure vSphere’s Platform Services Controllers in HA mode and load balance them via an NSX ESG, I did my due diligence, made sure that everything is prepared including the commands, scheduled the time of the maintenance period and it would take around 1 hour to finish everything with proper functionality and redundancy checks.

Usually when you run the UpdateLsEndpoint.py it throws a significant amount of output on the screen which is hard to track and to be honest I never did look into it because everytime I ran it (until now) it worked fine without any issues.

This time I ran UpdateLsEndpoint.py –lb-fqdn=VIPFQDN –user=administrator@vsphere.local –password=ComplexPasswordHere and it ran fine as I presumed and I didn’t look into the logs, and then when I went to validate my configuration I couldn’t failover to the second PSC and although I changed the vCenter Server Appliance to point to the VIP FQDN, in the advanced configuration it still pointed to the first PSC.

After several attempts and double checking on everything, I wasn’t able to achieve a successful configuration, I opened a case with GSS to double check and after thoroughly going through the logs we couldn’t find anything that would point us to the root cause of the issue.

I was advised to re-apply the PSC configuration, the only difference this time is that I was keen on the output of all issued commands, and when I attempted to apply UpdateLsEndpoint.py I noticed this:

2018-07-04 21:31:50,154 ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl – SOAP fault

com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.

        at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)

        at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:116)

        at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:259)

        at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289)

        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:161)

        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:114)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.    java:784)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenService    Impl.java:714)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:135)

       at com.vmware.vim.lookup.client.tool.util.SamlTokenUtil.acquireBearerToken(SamlTokenUtil.java:86)

        at com.vmware.vim.lookup.client.tool.util.LookupHelper$LsClient.createStub(LookupHelper.java:52)

        at com.vmware.vim.lookup.client.tool.command.Command.callLsEx(Command.java:180)

        at com.vmware.vim.lookup.client.tool.command.Command.callLs(Command.java:154)

        at com.vmware.vim.lookup.client.tool.command.ReregisterCommand.execute(ReregisterCommand.java:48)

        at com.vmware.vim.lookup.client.tool.LsTool.app(LsTool.java:67)

        at com.vmware.vim.lookup.client.tool.LsTool.main(LsTool.java:103)

2018-07-04 21:31:50,158 INFO  com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor – Provided credent    ials are not valid.

2018-07-04 21:31:50,159 WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase – S    hutting down the connection monitor.

2018-07-04 21:31:50,159 WARN  com.vmware.vim.vmomi.client.http.impl.HttpConfigurationCompilerBase$ConnectionMonitorThreadBase – I    nterrupted, no more connection pool cleanups will be performed.

com.vmware.vim.lookup.client.tool.exception.TokenAcquisitionException

        at com.vmware.vim.lookup.client.tool.util.SamlTokenUtil.acquireBearerToken(SamlTokenUtil.java:89)

        at com.vmware.vim.lookup.client.tool.util.LookupHelper$LsClient.createStub(LookupHelper.java:52)

        at com.vmware.vim.lookup.client.tool.command.Command.callLsEx(Command.java:180)

        at com.vmware.vim.lookup.client.tool.command.Command.callLs(Command.java:154)

        at com.vmware.vim.lookup.client.tool.command.ReregisterCommand.execute(ReregisterCommand.java:48)

        at com.vmware.vim.lookup.client.tool.LsTool.app(LsTool.java:67)

        at com.vmware.vim.lookup.client.tool.LsTool.main(LsTool.java:103)

Caused by: com.vmware.vim.sso.client.exception.AuthenticationFailedException: Provided credentials are not valid.

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition(SecurityTokenSer    viceImpl.java:852)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.    java:788)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenService    Impl.java:714)

        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireToken(SecurityTokenServiceImpl.java:135)

        at com.vmware.vim.lookup.client.tool.util.SamlTokenUtil.acquireBearerToken(SamlTokenUtil.java:86)

        … 6 more

The error did not make any sense as I was 100% sure that I was putting the correct credentials, at that point I thought why not attempt issuing the command without the password included and when I get prompted I would enter the password, and you know what IT WORKED!

I still do not know why this happened (I did update the case with GSS with my findings, and they get back to me I’ll update this blog post for sure), in addition we were using VMware vCenter Server 6.5 U1g.

Lessons learned: Read the logs, read the logs, eat the logs, memorize the logs, be the logs, you’re now the logs and you can validate if your configuration was sucessful ;-).

Thank you for reading,
(Abdullah)^2