VMware vCenter Smart Card Authentication Stops Working after Upgrading to 7.0 U3i

January 26, 2023

Reading time: 3 minutes

After testing a recent upgrade to VMware vCenter Server 7.0 Update 3i, I encountered an issue where the vCenter Server would no longer authenticate users via smart cards/X.509 certificates. The vCenter Server would not even request a certificate from the client’s browser anymore. This seemed odd as the functionality worked fine on the previous 7.0 Update 3h. Surely VMware wouldn’t make a breaking change within a minor patch release?

After reverting the upgrade and testing that it wasn’t an issue with the upgrade process itself, a support ticket was opened with VMware support. To their credit, they quickly answered back with a reference to VMware KB90542 and stated that the TCP port used for smart card authentication had changed and is now TCP 3128. Armed with this information, my team verified that indeed port TCP 3128 was being blocked at the firewall. Firewall rules were changed, and testing verified that smart card authentication was indeed successful again.

Moving on to a couple of weeks later, we began deploying this update to a production vCenter Server instance. Everything appeared to work great until we tested smart card authentication. At first, the process seemed normal as the browser prompted us to provide a certificate, but immediately after providing the certificate, the authentication would fail. My first instinct was to check the log /var/log/vmware/sso/websso.log on the vCenter Server to determine what the issue might be. Often this is where issues with OCSP revocation or other certificate trust-related items are logged. To my surprise, absolutely nothing was logged during the authentication attempt. This seemed quite odd and made me think that something was missing within the configuration that was preventing the VMware rhttpproxy service from evening passing the certificate information to the VMware SSO service.

My next instinct was to review the documentation for configuring smart card authentication. Specifically, I reviewed the documentation titled Configure the Reverse Proxy to Request Client Certificates. Immediately I noticed something interesting about step number 3 in this document. This step included a caveat that it applied only to vCenter versions prior to 7.0 U3i. Within this step, you are required to specify the path of your trusted CA certificates within the configuration file /etc/vmware-rhttpproxy/config.xml. I checked the configuration file on this vCenter Server and sure enough, the path specified was different from the path used in the documentation. I moved the trusted CA certificates file to /usr/lib/vmware-sso/vmware-sts/conf/clienttrustCA.pem as specified in the documentation and then noticed that step 4 also had instructions that differed based on the patch version. Instead of restarting the rhttpproxy service, it stated that you should restart the sts service instead. I followed the instructions for 7.0 Update 3i and again tested smart card authentication. Testing was successful and we were able to close out the maintenance successfully.

Based on the documentation changes and the KB article, it appears that within this minor patch release for VMware vCenter Server 7.0, a change was made to no longer utilize the VMware rhttpproxy instance to complete the smart card/X.509 authentication process and instead VMware moved the certificate exchange process to a new TCP port that is managed by the VMware STS service.

If you are encountering similar issues, carefully review VMware KB90542. Hopefully, this helps you resolve your issue as well.