JAVA_TOOL_OPTIONS on a VMware Horizon Connection Server = Not Good

I encountered another fun adventure recently while upgrading some VMware Horizon 8 Connection Servers to version 2306. The upgrade process itself went smoothly, as you would expect. The installer processed the upgrade and stated that it was completed successfully. I attempted to access the VMware Horizon Administrator web console and found it would not process my smartcard authentication. I then tried to authenticate using an account with a password assigned, and again, the authentication was unsuccessful.

I thought there might be an issue with a service that didn’t start or needed a restart to complete a post-upgrade task, so I restarted the VMware Horizon Connection Server service. I waited and waited some more for the VMware Horizon Administrator web interface to load, but it would not. All the services showed that they were running, but the server would not accept any connection requests. This seemed quite odd. Obviously, the upgrade process did not go as smoothly as I thought. So, I was off to review the log files.

Reviewing the debug logs for the server, I found many ERROR entries that stated that the JMS service could not find matching thumbprints for the broker certificate. The entries were similar to the following:

ERROR (299C-1B7C) <SwiftMQ-ConnectorPool-3> [JMSTunnelSSLSocketFactory] Certificate thumbprint verification failed, no router thumbprints available. Presented identity: router/<hostname>
ERROR (299C-1B7C) <SwiftMQ-ConnectorPool-3> [JMSTunnelSSLSocketFactory] General error occurred: No known thumbprints: router/<hostname> " 

That seemed odd, so I began investigating the thumbprints for the broker certificates in the Windows certificate store. I compared those to the thumbprints that were stored in the ADAM database. I found that the thumbprints matched. I continued to dig through the logs and came up with nothing. Finally, after a few hours of searching through the logs, I caved and opened a support case with VMware Technical Support.

Their initial response was that my issue matched a known internal KB article based on the error entries I saw in my log files. However, the KB’s description of the issue didn’t align as it stated “This issue occurs because windows is unable to read the certificates and Windows API fails to add the certificate thumb print to ADAM database”. I attempted to explain to them that, in my case, they did match. Still, they insisted that I follow the internal KB, which stated that I needed to uninstall the VMware Horizon software, delete the VMware Horizon certificates from the Windows certificate store, and then reinstall the VMware Horizon software. So, I did exactly as they requested, and the behavior persisted. Upon reinstall, the VMware Horizon Administrator web interface would load but not authenticate users. Upon restarting the VMware Horizon services, the server would stop accepting all connections.

VMware Technical Support scheduled a phone call with me and a senior support engineer a few days later. After reviewing the logs for virtually all ERROR and WARNING events, we ran across some additional service authentication error messages that led the engineer to an obscure internal KB article. He then asked me to review one of the installation logs for some entries regarding the generation of certificates. He asked that I look for the string “Sun Microsystems Inc.”, and sure enough, it was in the logs. The log entries resembled the following:

Running start "MakeTunnelKeyPair" /B /WAIT "C:\Program Files\VMware\VMware View\Server\jre\bin\java.exe" -cp "C:\Program Files\VMware\VMware View\Server\lib\*" com.vmware.vdi.messagesecurity.MakeTunnelKeyPair RQBNAE8ALQBDAFMARAAxADEALQBNA....
Output from command was Picked up JAVA_TOOL_OPTIONS: -Djava.vendor="Sun Microsystems Inc."
Running start "MakeMQSSLCertificates" /B /WAIT /D "C:\Program Files\VMware\VMware View\Server\bin" "C:\Program Files\VMware\VMware View\Server\jre\bin\java.exe" -Dcom.vmware.vdi.mfwj.nativelib=ws_java_nativeNODEP -cp "C:\Program Files\VMware\VMware View\Server\lib\*" com.vmware.vdi.messagesecurity.MakeMQSSLCertificates...
Output from command was Picked up JAVA_TOOL_OPTIONS: -Djava.vendor="Sun Microsystems Inc."

He then asked me to check the Windows server environment variables for a system variable containing the “Sun Microsystems” string. Sure enough, the server had this variable. The specific environment variable is: JAVA_TOOL_OPTIONS: -Djava.vendor="Sun Microsystems Inc."

The fix was to remove the environment variable and reinstall the VMware Horizon software. I tested this out, the upgrade worked, and everything was functional. A review of the debug logs also indicated that there were no longer any issues. I’ve tried to recreate this exact issue in my home lab with the current releases of Oracle Java Runtime, but it appears that it does not generate the problematic environment variable.

So, while I can’t explain how the problematic environment variables were created on these VMware Horizon Connection Servers, and I’m unable to recreate the situation with the latest releases of the Java Runtime from Oracle, the moral of the story is that you should not have any unnecessary software installed on your VMware Horizon Connection Servers.

Share:

Search

Get Notified of Future Posts

Follow Me

Recent Posts