It took me a while to understand why this was not working - it could be because I hate - actually loath -having to dig through logs because of Java and Tomcat issues, but I only have my self to blame for this one.
I am currently installing a new vCenter for my Production Environment (this is part of my MJTV series that I currently going through the process). The last time we installed - we were just starting out with VMware – and there have been a decent amount of problems that we have encountered because of lack of experience and knowledge. Therefore a new vCenter (not from scratch but that is another post entirely).
Fast forwarding a couple of years - the technology has evolved - and I have gained more knowledge. So one of the things that were never implemented correctly was an SSL Certificate for vCenter. I wanted to do this right so I started out on what and how this should be done.
Firstly – this is the official VMware reference document. Since we are a Microsoft shop with a established PKI Infrastructure I went to page 2 - Replacing Default Server Certificates with Certificates Signed by a Commercial CA.
Ok so first things first. In order to create the Certificate Signing Request (CSR) you will have to download the OpenSSL binaries from here. Since the vCenter is a 64-bit box – I got the 64-bit version. Before installing the software you will need to download and install I installed the Visual C++ 2008 Redistributables (x64) as well otherwise you will not be able to run the binaries.
I installed to it all to C:\Program Files\OpenSSL. In the bin Directory of the installation folder are the files you will work with.
First you generate an RSA key for your host.
C:\Program Files\OpenSSL\bin>openssl.exe genrsa 1024 > rui.key
A small pause here. The openssl.cfg is the configuration file for the application. I wanted to install a certificate with two different hostnames. Why you may ask? well actually it is very simple. Not all of the users will always remember to put in an Fully Qualified Domain Name when accessing the vCenter server. True they should – but it doesn’t always work that way. So i wanted the SSL certificate to be valid both for the FQDN and the short hostname - i.e. vcenter.maishsk.local and just plain vcenter. So how is this done - with a field in your certificate called subject alternative name (altName). How do you get this into your CSR - well following the great advice from this link, I added to the openssl.cfg file in the [req] section
[req] req_extensions = v3_req
And in the v3_req section:
[ v3_req ] subjectAltName = @alt_names [alt_names] DNS.1 = vcenter.maishsk.local DNS.2 = vcenter
Next I created the CSR
C:\Program Files\OpenSSL\bin>openssl req -new rui.key > rui.csr -config openssl.cfg
To check if the CSR was created correctly with the multiple hostnames, I ran
C:\Program Files\OpenSSL\bin>openssl req -text -noout -in $CSR_FILENAME
and got output similar to this
Requested Extensions: X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment X509v3 Subject Alternative Name: DNS:vcenter.maishsk.local, DNS:vcenter
From there to get the actual Certificate for your CA, browse to:
http://<CA_HOSTNAME>/certsrv
Click on Request a Certificate
Click on Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.
Open the rui.csr file that you saved a few steps above with notepad and copy all of the the contents including the
"-----BEGIN CERTIFICATE REQUEST-----"
and
"-----END CERTIFICATE REQUEST-----" lines
Paste the contents into the Saved Request field
Choose Web Certificate from the Certificate Template and click on the Submit button.
Select Base 64 Encoded and click Download Certificate and save the certificate to C:\Program Files\OpenSSL\bin> and make sure you save the file as rui.crt!
Next I create the .pfx (personal individual exchange) file for rui.crt.
C:\Program Files\OpenSSL\bin> openssl pkcs12 -export -in rui.crt -inkey rui.key -name vcenter.maishsk.local -out rui.pfx
I was prompted with a password request
Loading 'screen' into random state - done Enter Export Password: Verifying - Enter Export Password:
-------------- Make note of this part - because this is where it went wrong.------------------------ I pressed Enter twice - because I did not enter a password before when creating the CSR
I stopped both the VMware VirtualCenter Management Webservices and the VMware VirtualCenter Server. I then copied the three files rui.crt, rui.key, and rui.pfx to the default SSL location which is, according to the official Whitepaper from VMware:
Unfotunately - that is not 100% accurate. The correct path should be:
C:\ProgramData\VMware\VMware VirtualCenter\SSL
If you try to access the path in the Whitepaper you will get a nice
So I backed up the old SSL Certificate in the folder and pasted my new files.
And started the Service again.
I tried to access the server with https://vcenter.maishsk.local and https://maishsk and everything worked fine. Certificate was good no errors - all was hunky dorey!
This is what the Altname part of the Certificate looks like by the way
Opened up the vSphere client - accessed the vCenter server and was not presented with a Certificate warning so all was good - or at least I thought so.
Later on in the day I went to look to see if all was ok - and noticed that there was no vCenter Health Status Icon
So I looked at the plug-ins and got this
Now naturally - I went looking for the problem on the web and this post popped up.
http://kb.vmware.com/kb/1010641 did not resolve my problem.
I tried to access the the link which was https://vcenter.maishsk.local:8443/sdk and it was not working, not as if I was getting a blank page, there was no response from the other side
Next was to look at the vCenter server to see if it was listening on port 8433
That is easy enough, back to the command-line.
netstat -a | findstr 8443
(for those of you who do not know findstr is DOS tool like grep)
Findstr
Searches for patterns of text in files using regular expressions.
Syntax
findstr [/b] [/e] [/l] [/r] [/s] [/i] [/x] [/v] [/n] [/m] [/o] [/p] [/offline] [/g:file] [/f:file] [/c:string] [/d:dirlist] [/a:ColorAttribute] [strings] [[Drive:][Path] FileName [...]]
And that came up empty
C:\Program Files\OpenSSL\bin>netstat -a | findstr 8443
C:\Program Files\OpenSSL\bin>
Just to make sure that I was not using the wrong syntax I looked for port 443
C:\Program Files\OpenSSL\bin>netstat -a | findstr 443 TCP 0.0.0.0:443 VCENTER:0 LISTENING TCP 1xx.xx.3.xx:443 msaidelk-server:59260 ESTABLISHED TCP 1xx.xx.3.xx:443 msaidelk-server:59261 ESTABLISHED TCP [::]:443 VCENTER:0 LISTENING
C:\Program Files\OpenSSL\bin>
So SSL was working - but not on port 8443. Now this was wierd.
I went to look at the Tomcat Logs (located c:\Program Files (x86)\VMware\Infrastructure\tomcat\logs\catalina.2009-12-24.log)
And in the log I saw the following.
Dec 24, 2009 10:41:55 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080 Dec 24, 2009 10:41:56 PM org.apache.tomcat.util.net.jsse.JSSESocketFactory getStore SEVERE: Failed to load keystore type PKCS12 with path C:\ProgramData\VMware\VMware VirtualCenter\SSL\rui.pfx due to failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded …..
Dec 24, 2009 10:41:56 PM org.apache.coyote.http11.Http11Protocol init SEVERE: Error initializing endpoint java.io.IOException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded at com.sun.net.ssl.internal.ssl.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1275) ….
And so on and so on…
Now after searching for the topic on VMware forums and anything connected with VMware and coming up with a complete blank I widened the search to a pure Tomcat and Java issue and came up with Unable to import openssl key to java keystore and this post which were suggesting that the rui.pfx that I created earlier should have been password protected.
Now looking at C:\Program Files (x86)\VMware\Infrastructure\tomcat\conf\server.xml brought me to find this configuration
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="false"
clientAuth="false" sslProtocol="TLS"
keystoreFile="C:\ProgramData\VMware\VMware VirtualCenter\SSL\rui.pfx"
keystorePass="testpassword" keystoreType="PKCS12"
Now where had I seen that string before?? hmmm…
(And this is where things went wrong). I had started to follow a walkthrough that was posted on the VMTN, and on this page there was no mention of what the password was supposed to be. So I naturally pressed Enter - Twice - and continued.
Remember I said above
-------------- Make note of this part - because this is where it went wrong.------------------------ If I had read the White paper carefully - it explicitly states
So pushing Enter twice - was not a good idea after all, I should have entered the password as above when prompted or entered the command like above.
A quick change of the rui.pfx. Stop Services. Copy new file. Start Services.
And….
Logs were clean
Dec 24, 2009 11:28:22 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: C:\Program Files (x86)\VMware\Infrastructure\tomcat\bin;.;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem Dec 24, 2009 11:28:22 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-127.0.0.1-8080 Dec 24, 2009 11:28:22 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-8443 Dec 24, 2009 11:28:22 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 1080 ms Dec 24, 2009 11:28:22 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Dec 24, 2009 11:28:22 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.20 Dec 24, 2009 11:28:26 PM org.apache.catalina.loader.WebappClassLoader validateJarFile INFO: validateJarFile(C:\Program Files (x86)\VMware\Infrastructure\tomcat\webapps\sms\WEB-INF\lib\servlet-api.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Dec 24, 2009 11:28:30 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-127.0.0.1-8080 Dec 24, 2009 11:28:30 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-8443 Dec 24, 2009 11:28:30 PM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:8009 Dec 24, 2009 11:28:30 PM org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=0/31 config=null Dec 24, 2009 11:28:30 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 8060 ms
My vCenter Service was back
And all my Plug-ins were working
So what have I learned from this experience.
- RTFM!!!!!!!!!!!
- Read it again!!!
- Debugging an issue like this takes time. It does give a large amount of satisfaction actually finding the problem and even more so finding the solution.
- Writing a blog post like this can take over two hours :)
- Have a happy holidays everyone!