The SQL Dude!

I had posted quite a lengthy post on setting up SQL Server for SSL encryption back in October. This post has frequently ranked high on page views, thanks all for the overwhelming response. Since the time, I’ve got pinged so many times on this, that I really wanted to post another blog covering what I’ve learnt during this time and answers to some common problems faced when setting up SSL Encryption for SQL Server. So, here goes…

---

Basics

If a certificate is provided to SQL Server and for some reason it is not valid or SQL cannot find the certificate in the store, then it generates a self-signed certificate to encrypt communication between the server and the client. This method ensures that the login packet is always encrypted. Whether the data communication is also encrypted depends on both the server and the client. In order for the client to force encryption, the certificate used by the server should be signed by a trusted certificate authority.

When loading a certificate on the SQL Server machine, you have to keep in mind what the SQL startup account is.

1. If the SQL Server is running under a specific domain account, then you need to be logged in to the machine as the same domain account and when opening MMC, choose this option to load the Certificates snap-in, before doing the import. This makes sure that the certificate goes into the Personal store of the user who is also the SQL Service account.

2. If the SQL Server is running under any machine account like LocalSystem, NetworkService or LocalService, then you need to choose the option “Computer Account” in the above screenshot and then import the certificate. This will ensure that the certificate is placed under the Personal store of the machine account.

Needless to say for doing both of the above, the logged-in account must be an administrator on the machine.

---

Issues

1. SQL Server Configuration Manager (SSCM)  – How does it pull up the certificates and why doesn’t it show my valid certificate it the list?

Usually, SQL DBA’s don’t create certificates. This is usually done by someone else in your organization who will either install Certificate Services on a server and make it a trusted authority or buy a certificate from any on the certificate providers like VeriSign, GoDaddy.com etc. So when this certificate is not shown in SQL configuration manager, you are stuck. As I mentioned in the earlier blog post you can manually load the certificate by adding the thumbprint of the certificate into the SQL registry key “Certificate” in the location HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10.<INSTANCENAME>\MSSQLServer\SuperSocketNetLib

I always wanted to find out how SSCM shows the certificate list and why sometimes some certificates even if valid are not shown there. SSCM uses WMI infrastructure to find out the certificates which are valid to be used by SQL Server. Now there are some additional checks done here which is not done by SQLServr.exe when actually loading the certificate on service startup.

You can use the following WMI query to see a list of certificates. This will be the same output you see in the SSCM Certificate drop-down. See example below

From SSCM drop-down certificate picker

• As you can see above it listed 2 certificates on my machine.

Using the WMI Query

Save the below code as sqlcerts.vbs

strComputer = "."
Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\Microsoft\SqlServer\ComputerManagement10")
Set colItems = objWMIService.ExecQuery( _
"SELECT * FROM SecurityCertificate",,48)
For Each objItem in colItems
Wscript.Echo "———————————–"
Wscript.Echo "SecurityCertificate instance"
Wscript.Echo "———————————–"
Wscript.Echo "ExpirationDate: " & objItem.ExpirationDate
Wscript.Echo "FriendlyName: " & objItem.FriendlyName
Wscript.Echo "IssuedBy: " & objItem.IssuedBy
Wscript.Echo "IssuedTo: " & objItem.IssuedTo
Wscript.Echo "Name: " & objItem.Name
Wscript.Echo "SHA: " & objItem.SHA
Wscript.Echo "StartDate: " & objItem.StartDate
Wscript.Echo "SystemStore: " & objItem.SystemStore
Next

Run this from command prompt cscript sqlcerts.vbs and you will see the output as shown below. I see the same 2 certificates shown by SSCM. So now we know that the certificates are shown by pulling the information from the WMI class SecurityCertificate.

Note:
If you are using using SQL 2005, in the above VBScript, replace ComputerManagement10 with ComputerManagement.

C:\Users\sudarn\Desktop>cscript certs.vbs
Microsoft (R) Windows Script Host Version 5.8
Copyright (C) Microsoft Corporation. All rights reserved.

———————————–
SecurityCertificate instance
———————————–
ExpirationDate: 30363701
FriendlyName: SUDARN2 (SQLCert)
IssuedBy: US, VA, Somewhere, MyOrg, MyOU, SUDARN2.org.company.com
IssuedTo: US, VA, Somewhere, MyOrg, MyOU, SUDARN2.org.company.com
Name: SUDARN2 (SQLCert)
SHA: 876753a93eb4310e4415a563219a41dda732c260
StartDate: 30216851
SystemStore: MY
———————————–
SecurityCertificate instance
———————————–
ExpirationDate: 31674726
FriendlyName: SCVMM_CERTIFICATE_KEY_CONTAINERSUDARN2.org.company.com
IssuedBy: SUDARN2.org.company.com
IssuedTo: SUDARN2.org.company.com
Name: SCVMM_CERTIFICATE_KEY_CONTAINERSUDARN2.org.company.com
SHA: 4ca85862a6ed93e329f68283c6d2ad1710e2c23f
StartDate: 30205010
SystemStore: MY

Part 2 of this question is why the SSCM doesn’t show your certificate. Here are some common reasons :-

1. If yours is a clustered instance of SQL Server, then the CN in your certificate will contain the FQDN of your virtual server name of the instance and not the node names. SQL Configuration Manager does a direct match between the current machine name and the CN name in the certificate [i.e. certificates that match the machine name are only displayed], which will never match in case of a clustered instance. So you can ignore this and use the registry method to load the certificate.
   
2. The certificate has expired.
   
3. The Server Authentication property or the AT_KEYEXCHANGE property is not set.
   
4. The Certificate is using Subject Alternate Names (SAN), which is not understood by SSCM.

 

2. Subject Alternate Names (SAN’s) and whether a certificate having SAN’s can be used by SQL Server.

Subject Alternative Names allow you to specify a list of host names to be protected by a single SSL certificate. This is useful when you want to buy 1 single certificate to secure multiple SQL Servers. Cheaper option I guess . But anyways, certificates having SAN names are supported and SQL Server can use it for SSL encryption. If you provide the thumbprint value in registry, SQL Server will load this certificate. SQL Server doesn’t examine the SAN while choosing a certificate to load. I confess I hadn’t heard about SAN names before, so I tested this by creating a certificate using openssl.exe by having 2 alternate names. You can download openssl.exe for testing purposes from HERE.

1. Create a configuration file as follows. I called it cert_config.txt

distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = VA
L = Somewhere
O = MyOrg
OU = MyOU
CN = SQLSERVER.SUDARN.COM
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = alternatename1
DNS.2 = alternatename2
IP.1 = 10.191.84.22
IP.2 = 10.191.84.23

The alternate names I have used are alternatename1 and alternatename2. Replace this with actual server names as applicable.

2. Once you install the tool, go to the ..\bin\openssl.exe location from command prompt and run the following command to create the certificate.

openssl.exe req -x509 -nodes -days 730 -newkey rsa:2048 -keyout C:\Users\sudarn\Desktop\cert.pem -out C:\Users\sudarn\Desktop\cert.pem -config C:\Users\sudarn\Desktop\cert_config.txt

3. Run the following command to create the PFX file

openssl.exe pkcs12 -export -out C:\Users\sudarn\Desktop\cert.pfx -in C:\Users\sudarn\Desktop\cert.pem -name "SUDARN2 (SQLCert)" -passout pass:mypassword

This will create a file called cert.pfx which can be used to install the certificate or imported using the MMC certificates snap-in.

Now that you a certificate with 2 alternate names you can see for yourself that SQL Server can load this certificate fine. Again you will see the SQL Config Manager doesn’t show this certificate. Most cases the main CN name will not match the machine name in case of SAN certificates and that’s why SSCM doesn’t show you the certificate.

Important
The SAN field in the SSL certificate is not examined by all SQL Server client drivers when they are validating the server certificate. Only fairly recent drivers examine the SAN when validating a certificate, like the following.

– SQL Server Native Client 10.5 (and higher)
– ADO.Net SqlClient 4.5
– JDBC 4.0

So, if the client application is using .Net 4.0, for example, and you try to rely on the SAN, the client application will not accept it. Keep this factor in mind when deciding to use SAN certificates or go with individual certificates for each machine.

3. I’ve setup SSL encryption but when I connect to my SQL Server, I get this error

Cannot connect to thematrix.CORP.COMPANY.COM
——————————
ADDITIONAL INFORMATION:

A connection was successfully established with the server, but then an error occurred during the pre-login handshake.
(provider: SSL Provider, error: 0 – The certificate’s CN name does not match the passed value.)
(Microsoft SQL Server, Error: -2146762481)

I was able to reproduce this issue quite easily. As you can see in the above error message, the reason the connection didn’t go through was because the certificate’s CN Name did not match the SQL Server name.

As documented in Books Online, there are 5 properties for a certificate that are mandatory to be use by SQL Server which are documented here http://msdn.microsoft.com/en-us/library/ms189067.aspx 

Here is the catch!

If your SQL Server machine is called “THEMATRIX” and you create a certificate called “MyServerName” and you provide the thumbprint of this certificate directly in the SQL server registry, SQL will load this certificate just fine. Surprised? Have a look at my repro.

1. I created a certificate using openssl.exe with the following properties.
CN = MyServerName
extendedKeyUsage = serverAuth
Valid From: 29 March 2012 21:27:39
Valid To:    29 March 2014 21:27:39

2. I loaded this certificate using the SHA1 hash value directly in the registry key "Certificate" in the following location,
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQLServer\SuperSocketNetLib

3. SQL Server successfully loaded this on restart of the service.
2012-04-03 02:52:57.100 Server       The certificate [Cert Hash(sha1) "A1A3DB2BD029B39FF9329E46B730CF8DF4BE2383"] was successfully loaded for encryption.

4. When I connected from SSMS, I get the same error as reported by my customer.
provider: SSL Provider, error: 0 – The certificate’s CN name does not match the passed value.) (Microsoft SQL Server, Error: -2146762481)

So now read the documentation again and it will make sense. What Books Online is telling you is that for a successful SSL encryption and connection, the subject Name in the certificate has to match the FQDN/Server name. In the above scenario, I deleted the certificate “MyServerName” and created a new certificate with the CN as “TheMatrix” and was able to connect successfully from SSMS.

4. Again with a new SSL encryption setup, you are faced with this error when connecting from SSMS

(provider: SSL Provider, error: 0 – The certificate chain was issued by an authority that is not trusted.)

This messages tells us that a server side certificate is used but the client could not validate the server side certificate. If your certificate is from a trusted authority but the client connection is failing with above error, do this

1) Go to SQL Configuration Manager
2) Go to SQL Native Client 10.0 Configuration > Right Click and Properties > Flags tab >
3) Set the property "Trust Server Certificate" to Yes
4) Restart the SQL Service

When the above property is set to True, SSL is used to encrypt the channel whilst bypassing walking the certificate chain to validate trust. If TrustServerCertificate is set to true and Encrypt is set to false, the channel is not encrypted. So you need to have "Force Protocol Encryption"=Yes and "Trust Server Certificate"=Yes.

5. The certificate thumbprint value in the registry is getting automatically changed whenever a failover of SQL Server is done

I must confess this was a strange issue to encounter and I saw this happening with my own eyes. Every time the SQL instance was failed over, even though the certificate was present on both the nodes, SQL was using a different self-signed certificate when starting up. I initially suspected this to be a cluster registry checkpoint issue, but after making sure that the registry value was correct on both the nodes, still on next failover some new thumbprint was getting loaded and this thumbprint value was also being placed in the registry.

There is a thumb-rule you can follow. When in doubt, use Process Monitor. So I did . I collected a process monitor trace without any filters and saw this happening.

00:57:03.7155401 wmiprvse.exe 10752 RegQueryValue HKLM\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10.A1XX02\MSSQLServer\SuperSocketNetLib\Certificate 6.0.6002.18005 SUCCESS Type: REG_SZ, Length: 82, Data: 7697654a3be60f1931ec04b37eae21af98aea1bd

00:57:03.7263934 wmiprvse.exe 10752 RegSetValue HKLM\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL10.A1XX02\MSSQLServer\SuperSocketNetLib\Certificate 6.0.6002.18005 SUCCESS Type: REG_SZ, Length: 82, Data: b8c0cf223b0efb4ca8ede15dd910400488a06631

From the above snippet you can see that wmiprvse.exe (WMI) is setting the value of the certificate to a new value. The thumbprint value is getting changed to a new value from what was there before. On further research, it was identified that this environment had System Center 2012 Configuration Manager installed and these SQL Server nodes were being managed by System Center. My colleague from the System Center team confirmed that there is a component there called “Certificate Manager” that frequently polls all the systems (ones with any roles installed) to check whether the certificate information there matches with what is stored in the System Center database (for consistency reasons). Obviously, since this is a clustered instance of SQL Server, we have edited the registry to provide the certificate value. System Center is not aware of this change, and it thinks that the certificate has been changed and puts in a request to generate a new self-signed certificate and updates the registry with the thumbprint of the new certificate. This was what was happening here. I am told System Center 2012 Service Pack 1 will have a fix for this problem. If you want confirmation, contact System Center MS Support.

6. You have configured SSL encryption for your SQL Server by manually entering the Thumbprint of a certificate, and now the SQL Service fails to start

There are many reasons for this, which you need to find out from the SQL Error log. One common problem that can happen when inputting the thumbprint into the registry when you Ctrl+C the value and paste it into the registry, some extra spaces/junk/invisible characters get copied over if you copy this from MMC. Have a look at the following KB article.

2010-04-16 18:56:31.48 Server      Error: 17182, Severity: 16, State: 1.
2010-04-16 18:56:31.48 Server      TDSSNIClient initialization failed with error 0xd, status code 0x38

SQL Server fails to start with error 17182 "TDSSNIClient initialization failed with error 0xd, status code 0x38" when server is configured to use SSL
http://support.microsoft.com/kb/2023869

 

7. You don’t see the option to export the Private Key when trying to export the certificate from one machine to another machine

This is definitely outside the realms of SQL Server. When you try to export the certificate you get to the screen where you have to choose the option to export the Private Key of the certificate, but the option is greyed out as shown below.

The reason the option to export the private key is greyed out is because the certificate was created without the option to export keys. There is an option during certificate creation for Server Authentication to “Mark Keys as exportable”, and you should choose this option. If you don’t, you can still export the certificate and install it on another node, but it won’t have the private key and without the private key SQL Server will not be able to load the certificate and hence cannot start.

---

Well folks, I’ve covered a lot of topics here. Believe me if you can follow these steps, I’m pretty confident you should have any issues before, during and after configuring SQL Server for SSL encryption. If you do happen to run into any issues, please drop in a comment on this post and I will respond back to this. Cheers!