Users have reported problems with SSL certificates inside a distributed monitoring setup when they
- updated their Icinga 2 package to 2.7.0 on Windows or
- upgraded their distribution which included an update to OpenSSL 1.1.0.
The Windows package also uses OpenSSL 1.1.0 in v2.7.0. This seems to only affect certificates created in 2015 or older.
Problem description
If your client refuses to start, the error message will look like this:
critical/SSL: Error loading and verifying locations in ca key file 'C:\ProgramData\icinga2\etc/icinga2/pki/ca.crt': 219029726, "error:0D0E20DE:asn1 encoding routines:c2i_ibuf:illegal zero content" critical/config: Error: Cannot make SSL context for cert path: 'C:\ProgramData\icinga2\etc/icinga2/pki/client.crt' key path: 'C:\ProgramData\icinga2\etc/icinga2/pki/client.key' ca path: 'C:\ProgramData\icinga2\etc/icinga2/pki/ca.crt'.
This means that the CA public certificate contains an invalid serial number. More specifically, the serial number starts with a leading zero. OpenSSL versions older than 1.1.x ignored this behaviour silently, 1.1.0 will now reject the certificate(s) as invalid.
All technical details are collected in #5511. Please discuss and add your findings over there.
How can I manually analyse a certificate?
Print the certificates as text and analyse their attributes.
openssl x509 -text -in ca.crt openssl x509 -text -in client.crt
Verify that the CA certificate has the following attributes set:
- version: 3
- serial: hex-encoded string (no zero padding)
- x509v3 extensions with CA:TRUE constraint
A faulty CA certificate (ca.crt) looks like this:
Version: 1 (0x0) Serial Number: 0 (0x0) Signature Algorithm: sha256WithRSAEncryption Issuer: CN=Icinga CA Validity Not Before: Mar 20 12:56:47 2015 GMT Not After : Mar 12 12:56:47 2045 GMT
Version is 1, not 3 and the Serial Number is 0 which leads to the aforementioned error. You can also see that the CA certificate was generated in early 2015, and it lasts 30 years. The latter has been corrected to 15 years due to bugs with older OpenSSL versions.
Client certificates can be verified the same way, except that the x509v3 extensions have the CA:FALSE constraint set.
Deeper analysis
We could not reliably reproduce the error yet. According to this forum post, if only affects certificates which have been created in early 2015 with versions older than 2.3.8. This probably affects OpenSSL 1.0.x, but was not reproducible with OpenSSL 0.9.8.
It is highly likely that Debian Jessie at that time included OpenSSL 1.0.x which generated the faulty certificate serial without any error. Previously this has been a hidden warning only. Now OpenSSL 1.1.0 enforces a stricter policy and rejects these certificates as invalid.
We’ve found this discussion on the OpenSSL mailinglist which sounds like a similar issue. Fedora 26 also seems to have this problem.
Please share your details in #5511 which will help reproduce the issue. Thanks.
Possible solution
First off, check whether your CA certificate is affected. Create a new public CA certificate first, and check whether it fixes your issue. If not, regenerate the client certificates too and deploy them.
This requires native OpenSSL CLI commands. DO NOT create a new Icinga CA at all. Create a backup of your certificates in /etc/icinga2/pki (%ProgramData%\Icinga 2\etc\icinga 2\pki on Windows) and /var/lib/icinga2/ca first!
All steps involving certificate re-generation should be done on your Icinga 2 master instance.
Re-create your public CA certificate
This uses a small trick to just re-new an expired certificate.
- Generate a CSR from an existing CA key pair (public and private key)
- Sign the CSR with the CA’s private key
You’ll need to run both steps using OpenSSL CLI commands as we need to specify the v3 extensions including the CA:TRUE constraints. Note: The extensions must be specified as file content, the exact notation is shown below.
cd /var/lib/icinga2/ca openssl x509 -x509toreq -in ca.crt -signkey ca.key -out regenerated-ca.csr cat > icinga2-ext.cnf <<EOF [v3] basicConstraints = critical,CA:true EOF openssl x509 -in regenerated-ca.csr -out regenerated-ca.crt -signkey ca.key -req -days 5475 -extensions v3 -extfile icinga2-ext.cnf mv ca.crt ca.crt.borked mv regenerated-ca.crt ca.crt
5475 days is equal to 15 years, the same default the Icinga 2 cli commands use.
Copy the newly generated ca.crt certificate file to ALL your Icinga 2 instances into
- /etc/icinga2/pki (Linux)
- %ProgramData%\Icinga 2\etc\icinga 2\pki
If your clients do not use OpenSSL 1.1.0 yet, you can save this step for later. Sooner or later you will be bitten by this change, depending on distribution upgrades. Keep that in mind.
Re-create your client certificates
Note: Test if the re-generated ca.crt certificate already fixed the problem on one of your clients. This user reported that only the CA certificate was affected.
This works in a similar fashion like the CA key pair re-generation.
Create a CSR from the existing client key pair on the client:
openssl x509 -x509toreq -in client.crt -signkey client.key -out client.csr openssl req -text -in client.csr
Copy the CSR to your Icinga 2 master where the CA private key is stored in /var/lib/icinga2/ca. Sign the certificate using the Icinga 2 CLI command:
icinga2 pki sign-csr --csr client.csr --cert client.crt
Copy the newly created client private and public certificates to your client into
- /etc/icinga2/pki (Linux)
- %ProgramData%\Icinga 2\etc\icinga 2\pki
Conclusion
You are likely not affected if you started to deploy your Icinga 2 cluster with > 2.4.x. Unfortunately the error was not visible back in 2015, and neither OpenSSL 1.1.0 was ready at that time to test and immediately throw an error.
If you are in the unlucky position that either your distribution upgrade to Debian Stretch or Windows with v2.7.0 broke several clients, immediately halt the upgrade procedure and fix the affected clients first. Also re-generate the master’s public CA certificate with the steps shown above. Then include the re-generated certificates in your upgrade procedure.
Keep in mind that upcoming distributions will highly likely incorporate OpenSSL 1.1.0 in the future (enterprise linux only in major versions, rolling distributions will receive it sooner). Stay safe by analysing your existing environment using this advisory, and fix your certificates before it bites you in five years or similar!
We need your help: Let us know inside the Github issue whether you are affected. If you are able to, share the client’s .crt, .key and ca.crt files with us via info(at)icinga.com or an Icinga partner before re-creating them on a client of yours. Or provide insights on how to reliably reproduce the issue for others. Thanks in advance!
