In our community forum Michael already outlined the possibility to operate Icinga 2 with an external certification authority, not the one Icinga 2 generates by itself. Thomas, one of our NETWAYS colleagues, reported his experience in that field: in short, it’s easy to mess up lots of things and hard to debug them. And I absolutely agree. At the moment I’m reading TLS Mastery from Michael W. Lucas. And, oh dear, there’s so much to (over)see in TLS, from critical extensions to path lengths. But you can’t mess up by yourself what you delegate to others. So, to mess up as little as possible, I’ve delegated the certificate issuance to Let’s Encrypt, to demonstrate an Icinga 2 cluster with an external CA in this post.
Pros
- TLS clients all over the world trust the whole Icinga 2 cluster out-of-the-box, from curl(1) to your preferred browser. (Yes, at least GET requests against the Icinga 2 API work directly in a browser, too.)
- Certificates are issued automatically and without direct interaction between nodes or copying files across them as in:
- icinga2 pki sign-csr
- icinga2 pki ticket
- icinga2 ca sign
- External CAs not offering ACME
- I’ve done it because I can. (My favourite reason.)
Cons
- Every Icinga node must have an actual FQDN as NodeName, preferably even as the hostname(1) to make things easier, e.g. master1.example.com.
- Every certificate will be listed in public databases such as crt.sh, including:
- CN
- public key
- algorithms
- etc.
- To obtain certificates, one of the following is required:
- Every Icinga node’s port 80 must run a web server and be reachable from the internet. (HTTP-01 challenge, covered in this post)
- Every Icinga node must be able to manage its own ACME challenge DNS record, e.g. _acme-challenge.master1.example.com. Either on a central DNS server or on delegated ones running on each Icinga 2 node and being exposed to the internet over port 53. (DNS-01 challenge)
- While the certificate issuance itself is delegated to a CA, the certificates must be also installed in /var/lib/icinga2/certs/ with the correct permissions. (Normally Icinga 2 takes care of both.)
- Icinga doesn’t watch its certificate files, or even the CRL, for external changes. Due to that fact, the daemon has to be reloaded regularly not to use expired certificates.
Talking is cheap, show me the code!
All right, this was my proof of concept setup:
- Two Debian 12 VMs with public IPs
- Firewall rules allowing at least TCP in:
- port 80 from everywhere
- ports 22 and 5665 from internal
- DNS records, namely:
- 91-198-2-115.nip.io
- 91-198-2-157.nip.io
/etc/hostname and hostname(1)
First of all, I’ve replaced the content of the /etc/hostname files with the respective FQDNs and ran hostname `cat /etc/hostname` to apply those changes. That allowed me to use just `hostname --fqdn` instead of repeating myself subsequently on the command line. Also, by default Icinga 2 uses that hostname as NodeName.
Additional software
Only a few packages are needed for the whole setup on each VM:
icinga2-bin, of course- The web server being used anyway on port 80 for actual websites, if any, in my case
nginx certbotandpython3-certbot-nginx, to obtain certificates
root@91-198-2-115:~# apt install icinga2-bin nginx certbot python3-certbot-nginx
Getting the certificates
… is damn simple thanks to certbot:
root@91-198-2-115:~# certbot certonly --nginx -n --agree-tos --email tadig25348@ibtrades.com -d `hostname --fqdn` Saving debug log to /var/log/letsencrypt/letsencrypt.log Requesting a certificate for 91-198-2-115.nip.io Successfully received certificate. Certificate is saved at: /etc/letsencrypt/live/91-198-2-115.nip.io/fullchain.pem Key is saved at: /etc/letsencrypt/live/91-198-2-115.nip.io/privkey.pem This certificate expires on 2024-01-09. These files will be updated when the certificate renews. Certbot has set up a scheduled task to automatically renew this certificate in the background. ...
Indeed a scheduled task appeared:
root@91-198-2-115:~# tail -1 /etc/cron.d/certbot 0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew --no-random-sleep-on-renew
(The above email address is a throwaway one.)
Populating /var/lib/icinga2/certs/
Again, each node needs /var/lib/icinga2/certs/ itself and inside it:
ca.crtwith the root CA(s)`hostname --fqdn`.crtwith the full chain`hostname --fqdn`.keywith the private key, readable by Icinga
root@91-198-2-115:~# mkdir -p /var/lib/icinga2/certs root@91-198-2-115:~# wget https://letsencrypt.org/certs/isrgrootx1.pem root@91-198-2-115:~# wget https://letsencrypt.org/certs/isrg-root-x2.pem root@91-198-2-115:~# cat isrg* >/var/lib/icinga2/certs/ca.crt root@91-198-2-115:~# cp /etc/letsencrypt/live/`hostname --fqdn`/fullchain.pem /var/lib/icinga2/certs/`hostname --fqdn`.crt root@91-198-2-115:~# cp /etc/letsencrypt/live/`hostname --fqdn`/privkey.pem /var/lib/icinga2/certs/`hostname --fqdn`.key root@91-198-2-115:~# chown nagios: /var/lib/icinga2/certs/*
Of course, those will expire after 90 days and need to be refreshed regularly:
root@91-198-2-115:~# cat <<'EOF' >/etc/cron.d/le-icinga 30 8 * * 1 root cp /etc/letsencrypt/live/`hostname --fqdn`/fullchain.pem /var/lib/icinga2/certs/`hostname --fqdn`.crt; cp /etc/letsencrypt/live/`hostname --fqdn`/privkey.pem /var/lib/icinga2/certs/`hostname --fqdn`.key; chown nagios: /var/lib/icinga2/certs/*; systemctl reload icinga2 EOF
Assembling the nodes to a cluster
… via /etc/icinga2/zones.conf, of course:
object Endpoint "91-198-2-115.nip.io" {
host = "91-198-2-115.nip.io"
}
object Zone "91-198-2-115.nip.io" {
endpoints = [ "91-198-2-115.nip.io" ]
}
object Endpoint "91-198-2-157.nip.io" {
host = "91-198-2-157.nip.io"
}
object Zone "91-198-2-157.nip.io" {
endpoints = [ "91-198-2-157.nip.io" ]
parent = "91-198-2-115.nip.io"
}
Finally, after the cluster functionality itself is enabled, reload Icinga 2:
root@91-198-2-115:~# icinga2 feature enable api root@91-198-2-157:~# systemctl reload icinga2
It works!
After all of the above the nodes should communicate with each other:
root@91-198-2-157:~# tail /var/log/icinga2/icinga2.log [2023-10-11 14:19:13 +0000] information/ApiListener: Sending config updates for endpoint '91-198-2-115.nip.io' in zone '91-198-2-115.nip.io'. [2023-10-11 14:19:13 +0000] information/ApiListener: Finished sending config file updates for endpoint '91-198-2-115.nip.io' in zone '91-198-2-115.nip.io'. [2023-10-11 14:19:13 +0000] information/ApiListener: Syncing runtime objects to endpoint '91-198-2-115.nip.io'. [2023-10-11 14:19:13 +0000] information/ApiListener: Finished syncing runtime objects to endpoint '91-198-2-115.nip.io'. [2023-10-11 14:19:13 +0000] information/ApiListener: Finished sending runtime config updates for endpoint '91-198-2-115.nip.io' in zone '91-198-2-115.nip.io'. [2023-10-11 14:19:13 +0000] information/ApiListener: Sending replay log for endpoint '91-198-2-115.nip.io' in zone '91-198-2-115.nip.io'. [2023-10-11 14:19:13 +0000] information/ApiListener: Finished sending replay log for endpoint '91-198-2-115.nip.io' in zone '91-198-2-115.nip.io'. [2023-10-11 14:19:13 +0000] information/ApiListener: Finished syncing endpoint '91-198-2-115.nip.io' in zone '91-198-2-115.nip.io'. [2023-10-11 14:19:13 +0000] warning/ApiListener: Ignoring config update. 'api' does not accept config.
And with the usual TLS clients:
root@91-198-2-115:~# openssl s_client -connect 91-198-2-157.nip.io:5665 ... Verify return code: 0 (ok) ...
Conclusion
In most cases Icinga’s own CA is absolutely fine. But sometimes you as a company really want to use your own CA for Icinga 2, how to accomplish that? What if the Redis for Icinga DB gets certificates directly from AWS? Maybe Icinga Web should actually be equipped with certificates from Let’s Encrypt? Don’t hesitate to ask our customer support!
