Lots of big companies lock down their IT infrastructure in the internal network, sometimes they even use only locally mirrored repositories. I totally understand this, especially since our CVE-2024-49369. Nowadays, when LLMs find security holes even in OpenBSD, you definitely shouldn’t expose any services to the public without need.
But there’s a disadvantage. Firewalling a service makes it harder to get a regular domain validated certificate, e.g. from Let’s Encrypt. It can’t simply verify your web server because latter is behind your firewall. In the past, sysadmins just bought a TLS certificate once a year and updated their Puppet/Ansible. Unfortunately, since March 15 this year, newly issued certificates must not be valid for longer than 200 days. Next year it will be 100 days and in 2029 even only 47 days. Imagine you have to manually renew all services’ TLS certificates every 47 days…
Luckily, the ACME protocol offers an alternative to the CA directly contacting the service to issue a certificate for. This allows you to fully automate certificate renewal while keeping your internal services protected via firewall. All you need is a dedicated authoritative nameserver, which is reachable (port 53, both UDP and TCP) from both the internet and the internal services to issue certificates for. For demonstration, I’ll use the good old named(8):
[root@aklimov-rhel10-1 ~]# dnf install bind
By the end of this blog post, certbot(7) will communicate with Let’s Encrypt by updating TXT records on that BIND 9 server. Of course, each machine should have write access only to its own records and needs a separate shared secret:
[root@aklimov-rhel10-1 ~]# tsig-keygen -a hmac-sha512 icingaweb2
key "icingaweb2" {
algorithm hmac-sha512;
secret "XK0FGa571yyykKr/2fIc0VNhfTK8lLMRvq3NaILctUQDbkJEDhC2tuYo4j0pKElkTo3bx7zs+BrMaHtAHc/jkg==";
};
[root@aklimov-rhel10-1 ~]#
Append such config snippets from tsig-keygen(8) to /etc/named.conf. Then create a master zone, e.g. _acme-challenge.icingaweb2.aklimov.net-dump.de, and allow the above key to update that TXT record. This zone must be named _acme-challenge. plus the domain of the service. All in all, the necessary modifications to /etc/named.conf look like this:
--- /etc/named.conf
+++ /etc/named.conf
@@ -11,2 +11,2 @@
- listen-on port 53 { 127.0.0.1; };
- listen-on-v6 port 53 { ::1; };
+ listen-on port 53 { any; };
+ listen-on-v6 port 53 { any; };
@@ -19 +19 @@
- allow-query { localhost; };
+ allow-query { any; };
@@ -31 +31 @@
- recursion yes;
+ recursion no;
@@ -59,0 +60,11 @@
+key "icingaweb2" {
+ algorithm hmac-sha512;
+ secret "XK0FGa571yyykKr/2fIc0VNhfTK8lLMRvq3NaILctUQDbkJEDhC2tuYo4j0pKElkTo3bx7zs+BrMaHtAHc/jkg==";
+};
+zone "_acme-challenge.icingaweb2.aklimov.net-dump.de" IN {
+ type master;
+ file "_acme-challenge.icingaweb2.aklimov.net-dump.de.zone";
+ update-policy {
+ grant icingaweb2 name _acme-challenge.icingaweb2.aklimov.net-dump.de. txt;
+ };
+};
The file _acme-challenge.icingaweb2.aklimov.net-dump.de.zone must exist under /var/named and look like this:
$TTL 10
@ IN SOA acmens.aklimov.net-dump.de. mailbox.example.com. 2026041301 300 300 300 300
NS acmens.aklimov.net-dump.de.
I’ve named my nameserver acmens.aklimov.net-dump.de. Yours is up to you, but don’t forget to (re)start it:
[root@aklimov-rhel10-1 ~]# systemctl enable --now named
Now is a good opportunity to test your authoritative nameserver, e.g. 192.0.2.53, via dig(1):
$ dig @192.0.2.53 ns _acme-challenge.icingaweb2.aklimov.net-dump.de
Once everything works, delegate the responsibility for the zone in the parent zone via NS and A records:
acmens.aklimov.net-dump.de. 3600 IN A 192.0.2.53 _acme-challenge.icingaweb2.aklimov.net-dump.de. 3600 IN NS acmens.aklimov.net-dump.de.
Make sure the above query also succeeds against a regular resolver then:
$ dig ns _acme-challenge.icingaweb2.aklimov.net-dump.de
Now BIND is ready to mediate between your preferred CA and Icinga Web 2. Let’s quickly setup the latter for demonstration purposes:
[root@aklimov-rhel10-2 ~]# curl --fail --silent --show-error --location \ --output /etc/yum.repos.d/ICINGA-release.repo \ https://packages.icinga.com/subscription/rhel/ICINGA-release.repo
Set user and password:
[root@aklimov-rhel10-2 ~]# vim /etc/yum.repos.d/ICINGA-release.repo
[root@aklimov-rhel10-2 ~]# dnf install icingaweb2 mod_ssl
[root@aklimov-rhel10-2 ~]# systemctl enable --now php-fpm httpd
Yet, curl(1) would complain about a “self-signed certificate in certificate chain”. To fix this, install certbot(7) from the EPEL repository first:
[root@aklimov-rhel10-2 ~]# dnf install epel-release
[root@aklimov-rhel10-2 ~]# dnf install certbot \ python3-certbot-apache python3-certbot-dns-rfc2136
The Certbot needs only two things to operate properly: credentials for updating the DNS record and a virtual host to attach the obtained certificate to. The DNS credentials have been generated by tsig-keygen(8) above and /etc/letsencrypt/rfc2136.ini is a good place for them:
dns_rfc2136_server = 192.0.2.53 dns_rfc2136_name = icingaweb2 dns_rfc2136_secret = XK0FGa571yyykKr/2fIc0VNhfTK8lLMRvq3NaILctUQDbkJEDhC2tuYo4j0pKElkTo3bx7zs+BrMaHtAHc/jkg== dns_rfc2136_algorithm = hmac-sha512
The desired virtual host can only be found if named correctly:
--- /etc/httpd/conf.d/ssl.conf +++ /etc/httpd/conf.d/ssl.conf @@ -44 +44 @@ <VirtualHost _default_:443> -#ServerName www.example.com:443 +ServerName icingaweb2.aklimov.net-dump.de
And that’s all – everything is ready for the first valid certificate:
[root@aklimov-rhel10-2 ~]# certbot run --noninteractive --agree-tos \ --email mailbox@example.com \ --domain icingaweb2.aklimov.net-dump.de \ --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/rfc2136.ini \ --installer apache
[root@aklimov-rhel10-2 ~]# systemctl restart httpd
Now curl says: “SSL certificate verify ok.” 🎉 To keep it this way, I recommend enabling automatic renewal and httpd(8) restart:
[root@aklimov-rhel10-2 ~]# systemctl enable --now certbot-renew.timer
--- /etc/sysconfig/certbot +++ /etc/sysconfig/certbot @@ -29 +29 @@ -POST_HOOK="" +POST_HOOK="--post-hook 'systemctl restart httpd'"
And for more on using Let’s Encrypt certificates for Icinga, I also recommend this post of mine.
