Monitor all operating systems with one solution: Icinga 2

by | Aug 16, 2023

We as a company build monitoring software. And we have committed to diversity. It is just logical and consequent for us to apply this principle not only to the people who do the work, but also to the work itself. To the monitoring software we build. Especially to Icinga 2 which, in a perfectly monitored environment, runs on every single machine. I.e. on every single OS powering all those machines. We already officially support and (of course) package several major Linux distributions expected to be found in production.

But Linux isn’t the only platform, not even the only one of its kind, the *nix family. E.g. FreeBSD is also used by several large companies. Admittedly my personal favourite OS, OpenBSD, is less popular, but a public user list includes e.g. the deNIC eG. Not too bad. While we don’t have the resources for official support (not to mention packaging) of those two, we as developers can ease the proper functionality of our software on the BSD family as follows.

Those OSes maintain so-called “ports”, one large VCS repository with scripts and patch files for building packages of additional software beyond the base system. Its maintainers are responsible for making and keeping all of it working in the first place as they know their OS better than developers of ported add-on software. But on the other hand the main developers of a software know it better than maintainers of any OS.

For this reason, if a port maintainer hits any problems they can’t resolve by themselves, they ask us. From simple compile errors (FreeBSD, 2012) to inadequate signal(7) handling due to hidden origin PID (OpenBSD, 2022). This is the most important help we can provide. E.g. in both of the above cases we’ve fixed our code base and the maintainer could even integrate the diff(1) into the ports until a release of the first Icinga 2 version containing the fix.

But we as developers can do even more and save more of those people’s time. We can avoid such errors proactively, even before a change set is merged. The most simple case is the usage of yet another function listed in syscalls(2) for the first time. And the solution is even simpler. We can ask our preferred search engine for “man” (for manual) and the system call name together with the OS we’d like not to break. A good search engine will yield the manual page (if any) among the first few hits. And a good developer will check it for the desired use case.

Unfortunately(?) manpages don’t specify everything, especially not all implementation details. E.g. in Icinga 2.14 we’ve repaired the missing last failed reload time on *nix. To achieve that we needed to store it in memory shared across all Icinga 2 processes. To make sure the shared memory “I/O” behaves as desired no matter the platform we required lock-free atomic floating point numbers. Admittedly a very special feature not necessarily easy to look up. In such cases it can be much easier to “just compile” Icinga with the desired changes on the OS in question.

I actually did that. The following instructions for FreeBSD and OpenBSD are mainly a cheat sheet for me and my colleagues. But an adapted version may help you as well. Both OSes (once installed) share the same four steps: get the ports tree, change the “version” of Icinga being built in the corresponding Makefile (make a backup!), start the build process and adjust (mostly just shorten) patch files if necessary. Once the latter is done, run make clean, start the build process again and so on… The target version of Icinga should be in a public GitHub repository, at best in a branch without too special characters, even slashes. The following examples use the master branch. Admittedly this is a bit too late, so please adjust as needed.

Compile Icinga on FreeBSD

Get the ports tree

Normally admins shall not(!) install any pre-built packages if they’re directly using ports and vice-versa. But FreeBSD tracks its ports tree in a Git repository and Git isn’t part of the FreeBSD base system. So we need to make an exception and install Git via pkg(8) before cloning the ports tree itself. (Not to mention the pkg package itself which is automatically installed by pkg(8) in order to have a proper pkg command. Yes, pkg is a package manager installed by the pkg package manager. 👻)

pkg install git
git clone https://git.FreeBSD.org/ports.git /usr/ports
cd /usr/ports/net-mgmt/icinga2
git checkout `git branch -a | grep Q | tail -1 | cut -d / -f 3-`

Change the Icinga version in its Makefile

Compared to the original it should look more or less like this:

$ diff -U3 Makefile.orig Makefile
--- Makefile.orig       2023-06-12 16:10:26.816377000 +0000
+++ Makefile    2023-06-12 16:19:16.190076000 +0000
@@ -1,6 +1,6 @@
 PORTNAME=      icinga2
 DISTVERSIONPREFIX=     v
-DISTVERSION=   2.13.6
+DISTVERSION=   2.14.0
 PORTREVISION=  0
 CATEGORIES=    net-mgmt

@@ -18,8 +18,9 @@
 RUN_DEPENDS=   ${LOCALBASE}/bin/bash:shells/bash

 USES=          alias bison cmake:insource compiler:c++0x libedit ssl
-USE_GITHUB=    yes
-GH_ACCOUNT=    icinga
+MASTER_SITES = https://github.com/Icinga/icinga2/archive/refs/heads/
+DISTNAME = master
+WRKSRC = ${WRKDIR}/icinga2-master

 USE_LDCONFIG=  yes

$

Start the build process

make makesum
env BATCH=yes make

make(1) first applies all patches to Icinga’s dependencies and Icinga itself and then compiles those. Due to this fact you can put the VM/SSH session into the background once a compilation process is actually started as the patching is already done. Apropos, the patches are in ./files/patch-* and in the Makefile itself starting with “PATCH”. The latter are probably only referring to commits merged into Icinga master with build fixes and can be removed.

Compile Icinga on OpenBSD

At least one of the partitions involved in building the ports must be mounted with the wxallowed option. Either adjust all of them in /etc/fstab or make one large partition during installation.

Get the ports tree

In contrast to FreeBSD, the OpenBSD base system already provides everything needed to download and verify the ports collection:

ftp $(cat /etc/installurl)/$(uname -r)/SHA256.sig
ftp $(cat /etc/installurl)/$(uname -r)/ports.tar.gz
signify -C -p /etc/signify/openbsd-$(uname -r|tr -d .)-base.pub -x SHA256.sig ports.tar.gz
cd /usr
tar -xzf /root/ports.tar.gz
cd ports/net/icinga/core2

Change the Icinga version in its Makefile

Compared to the original it should look more or less like this:

$ diff -U3 Makefile.orig Makefile
--- Makefile.orig       Mon Jun 12 09:32:46 2023
+++ Makefile    Mon Jun 12 09:40:54 2023
@@ -2,10 +2,11 @@
 COMMENT-mysql =        MySQL support for icinga2
 COMMENT-pgsql =        PostgreSQL support for icinga2

-V =            2.13.7
+V =            2.14.0

-GH_PROJECT =   icinga2
-GH_TAGNAME =   v$V
+MASTER_SITES = https://github.com/Icinga/icinga2/archive/refs/heads/
+DISTNAME = master
+WRKSRC = ${WRKDIR}/icinga2-master
 PKGNAME-main = icinga2-$V
 PKGNAME-mysql =        icinga2-ido-mysql-$V
 PKGNAME-pgsql =        icinga2-ido-pgsql-$V
$

Start the build process

make makesum
make

In contrast to FreeBSD, OpenBSD patches and builds the dependency tree leaf by leaf. So it will build all dependencies and then maybe fail on outdated patches. Those are by the way located in ./patches/.

Conclusion

We’re hiring! Your religion doesn’t matter, your anatomy doesn’t matter and your preferences don’t matter. Your favourite operating system doesn’t either – not to mention the text editor of your choice.

PS: Oh, almost forgot. There’s also Windows. But this is another story.

You May Also Like…

Icinga 2 API and debug console

Icinga 2 API and debug console

Have you ever experienced configuration issues, such as notifications not being sent as expected or apply rules not...

Subscribe to our Newsletter

A monthly digest of the latest Icinga news, releases, articles and community topics.