Releasing Icinga DB v1.0-RC2

We are super excited to release the second Release Candidate of Icinga DB! This release comes after many hours, days and months of experimenting, re-thinking and rebuilding our own code and marks a huge step towards a new data backend for Icinga.

 

Why Icinga DB

Since the early days of Icinga (first Release was in 2009) we rely on the Icinga Data Output, in short IDO. The IDO is the feature that stores the whole Icinga configuration into a database and updates the monitoring objects accordingly with the collected monitoring data. Basically, the IDO writes everything into the database that Icinga 2 is aware of. The data is then used to be displayed in Icinga Web.

While this approach worked out very well for many years, the amount of servers, virtual machines, services and applications our customers and users monitor with Icinga has increased significantly through the past years. For very huge environments, the IDO can be a performance bottleneck. With Icinga DB we enable users to monitor massive amounts of data.

 

A little bit of history

For our beloved community it hasn’t been a secret that we’ve been working on a new data backend for Icinga for a couple of years now. In the early days we experimented with many different types of databases: relational databases, no-sql databases, document based databases and any other different way of storing data efficiently. We finally came up with the approach of separating volatile monitoring data with data that is actually relevant to understand the behaviour and history of hosts and applications. We decided to continue with a combination of the in-memory database Redis and a traditional relational database (MySQL or PostgreSQL). In March 2020 we released the first Release Candidate of Icinga DB (v1.0-RC1), which turned out to be the right decision later.

With the support of many users who helped in testing we figured out that our approach was working in general, but we needed to change the whole code for a better implementation of our ideas. Throwing away thousands of lines of code is one of the hardest decisions that you can make in software development. Still believing in our ideas and experiments allowed us to do so, and we started from scratch in December 2020. The now available RC2 of Icinga DB is the result of all the empirical values we established through the whole journey.

With Icinga DB the whole monitoring configuration is still stored in MySQL/PostgreSQL. The main difference between the IDO is that monitoring data that is not relevant for history purposes is only stored in Redis. Let’s say you have 100 service checks, and 95 of them return the state OK all of the time, because your services are working as expected. All of these OK states are not relevant for you to understand at which point the state of a service actually changed. This is how we separate the data, roughly speaking.

Extrapolating this scenario to the real-world means that you most likely have hundreds of thousands of hosts and services. The amount of updates to the relational database we save by leveraging Redis is enormous, and influences the overall performance significantly. Still, the most current monitoring results are displayed on the Icinga web interface, since it connects also to Redis in addition to the relational database.

To stretch the performance even further, with Icinga DB we parallelize the updates to the relational database. By default we use 16 connections to MySQL, which is only possible because we created a completely new database schema that allows Icinga to execute multiple updates at the same time, without having to wait for foreign entries.

 

Exclusive Icinga DB Features

Besides the improved performance, a new data backend gives us the chance to build long desired features. We took the chance to include some new features on the backend side, but also enhance the whole web interface that visualises the monitoring data.

The Icinga DB daemon handles high availability. This means that the daemon can be started on multiple nodes. The instances will communicate with each other through the connected database and switch over if one side is not working properly. Additionally, we already include “environments” in our database schema. The schema therefore is prepared to store the data from multiple Icinga installations. While this is not fully supported in the web interface yet, the schema allows us to unite multiple Icinga environments into one interface in the future – which in turn leads to a complete new and different way of scaling Icinga.

We also took the chance to consider SLA reporting from the beginning. The new schema supports this out of the box, which makes it much easier for us to create SLA reports but also many other different types of analytics.

The new web interface (Icinga DB Web) is a separate module for Icinga Web, but will be shipped by default once the final v1.0 is ready. It includes a bunch of changes making the whole web interface more user-friendly. We reworked the detail view for hosts and services. While it still displays similar data compared to the current “monitoring” module, the way they are display differs a lot.

 

The additional Source tab within the detail view of an object displays very detailed data about the check execution. Additionally to the runtime information, you also see the exactly executed command to reproduce it easier and find out if the correct parameters are set.

 

With the new view mode switcher you can switch between three different views when looking at lists of items. It covers the requirements of different Icinga users and allows you to see your favourite amount of data.

Executing actions such as creating downtimes or setting acknowledgments are now executed within modals. From the UI perspective this helps a lot to stay focused on whatever you are currently doing.

The new and completely rewritten filter editor enables to you search efficiently for certain objects. This comes in handy especially when working with large environments, but also makes it fairly easy to create special dashlets for your dashboard. The new filter editor has a strong focus the user experience and combines powerful functionality with a simple user interface. Especially with the new autocomplete functionality you are much faster when searching for certain filters but also host and service names. Grouping may be changed with drag and drop.

There are plenty of more changes in the web interface, such as new visualisation of downtimes and check executions. Listing all of them here would be going to far. We highly recommend to try it out yourself, the installation of Icinga DB is fairly easy.

 

Get started

We provide packages for Icinga DB for all common Linux distributions. Besides of our own components, we provide packages for Redis 6 as well. This makes it easier for you to get started, since you don’t have to bother about the right version and configuration. Redis 6 comes with native TLS support, so distributing the setup can be achieved in a secure way.

Icinga DB requires the latest version of Icinga 2 (v2.13.2) and the latest version of Icinga Web (v2.9.5). We published the Icinga DB packages within our “testing” repositories. The first step is to include the “testing” repository in addition to the regular repositories. Here’s an example on how to install Icinga DB on Ubuntu 20.04:

 

Add the testing repository

echo 'deb https://packages.icinga.com/ubuntu icinga-focal main' > /etc/apt/sources.list.d/focal-icinga.list

echo 'deb https://packages.icinga.com/ubuntu icinga-focal-testing main' > /etc/apt/sources.list.d/focal-icinga-testing.list

apt update

Install Icinga DB

apt install icingadb icingadb-redis icingadb-web

Import MySQL Schema (requires a blank database)

mysql -u root -D icingadb -p < /usr/share/icingadb/schema/mysql/mysql.schema.sql

Remember to set the correct database credentials within /etc/icingadb/config.yml

Run Icinga DB

systemctl enable icingadb
systemctl enable icingadb-redis

systemctl start icingadb
systemctl start icingadb-redis

Enable the Icinga DB Writer

icinga2 feature enable icingadb

systemctl restart icinga2

 

For detailed installation instructions please refer to the official Icinga DB documentation. You do not need to change anything on your current configuration or enabled modules. Icinga DB can be installed in parallel with the “monitoring” module, in that case you will see a separate “Icinga DB” menu entry. If you disable the monitoring module, Icinga DB will become the primary view for all menu entries.

What’s next

Releasing this version of Icinga DB is a big milestone for us. Through all the testing we’ve done internally we used custom generated configurations but also real-world user configurations we collected from our customers. We also received valuable feedback provided by early adopters. We are very happy with the results now and pretty sure that there won’t be any bigger changes anymore. Nevertheless we will wait for more feedback before we publish the final release to be used in production.

In the meantime we are working on finishing some other important features that did not made it into this Release Candidate. There’s the support for PostgreSQL which we did not include into this release, even though it’s very close to be finished. Additionally we’ve been working on a migration tool, to copy historical data from an existing IDO database into the new Icinga DB schema. Our tests look very good, but it’s not finished yet. Additionally there’s the housekeeping topic which needs some polishing before we can mark it as stable. For the final v1.0 release we’re aiming to deliver the correct schema for SLA reports as well.

Icinga DB is more than just a new data backend. It is the foundation for features that many user wished for and that we could not build. It includes many libraries that we will use in the future for new projects. And it marks the first step towards a faster Icinga 2 core, by moving certain tasks into dedicated daemons.

While we’re excited to release this version, we’re at the same time just as excited to get your feedback and make the final version the best it can be!