Operations

This section is a loose collection of various topics and external references for running Icinga DB on a day-to-day basis. It covers topics such as self-monitoring, backups, and specifics of third-party components.

Monitor Icinga DB

It is strongly recommended to monitor the monitoring.

There is a built-in icingadb check command in the Icinga 2 ITL. It covers several potential errors, including operations that take too long or invalid high availability scenarios. Even if the Icinga DB has crashed, checks will still run and Icinga 2 would generate notifications.

In addition, both the Redis® and the relational database should be monitored. There are predefined check commands in the ITL to choose from.

• redis
• mysql
• mysql_health
• postgres

A simpler approach would be to check if the processes are running, e.g., with proc or systemd.

Backups

There are only two things to back up in Icinga DB.

1. The configuration file in /etc/icingadb and
2. the relational database, using mysqldump, mariadb-dump or pg_dump.

Warning

When creating a database dump for MySQL or MariaDB with mysqldump or mariadb-dump, use the --single-transaction command line argument flag to not lock the whole database while the backup is running.

Third-Party Configuration

Icinga DB relies on external components to work. The following collection is based on experience. It is a target for continuous improvement.

MySQL and MariaDB

max_allow_packets

The max_allow_packets system variable limits the size of messages between MySQL/MariaDB servers and clients. More information is available in MySQL’s “Replication and max_allowed_packet” documentation section, MySQL’s variable documentation and MariaDB’s variable documentation.

The database configuration should have max_allow_packets set to at least 64M.

Amazon RDS for MySQL

When importing the MySQL schema into Amazon RDS for MySQL, the following may occur.

Error 1419: You do not have the SUPER privilege and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable)

This error can be mitigated by creating and modifying a custom DB parameter group as described in the related AWS Knowledge Center article.

Galera Cluster

Starting with Icinga DB version 1.2.0, Galera support has been added to the Icinga DB daemon. Its specific database configuration is described in the Galera configuration section.

As mentioned in MariaDB’s known Galera cluster limitations, transactions are limited in both amount of rows (128K) and size (2GiB). A busy Icinga setup can cause Icinga DB to create transactions that exceed these limits with the default configuration.

If you get an error like Error 1105 (HY000): Maximum writeset size exceeded and your Galera node logs something like WSREP: transaction size limit (2147483647) exceeded, decrease the values of max_placeholders_per_statement and max_rows_per_transaction in Icinga DB’s Database Options.

Redis®

The official Redis® administration documentation is quite useful regarding the operation of Redis® and is the recommendation for this topic in general. Below, we will address topics specific to Icinga setups.

Redis® requires memory overcommit on Linux

On Linux, enable memory overcommitting.

sysctl vm.overcommit_memory=1

To persist this setting across reboots, add the following line to sysctl.conf(5). If your distribution uses systemd, a configuration file under /etc/sysctl.d/ is required, as described by systemd-sysctl.service(8) and sysctl.d(5).

vm.overcommit_memory = 1

Huge memory footprint and IO usage in large setups

For large setups, the default Redis® configuration is slightly problematic, since Redis® will try to perpetually save its state to the filesystem, a process that gets triggered often enough to make the save process a permanent companion. This will increase the memory usage by a factor of two and also cause a troublesome amount of IO.

As an improvement, Redis® can be configured to produce the dump less often or not at all with the save setting in the configuration. Be warned here, that in case of a crash (of Redis® or the whole machine) all the data after the last dump is lost, meaning that all the events which were not already persisted by Icinga DB or persisted by Redis® are lost then.

To completely disable retention, the save setting must be given an empty argument:

save ""

Alternatively, to reduce the occurrences of dumps, something similar to

save 3600 1 900 100000

can be used. In this example, a dump is performed every hour (3600s) if at least on changes occurred in that time frame and every fifteen minutes (900s) if at least 100,000 changes occurred.