Configuration¶
The configuration for Icinga Notifications is twofold. The main configuration resides in the database, shared between the Icinga Notifications daemon and Icinga Notifications Web. However, as the Icinga Notifications daemon needs to know how to access this database and some further settings, it needs its own configuration as well.
This configuration may be a YAML file, environment variables, or both. Environment variables take precedence and override previously defined values from the configuration file.
The YAML configuration file is stored in /etc/icinga-notifications/config.yml.
See config.example.yml for an example configuration.
The following subsections describe the configurations of the various modules.
For the YAML configuration file, each option is written in lowercase, as shown in the tables.
When using environment variables, the variable name is constructed by concatenating ICINGA_NOTIFICATIONS_,
the module name in uppercase followed by an underscore, and the option name in uppercase.
The hyphens in the names are to be replaced by underscores.
For example, to set the database host, the ICINGA_NOTIFICATIONS_DATABASE_HOST environment variable is used.
Passwords can be set directly or stored in a separate file, referenced via the password_file YAML key or PASSWORD_FILE environment variable.
Only one of these two options can be used.
Top Level Configuration¶
For YAML configuration, these options are on the top level, not part of a dictionary.
For environment variables, each option is prefixed with ICINGA_NOTIFICATIONS_.
Icinga Web 2¶
The icingaweb2_url is expected to point to the base directory of your Icinga Web 2 installation,
i.e., https://example.com/icingaweb2/, to be used for URL creation.
Channels Directory¶
All available Icinga Notifications channels should reside in the channels_dir directory.
For a package installation, the default will point to the correct location and must not be changed.
This directory should be /usr/libexec/icinga-notifications/channels on systems that follow the Filesystem Hierarchy Standard.
It may also be /usr/lib/icinga-notifications/channels, depending on the operating system conventions.
HTTP API Configuration¶
Configuration of the HTTP API listener for event submission and debugging endpoints.
Icinga Notifications can run a TCP listener, a Unix socket listener, or both simultaneously.
When neither address nor socket is configured, the TCP listener defaults to localhost:5680.
TLS options apply only to the TCP listener.
When a Unix socket is configured, Icinga Notifications identifies connecting processes by their OS user, and matches the OS username against the source’s configured username. No password or HTTP Basic Auth is involved.
For YAML configuration, the options are part of the listener section.
For environment variables, each option is prefixed with ICINGA_NOTIFICATIONS_LISTENER_.
| Option | Description |
|---|---|
| address | Optional. TCP address to bind to. Defaults to localhost:5680 when socket is not set. |
| socket | Optional. Path to a Unix domain socket for local event submission. |
| socket_mode | Optional. Permission bits for the Unix socket file, as an octal. Defaults to 0660. |
| socket_group | Optional. OS group to assign to the Unix socket file. Defaults to Icinga Notifications’ primary group. |
| debug_password | Password expected via HTTP Basic Authentication for debug endpoints. |
| debug_password_file | debug_password in a file. |
| tls | Optional. Whether to require TLS for the TCP listener. Defaults to false. |
| cert | Optional. Path to TLS server certificate. Required if tls is enabled. |
| key | Optional. Path to the TLS private key. Required if tls is enabled. |
| ca | Optional. Path to TLS CA cert/bundle to verify client certs. Required if tls is enabled. |
Important
The socket permissions restricts who can access the Unix socket.
A process can only submit events for sources whose configured listener_username matches the process’s OS username.
So each source requires its own OS user; set 0600 (owner-only) would limit the socket to a single source.
Database Configuration¶
Connection configuration for the database where Icinga Notifications stores configuration and historical data. This is also the database used in Icinga Notifications Web to view and work with the data.
For YAML configuration, the options are part of the database dictionary.
For environment variables, each option is prefixed with ICINGA_NOTIFICATIONS_DATABASE_.
| Option | Description |
|---|---|
| type | Optional. Either mysql (default) or pgsql. |
| host | Required. Database host or absolute Unix socket path. |
| port | Optional. Database port. By default, the MySQL or PostgreSQL port, depending on the database type. |
| database | Required. Database name. |
| user | Required. Database username. |
| password | Optional. Database password. |
| password_file | Optional. Database password file. |
| tls | Optional. Whether to use TLS. |
| cert | Optional. Path to TLS client certificate. |
| key | Optional. Path to TLS private key. |
| ca | Optional. Path to TLS CA certificate. |
| insecure | Optional. Whether not to verify the peer. |
| options | Optional. List of low-level database options that can be set to influence some Icinga Notifications internal default behaviours. |
Database Options¶
Each of these configuration options are highly technical with thoroughly considered and tested default values that you should only change when you exactly know what you are doing. You can use these options to influence the Icinga Notifications default behaviour, how it interacts with databases, thus the defaults are usually sufficient for most users and do not need any manual adjustments.
Important
Do not change the defaults if you do not have to!
For YAML configuration, the options are part of the database.options dictionary.
For environment variables, each option is prefixed with ICINGA_NOTIFICATIONS_DATABASE_OPTIONS_.
| Option | Description |
|---|---|
| max_connections | Optional. Maximum number of database connections Icinga Notifications is allowed to open in parallel if necessary. Defaults to 16. |
| max_connections_per_table | Optional. Maximum number of queries Icinga Notifications is allowed to execute on a single table concurrently. Defaults to 8. |
| max_placeholders_per_statement | Optional. Maximum number of placeholders Icinga Notifications is allowed to use for a single SQL statement. Defaults to 8192. |
| max_rows_per_transaction | Optional. Maximum number of rows Icinga Notifications is allowed to SELECT,DELETE,UPDATE or INSERT in a single transaction. Defaults to 8192. |
| wsrep_sync_wait | Optional. Enforce Galera cluster nodes to perform strict cluster-wide causality checks. Defaults to 7. |
Retention Configuration¶
You can configure the data retention and cleanup behavior of Icinga Notifications to automatically remove old data from the database after a certain period. This helps to manage the size of the database and ensure that it does not grow indefinitely.
For YAML configuration, the options are part of the retention dictionary.
For environment variables, each option is prefixed with ICINGA_NOTIFICATIONS_RETENTION_.
| Option | Description |
|---|---|
| period | Optional. Retention period defined as duration string. Defaults to 0s (disabled). Data older than this period will be automatically deleted from the database. |
| interval | Optional. Interval for periodic retention checks defined as duration string. Defaults to 1h. |
| batch_size | Optional. Maximum number of rows to delete in a single transaction during retention cleanup. Defaults to 5000. |
| options | Optional. Map of component name to retention period in order to set a different retention period for each component instead of the default one. See retention components for details. |
Info
By default, Icinga Notifications does not delete any data from the database, since the default retention period is
0s, which means that all data is retained indefinitely. You can set the period option to a non-zero value to
enable automatic data cleanup after the specified period has passed.
If you don’t want to enable automatic data cleanup for all components, you can let the period option be 0s and set
non-zero retention periods for individual components in the options as described below.
This goes also for the other way around, if you want to enable automatic data cleanup for all components, you can set
the period option to a non-zero value and set 0s retention periods for a specific component to disable automatic
data cleanup for that component.
Also note that Icinga Notifications may still keep cleaning up data of some components that can’t be influenced by the
retention configuration, such as the object table and its related tables, which are cleaned up automatically when
there are no more references to them from the incident table. This is necessary to avoid unnecessarily bloating
the database with orphaned data that is no longer relevant to any incident.
Retention Components¶
In addition to the default retention period, you can set a different retention period for each component of Icinga Notifications. This allows you to keep certain types of data for a longer or shorter period than others, depending on their importance or relevance.
For YAML configuration, the options are part of the retention.options dictionary.
For environment variables, each option is prefixed with ICINGA_NOTIFICATIONS_RETENTION_OPTIONS_ and expects
a duration string as value.
Currently, the following components are available:
| Component | Description |
|---|---|
| incident | Incidents and all related data, such as their history, contacts, etc. |
Info
Incidents that are still open are not eligible for retention cleanup, even if they are open for months or years. The retention policy only applies to closed incidents, and the retention period is computed from the time they were closed, not from the time they were opened. If you want to clean up some incidents that are still open too, you need to close them first, and then wait for the retention period to pass.
Logging Configuration¶
Configuration of the logging component used by Icinga Notifications.
For YAML configuration, the options are part of the logging dictionary.
For environment variables, each option is prefixed with ICINGA_NOTIFICATIONS_LOGGING_.
| Option | Description |
|---|---|
| level | Optional. Specifies the default logging level. Can be set to fatal, error, warn, info or debug. Defaults to info. |
| output | Optional. Configures the logging output. Can be set to console (stderr) or systemd-journald. Defaults to systemd-journald when running under systemd, otherwise to console. See notes below for systemd-journald. |
| interval | Optional. Interval for periodic logging defined as duration string. Defaults to "20s". |
| options | Optional. Map of component name to logging level in order to set a different logging level for each component instead of the default one. See logging components for details. |
Logging Components¶
The independent components of Icinga Notifications produce log entries. Each log entry is linked to its component and a log level.
By default, any log message will be displayed if its log level is at or above the level configured above.
However, it is possible to override the log level for each component individually to show more or less information.
For YAML configuration, the options are part of the logging.options dictionary.
For environment variables, ICINGA_NOTIFICATIONS_LOGGING_OPTIONS expects a single string of component:level pairs joined with ,.
The following example would log everything with at least info level, except database and listener entries, where the level is one time raised and one time lowered.
# YAML Configuration File
logging:
level: info
options:
database: error
listener: debug
# Environment Variables
ICINGA_NOTIFICATIONS_LOGGING_LEVEL=error
ICINGA_NOTIFICATIONS_LOGGING_OPTIONS=database:error,listener:debug
| Component | Description |
|---|---|
| channel | Notification channels, their configuration and output. |
| database | Database connection status and queries. |
| incident | Incident management and changes. |
| event-queue | Event queue handles events enqueued in the database. |
| listener | HTTP listener for event submission and debugging. |
| retention | Data retention and cleanup. |
| runtime-updates | Configuration changes through Icinga Notifications Web from the database. |
Appendix¶
Duration String¶
A duration string is a sequence of decimal numbers and a unit suffix, such as "20s".
Valid units are "ms", "s", "m" and "h".
Galera Cluster¶
Icinga Notifications expects a more consistent behaviour from its database than a
Galera cluster provides by default. To accommodate this,
Icinga Notifications sets the wsrep_sync_wait system
variable for all its database connections. Consequently, strict cluster-wide causality checks are enforced before
executing specific SQL queries, which are determined by the value set in the wsrep_sync_wait system variable.
By default, Icinga Notifications sets this to 7, which includes READ, UPDATE, DELETE, INSERT, REPLACE query types and is
usually sufficient. Unfortunately, this also has the downside that every single Icinga Notifications query will be blocked until
the cluster nodes resynchronise their states after each executed query, and may result in degraded performance.
However, this does not necessarily have to be the case if, for instance, Icinga Notifications is only allowed to connect to a
single cluster node at a time. This is the case when a load balancer does not randomly route connections to all the
nodes evenly, but always to the same node until it fails, or if your database cluster nodes have a virtual IP address
fail over assigned. In such situations, you can set the wsrep_sync_wait system variable to 0 in the
/etc/icinga-notifications/config.yml file to disable it entirely, as Icinga Notifications doesn’t have to wait for cluster
synchronisation then.
Systemd Journald Fields¶
When examining the journal with journalctl, fields containing additional information are hidden by default.
Setting an appropriate
--output option
will include them, such as: --output verbose or --output json.
For example:
journalctl --unit icinga-notifications.service --output verbose
All Icinga Notifications fields are prefixed with ICINGA_NOTIFICATIONS_,
e.g., ICINGA_NOTIFICATIONS_ERROR for error messages.