Icinga 2 Release News: 2.1.0 is out!

icinga_bannerThe new release 2.1.0 adds new features to Icinga 2 – all thanks to your great feedback on the 2.0 releases. You’ll find plenty of documentation updates (improved “Getting started” and “Distributed Monitoring and High Availability” documentation) and also package enhancements (mostly SUSE related, as Icinga 2 hit their repos lately).
Other than that, this release is all about:

  1. Making it even easier to work with the Icinga 2 configuration format
  2. Better application logging
  3. Advanced high availability cluster functionality

On the side, we’ve continued to polish up docs.icinga.com and streamline the Icinga Web 2 Docs viewer, freshen up our dev.icinga.com, update the Icinga Vagrant Demo Boxes and further develop Ansible and Puppet modules for Icinga 2 (announcements on the latter shortly).
Download Icinga 2 and upgrade now. Don’t forget to update the DB IDO schema – find details on the Changelog.
PS: I’m going to San Francisco for the Icinga Camp too. Join us with your talk on Icinga and have a beer (or two) with us 🙂

Configuration Enhancements

icinga2_list_objectsThe icinga2-list-objects script does not only show all objects (also those generated from apply rules), it also enables you to use one of the coolest features we’ve ever implemented: All object attributes will show their modification origin (templates, dictionary add/remove, etc) including the file name, line number, character from-to. The curious can read on in Gunnar’s recent post on “Analysing configuration problems with Icinga 2“.

# icinga2-list-objects --color | less -R

We’ve also added an application search path for the default icinga2.conf file – this enables shorter config validation, and helps tests and debug runs too.

# icinga2 -C

Our newest core team member Carlos did some great work on the optional Check Command plugins for the Manubulon SNMP plugins which are now integrated into Icinga 2. Details on the documentation – in short:

# vim /etc/icinga2/constants.conf
const ManubulonPluginDir = PluginDir
# vim /etc/icinga2/icinga2.conf
include <manublon>

Thanks Carlos!

Error Logging

icinga2_error_loggingWhen a command fails to execute you would normally want to see why. As we feel the same way about it, we’ve changed all failed commands to log severity warning. There are similar enhancements in other areas where logging was unclear – for example when the performance data parser fails to collect metrics for Graphite (there’s a discussion on wrong locales vs plugin api still in progress here).
I’ve set a wrong path for the PluginDir constant in constants.conf for this example.

Cluster HA

icinga2_cluster_icingaweb2_vagrantThe DB IDO feature offers an integrated ‘high availability‘ mode. As such it ran on the zone master in a HA zone setup, which confused users. We still think that a shared database using a virtual IP address is the best default setting, but you can choose to disable the HA functionality on all nodes by editing the DB IDO config like this:

object IdoMysqlConnection "ido-mysql" { enable_ha = false }

The DB IDO programstatus table requires the additional endpoint_name column to ensure that there’s only one active node populating the database, even in a split-brain failure scenario in an HA zone.
The high availability functionality has been added to the notification feature but in a different fashion. Similar to check load balancing inside a cluster zone, notifications are now load balanced too. If you prefer to maintain redundancy when sending notifications, set enable_ha = false in the NotificationComponent configuration (details on the documentation).

2.1.0 Changelog

Find details on the issues on https://dev.icinga.com/versions/199

  • DB IDO schema upgrade required! New schema version: 1.11.7
  • RPMs install the schema files into `/usr/share/icinga2-ido*` instead of `/usr/share/doc/icinga2-ido*` #6881
  • Information for config objects using `icinga2-list-objects` script #6702
  • Add Python 2.4 as requirement #6702
  • Add search path: If `-c /etc/icinga2/icinga2.conf` is omitted, use `SysconfDir + “/icinga2/icinga2.conf”` #6874
  • Change log level for failed commands #6751
  • Notifications are load-balanced in a High Availability cluster setup (New config attribute: `enable_ha`) #6203
  • DB IDO “run once” or “run everywhere” mode in a High Availability cluster setup (New config attributes: `enable_ha` and `failover_timeout`) #6203 #6827
  • RPMs use the `icingacmd` group for /var/{cache,log,run}/icinga2 #6948


  • Bug #6881: make install does not install the db-schema
  • Bug #6915: use _rundir macro for configuring the run directory
  • Bug #6916: External command pipe: Too many open files
  • Bug #6917: enforce /usr/lib as base for the cgi path on SUSE distributions
  • Bug #6942: ExternalCommandListener fails open pipe: Too many open files
  • Bug #6948: check file permissions in /var/cache/icinga2
  • Bug #6962: Commands are processed multiple times
  • Bug #6964: Host and service checks stuck in “pending” when hostname = localhost a parent/satellite setup
  • Bug #7001: Build fails with Boost 1.56
  • Bug #7016: 64-bit RPMs are not installable
  • Feature #5219: Cluster support for modified attributes
  • Feature #6066: Better log messages for cluster changes
  • Feature #6203: Better cluster support for notifications / IDO
  • Feature #6205: Log replay sends messages to instances which shouldn’t get those messages
  • Feature #6702: Information for config objects
  • Feature #6704: Release 2.1
  • Feature #6751: Change log level for failed commands
  • Feature #6874: add search path for icinga2.conf
  • Feature #6898: Enhance logging for perfdata/graphitewriter
  • Feature #6919: Clean up spec file
  • Feature #6920: Recommend related packages on SUSE distributions
  • API – Bug #6998: ApiListener ignores bind_host attribute
  • DB IDO – Feature #6827: delay ido connect in ha cluster
  • Documentation – Bug #6870: Wrong object attribute ‘enable_flap_detection’
  • Documentation – Bug #6878: Wrong parent in Load Distribution
  • Documentation – Bug #6909: clarify on which config tools are available
  • Documentation – Bug #6968: Update command arguments ‘set_if’ and beautify error message
  • Documentation – Bug #6995: Keyword “required” used inconsistently for host and service “icon_image*” attributes
  • Documentation – Feature #6651: Migration: note on check command timeouts
  • Documentation – Feature #6703: Documentation for zones and cluster permissions
  • Documentation – Feature #6743: Better explanation for HA config cluster
  • Documentation – Feature #6839: Explain how the order attribute works in commands
  • Documentation – Feature #6864: Add section for reserved keywords
  • Documentation – Feature #6867: add section about disabling re-notifications
  • Documentation – Feature #6869: Add systemd options: enable, journal
  • Documentation – Feature #6922: Enhance Graphite Writer description
  • Documentation – Feature #6949: Add documentation for icinga2-list-objects
  • Documentation – Feature #6997: how to add a new cluster node
  • Documentation – Feature #7018: add example selinux policy for external command pipe
  • Plugins – Feature #6650: Plugin Check Commands: add manubulon snmp plugins

Analysing configuration problems with Icinga 2

In order to make it even easier to find configuration mistakes we’re going to provide a new tool as part of the upcoming Icinga 2.1 release:

# icinga2-list-objects
Object 'api' of type 'ApiListener':
  * templates = ['api']
    % modified in /etc/icinga2/features-enabled/api.conf, lines 5:1-11:1
  * bind_port = '5665'
    % modified in /etc/icinga2/features-enabled/api.conf, lines 10:3-10:20
  * __name = 'api'
  * ca_path = '/etc/icinga2/pki/ca.crt'
    % modified in /etc/icinga2/features-enabled/api.conf, lines 8:3-8:46
  * cert_path = '/etc/icinga2/pki/ztv.beutner.name.crt'
    % modified in /etc/icinga2/features-enabled/api.conf, lines 6:3-6:62
  * type = 'ApiListener'
  * key_path = '/etc/icinga2/pki/ztv.beutner.name.key'
    % modified in /etc/icinga2/features-enabled/api.conf, lines 7:3-7:61

The “icinga2-list-objects” tool lists all configuration objects. This includes objects which have been created using “apply” rules and should help with diagnosing whether your rules work as you expect them to.
In addition to the objects we’re also listing all attributes and their values. Icinga 2 also keeps track of where exactly each attribute was modified.

Icinga 2 0.0.10 released

Download_Icinga2v0.0.10The more we’ve been playing with Icinga 2 in our test labs, the better insight we’ve gotten into how the configuration works in “real life”. One of the outcomes from our testing was already addressed in 0.0.9, which introduced the new apply rules for objects.
A new configuration format would not be justified in a new major release (2.0.0 is near), if we didn’t tinker a bit more and try to make Icinga 2 even more satisfying to use. For a detailed list on all config-related changes check #5909.
Find the change log at docs.icinga.com – Icinga 2 0.0.10 is available for download.

More ‘Apply’ Goodness

The new apply rules are really sexy – they now generate errors and warnings if used in the wrong scope too (#5911). Furthermore a new apply target type was introduced (#5924, for example when applying a notification object to a host or service object).

apply Notification "mail-icingaadmin" to Service {
  import "mail-service-notification"
  user_groups = [ "icingaadmins" ]
  assign where "generic-service" in service.templates

Apply rules generate new objects each time (services linked to hosts for example) – if you want to assign members to a group based on such a rule set (#5910), you cannot use the ‘apply’ keyword. Instead, you’ll just use the ‘assign’ and ‘ignore’ rules within group objects to assign all matching objects a membership to this group (and all nested groups too). Nested groups for better interface grouping or web interface permissions can be defined using the additional group object attribute ‘groups’.
The example below shows a host with the custom attribute ‘os’ being set to ‘Linux’. The host group ‘linux-servers’ matches that pattern, and assigns the membership for localhost into that group. We’ve also provided a similar example using service groups – make all services a member of the group ‘http’ based on the ‘check_command’ attribute matching ‘http_*’.

object Host "localhost" {
  import "generic-host"
  address = ""
  address6 = "::1"
  vars.os = "Linux"
object HostGroup "linux-servers" {
  display_name = "Linux Servers"
  assign where host.vars.os == "Linux"
object Service "http" {
  import "generic-service"
  host_name = "localhost"
  check_command = "http"
object ServiceGroup "http" {
  display_name = "HTTP Checks"
  assign where match("http_*", service.check_command)

Relative Service Object Names

We’ve also found it hard to keep unique service object names everywhere. A service object is still bound to a host object, making that relationship unique. Therefore relative service object names are introduced in #5925 allowing the same service object name to be defined multiple times with a different ‘host_name’ attribute. Icinga 2 will create new objects similar to the objects generated by the apply rules, internally.

# node1/services.conf
object Service "mail" {
  host_name = "node1"
# node2/services.conf
object Service "mail" {
  host_name = "node2"

What’s in a Name?

When it comes to naming conventions, we’ve changed various configuration attributes to a shorter name or sometimes more self-explanatory names, for example ‘notification_type_filter’ is now called ‘types’ only (#5906, #5857, #5960 ). Furthermore the notification and dependency filters don’t have their own prefix anymore (for example ‘Up’ or ‘Critical’, #5906)

template Notification "mail-service-notification" {
  command = "mail-service-notification"
  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  period = "24x7"


The ‘macros’ and ‘custom’ dictionaries have been merged into a general ‘vars’ dictionary (details in #5855). This may sound like existing 1.x custom variables and there is certainly a similarity when making them available to existing interfaces. Custom attributes are available in command objects too – DB IDO, Livestatus and status.dat schemas have been updated for that reason.
These custom attributes are available as runtime macros in their local object scope as $varname$, or as object attributes in their inherited scope, like $host.vars.varname$ accessed in a certain service check command. The previous notation ${HOST,SERVICE}attributename$ has been dropped (same for $_{HOST,SERVICE}varname$ when accessing custom attributes).
Special runtime macros such as state information or the application time can be accessed using their scope prefix for example $host.state$ or $icinga.datetime$. This change applies to performance data file templates as well as existing command definitions. If the address macro cannot be resolved, it won’t be replaced by the host name – that is one major difference to the workaround in Icinga 1.x.
Previously named ‘export_macros’, the command attribute is now called ‘env’ and the dictionary enabling  environment variables and their values ‘escape_macros’ to be defined  is gone. All macros are escaped if a string is provided.

object Host "macro-test-host1" {
  import "generic-host"
  vars.macro1 = "foo1"
  vars.macro2 = "bar1"
object Host "macro-test-host2" {
  import "generic-host"
  vars.macro1 = "foo2"
  vars.macro2 = "bar2"
apply Service "macro-test-service" {
  check_command = "macro-test-check"
  vars.address = "$host.vars.macro1$"
  assign where match("macro-test-host*", host.name)
object CheckCommand "macro-test-check" {
  command = "echo "address macro: '$address$' env var: $$MACROTEST""
  env.MACROTEST = "$host.name$ $address$"

Custom attributes can be modified and reset at runtime using external commands. All external interfaces will reflect these changes properly.

[2014-04-15 17:26:08 +0200]  information/compat: Executing external command: [1397575568] CHANGE_CUSTOM_HOST_VAR;macro-test;urlpath;override_custom_var_value
[2014-04-15 17:26:08 +0200]  information/icinga: Changing custom var 'urlpath' for host 'macro-test' to value 'override_custom_var_value'

More Macros & Attributes

When it comes to runtime macros, we found it reasonable to access object attributes in apply rules . Therefore the runtime macros are evaluated as such too – $host.address$ accesses the host’s ‘address’ attribute for example. Under certain circumstances custom attributes may be overridden by macro-evaluated values (#5959).

object Host "keks" {
  import "generic-host"
  vars.ip_address = "$vars.my_customer_ip$"
  vars.my_costumer_ip = ""
  vars.my_service_http_ip = ""
apply Service "schaschlik" {
  import "generic-service"
  vars.ip_address = "$host.vars.my_service_http_ip$"
  assign where host.vars.my_service_http_ip

Furthermore the most common host, service and group attributes (#5856) have been added again – ‘address’, ‘address6’, ‘notes’, ‘notes_url’, ‘action_url’, ‘icon_image’, ‘icon_image_alt’. You may use them for all available backends (and web interfaces can use them again). If you require additional custom attributes, the ‘vars’ dictionary is still available. Last but not least, global constants do not have the ‘Icinga’ prefix anymore.

Mapping Host Status

Aside from configuration changes, we’ve thought about the decision to make a service the virtual host check, and require everyone to follow in that spirit. Interfaces like DB IDO, Livestatus, Status files, Performance data writer were just mapping the host check’s service for that reason and duplicating the data somehow.
Furthermore the handling of a critical service is influenced by different rules and thresholds to a host being down – be it for notifications, SLA reporting or dependencies.
Thanks to our 5th (felt like the 1000th) discussion on that topic, the host ‘check_command’ is now back on duty. This means that the default configuration does not map ‘ping4’ as a virtual host check but ships the ‘hostalive’ check command with different thresholds as defined host check. The idea of cloning a service’s state into a host is not gone – we found it very useful, and will properly implement a general ‘clone state’ functionality for all checkable objects (hosts and services). The ‘hostalive’ check command is shipped using  ITL.

template Host "generic-host" {
  max_check_attempts = 5
  check_interval = 5m
  retry_interval = 1m
  check_command = "hostalive"

This is also visible when providing host performance data in PerfdataWriter/GraphiteWriter features. Meanwhile DB IDO, Livestatus and the status files fetch their data from the defined host state (no direct visible change).


The updated example configuration in /etc/icinga2/conf.d reflects all those changes (and is incompatible with previous configurations). The example ‘localhost’ configuration has been moved into a separate directory named ‘hosts’. An experimental not yet documented feature is optional command arguments for commands – for details, see #5933. We plan to finish this one in the next version.
If you have made it this far, you are most certainly ready to test. So please, download away and share your thoughts with us on the usual channels.

Icinga 2 0.0.9 – Introducing Configuration Apply Rules

Download_Icinga2v0.0.9This release introduces a cool new feature for your Icinga 2 configuration: You can now use a rule based apply logic to assign services to hosts (and notifications, downtimes, dependencies to services, that is). The conditions for assigning (and ignoring for exclusion) are capable of accessing all object attributes, not only host or host group names. Apply rules can also import existing templates.
A simple example assigning the service “ping6” to all hosts which have the “address6” macro defined and import the “ripe-host” template looks like this:

template Host "ripe-host" {
  check = "ping6"
object Host "atlas.ripe.net" {
  import "ripe-host",
  macros.address6 = "2001:67c:2e8:22::c100:69e"
object Host "labs.ripe.net" {
  import "ripe-host",
  macros.address6 = "2001:67c:2e8:22::c100:699"
apply Service "ping6" {
  import "generic-service",
  check_command = "ping6",
  assign where "ripe-host" in host.templates,
  ignore where !host.macros.address6

The updated localhost.conf also reflects that new configuration method.
We’ve discussed quite a lot during the past weeks about the configuration format, and especially the collected feedback. While using templates and inline object definitions will work for generic configurations, it will get tricky with the obvious requirement of “assign a service to a group of hosts”. Every admin asks about it, and the new apply logic simply allows you to do it. Not only with groups, but with every other object attribute (define your own macro “sla” and match that for example).
Since this release is also about changes while improving features: The apply logic replaces the inline object definitions. They’re gone, so make sure to update your object configuration right away. There are some more to come for the 0.0.10 release prior a final release.
Further changes introduce the “import” keyword instead of the object “inherits” keyword. The global $plugindir$ macro has been replaced by the constant named “PluginDir” defined in constants.conf (newly included in icinga2.conf, required by the Icinga Template Library (ITL)).
If you’re running into configuration errors – their output has been significantly improved by providing detailed information on which string is incorrect. Unknown attributes and duplicate object definitions now result in a configuration error. Last but not least: The new host check attribute requires Classic UI 1.11.1 if you’re using this user interface for Icinga 2.
Download Icinga 2 0.0.9 now – package builds are available, for demo cases try the Vagrant Box.
Thanks for all your feedback & bug reports – they are paving the way towards a final release! 🙂

  • new apply rules for assigning objects based on attribute conditions
  • inline object definitions removed in favor of apply rules
  • import keyword instead of inherits keyword for all objects
  • new constants.conf providing PluginDir constant instead of $plugindir$ macro
  • unknown attributes and duplicate objects generate a configuration error
  • improved configuration error output
  • create endpoint tables for legacy interfaces (status data, DB IDO, Livestatus)
  • export host check attribute in legacy interfaces (status data, DB IDO, Livestatus)
  • add documentation about cluster scenarios
  • Livestatus: add check_source attribute to services table
  • Compat: Fix host service order for Classic UI
  • Remove comments when clearing acknowledgements
  • Recovery Notifications require StateFilterOK


  • removed deprecated var/set identifier, use const instead
  • constants.conf needs to be included in icinga2.conf before ITL inclusion
  • import instead of inherits (examples in localhost.conf)
  • apply rules instead of inline definitions for Service, Dependency, Notification, ScheduledDowntime objects (examples in localhost.conf).
  • unknown attributes and duplicate objects generate a configuration error
  • DB IDO: schema update for 0.0.9 (MySQL, PostgreSQL)