Icinga 2 v2.3.0 released

You may have heard it already – 2.3 adds lots of new features, for example object attribute accessors at runtime accompanied by functions, loops, conditionals and much more. Bringing you Icinga 2 v2.3.0 also means: 660 Git commits since 2.2.0, 94 features & 127 bug fixes.
While upgrading your Icinga 2 installation, you can test-drive the new language features in the new live console online on icinga.org. Grab a coffee, check additional feature details below, switch to the Changelog and once your upgrade has finished, get to work with the all new shiny Icinga 2 v2.3.0. The online documentation is currently undergoing changes, things to note: live search and removable tables of content tab.

Conditional statements

icinga2_2.3_conditions_icingaweb2This was frequently asked in the past: “How can I inherit values from the host to the service, and leave it to a default value if not set?” Consider it done with if-then-else-if-else conditions inside the Icinga 2 configuration language. The example shows a fallback to the host’s FQDN if its address6 attribute has not been set – cool, isn’t it?

Functions – what for?

You can now define your own functions including the return keyword. That includes locally scoped variables identified by the var keyword and anonymous lambda functions too. We’ve thought about functions and their use cases for Icinga 2. One thing we came up with is the Boolean return value for set_if inside command arguments – not only a macro string value, but also nearly any condition. Same applies for command argument values. The short way of assigning return values is putting them in double curly brackets {{ …. }}.

Loops, loops, loops, …

You’ve seen for loops already inside the fancy apply for rules introduced with 2.2. Using while and for loops including break and continue keywords has been experimental for Icinga Web 2’s Vagrant Box quite a while. In real-life scenarios you would use them in combination with functions and if-then-else conditions iterating over arrays and dictionaries defined in custom attributes.

Object attribute accessors for clustered checks

icinga2_2.3_object_runtime_attributes_cluster_bp_icingaweb2Digging up the old problem with so-called “on-demand macros” in Icinga 1.x and migration issues we’ve tackled this one differently: Instead of obfuscating the macro parser once again, the problem is solved differently – you’d want to access objects and their run-time attributes. Most commonly, get_host(NodeName).state for the old-fashioned check_cluster plugin. And many, many objects and attributes more …
That syntax could be used for cluster checks and business processes inside Icinga 2 – we’ll tackle the dummy check problem sooner or later, promise!

Time dependent thresholds

Right before the feature freeze one of our colleagues approached us with the request of time-dependent check threshold values. We’ve already had if-conditions and object accessor functions thus far, and so added the is_inside run-time attribute to the get_time_period() function. That way you can set thresholds depending on the current time of the day.

Type methods

Defined an array, but need it in a sorted manner? Remove a dictionary item inherited from a template? Split a string into parts? Not an issue anymore.

console CLI command

icinga_org_live_consoleTest all the language features inside the Icinga 2 console. Install rlwrap to keep history and line continuation and test-drive your new configuration before putting into the files.

Misc features

From OpenTSDB support to ignoring soft states in dependencies. Additional ITL plugin check commands (interfacetable, IPMI, webinject, vmware_esx, local ‘nscp client’ commands for the Windows agent). Livestatus header support, improved performance and additional bygroup tables. Improved cluster stability and scalability using fewer threads for socket I/O and SNI TLS support. ‘icinga2 troubleshoot’ cli command for better community support … check the Changelog below and in the documentation for more details.
PS: I’ve uploaded the configuration samples made for this blog post into the Vagrant boxes.

2.3.0 Changelog

  • Improved configuration validation
    • Unnecessary escapes are no longer permitted (e.g. \’)
    • Dashes are no longer permitted in identifier names (as their semantics are ambiguous)
    • Unused values are detected (e.g. { “-M” })
    • Validation for time ranges has been improved
    • Additional validation rules for some object types (Notification and User)
  • New language features
    • Implement a separate type for Boolean values
    • Support for user-defined functions
    • Support for conditional statements (if/else)
    • Support for ‘for’ and ‘while’ loops
    • Support for local variables using the ‘var’ keyword
    • New operators: % (modulo), ^ (xor), – (unary minus) and + (unary plus)
    • Implemented prototype-based methods for most built-in types (e.g. [ 3, 2 ].sort())
    • Explicit access to local and global variables using the ‘locals’ and ‘globals’ keywords
    • Changed the order in which filters are evaluated for apply rules with ‘for’
    • Make type objects accessible as global variables
    • Support for using functions in custom attributes
    • Access objects and their runtime attributes in functions (e.g. get_host(NodeName).state)
  • ITL improvements
    • Additional check commands were added to the ITL
    • Additional arguments for existing check commands
  • CLI improvements
    • Add the ‘icinga2 console’ CLI command which can be used to test expressions
    • Add the ‘icinga2 troubleshoot’ CLI command for collecting troubleshooting information
    • Performance improvements for the ‘icinga2 node update-config’ CLI command
    • Implement argument auto-completion for short options (e.g. daemon -c)
    • ‘node setup’ and ‘node wizard’ create backups for existing certificate files
  • Add ignore_soft_states option for Dependency object configuration
  • Fewer threads are used for socket I/O
  • Flapping detection for hosts and services is disabled by default
  • Added support for OpenTSDB
  • New Livestatus tables: hostsbygroup, servicesbygroup, servicesbyhostgroup
  • Include GDB backtrace in crash reports
  • Various documentation improvements
  • Solved a number of issues where cluster instances would not reconnect after intermittent connection problems
  • A lot of other, minor changes
  • DB IDO schema upgrade to 1.13.0 required!

Find the detailed Changelog in the “What’s new” section in the documentation!

New features in Icinga 2.3

The next major version of Icinga 2 will introduce a bunch of interesting features which should make it even easier to define exceptions for services (as in, all services have a check_interval of 5 minutes except for…). Version 2.3 won’t be available for another couple of weeks but here’s a quick introduction to some of its features:

Conditional Statements

More commonly known as if/else: In 2.3 it’s possible to set attributes based on whether some condition is true. Here’s an example:

object Host "localhost" {
  check_command = "hostalive"
  address = ""
  vars.http_vhosts["icinga.org"] = {
    http_address = "icinga.org"
    interval = 1m
  vars.http_vhosts["dev.icinga.com"] = {
    http_address = "dev.icinga.com"
apply Service "vhost " for (vhost => config in host.vars.http_vhosts) {
  host_name = "localhost"
  check_command = "http"
  if (config.interval) {
    check_interval = config.interval
  } else {
    check_interval = 5m
  assign where host.vars.http_vhosts

Debug Console

In order to make testing filter rules for “apply” as well as other expressions easier we have implemented a CLI-based console which can be used to evaluate arbitrary expressions and show their results:

$ icinga2 console
Icinga (version: v2.2.0-282-g9898971)
<1> => config = { http_address = "icinga.org", interval = 1m }
<2> => if (config.interval) { check_interval = config.interval } else { check_interval = 5m }
<3> => check_interval


All built-in data types (i.e. strings, numbers, arrays and dictionaries) now have their own methods. Here’s an example how we can use those methods to manipulate dictionaries:

<1> => vhosts = { "icinga.org" = { http_address = "icinga.org" },
"dev.icinga.com" = { http_address = "dev.icinga.com" } }
<2> => vhosts.remove("icinga.org")
<3> => vhosts
<4> => vhosts.len()

Using Dictionary#remove we can remove specific dictionary items which is rather useful if those dictionary keys should not be set for some hosts or services.

Icinga 2 release 2.2.0 – Agent, Cli & Apply For Rules

When releasing Icinga 2 in its final version, we got quite a lot of feedback on why there is no agent, and how to monitor remote clients. Back in the beginning of 2014 we already had an idea on how to implement it, but you’d better design, re-design and even re-design such an important feature last. Especially when there are existing solutions out there that users can adopt.

Remote Clients – the “Agent”

icinga2_execute_remote_checksThat is merely the reason why everything took longer than expected. Furthermore, an “agent” is understood quite differently by different users. One thinks of a full-featured Icinga 2 instance with local configuration, local check scheduler and host/service discovery on the master. Others just want to a secure way of executing checks remotely, and drop the old-fashioned insecure NRPE protocol. And yet, users don’t care about their operating system – an agent must run on Linux, Unix – and Windows. Luckily Icinga 2 was designed to work on Windows from conception. Before you continue reading – all the mentioned roles and distributions work and have been implemented in this release 2.2.0 already.

Installation & CLI idea

But there is more – how to safely install that agent? While installing an Icinga 2 (HA) Cluster is relatively easy, it still requires knowledge on SSL certificates and other manual configuration steps. So we noticed that many users struggled with it. And such a thing just for an agent? No, there must be something more simple. In the design and concept phase of the agent bits in Icinga 2, we came up with our very own cli implementation. It supports you by adding setup wizards for master and client nodes, doing all the SSL setup magic. Furthermore, the Icinga 2 cluster protocol was extended to support CSR-Auto-Signing. Simply said – your client nodes can be installed with a single generated ticket number, no need for local SSL certificates.
Going further, remote satellite nodes with their local config will report their objects to the central master. There are cli commands to assist you in adding the nodes, black/whitelisting hosts and services and also generating local configuration on the master. There is plenty of newly written documentation for that, so be sure to check it out. Oh, and if you don’t require that, and just want to execute remote commands – that’s all in there as well.

CLI with Auto-Completion

That cli framewicinga2_cli_command_feature_auto_completionork also supports context based bash auto-completion. We’ve also integrated existing scripts like “feature enable” or “object list” into the cli, and removed the old shell and python scripts. The “python-icinga2” package is gone for that very reason too. Generating your CA and SSL certificates is also supported out of the box. Similar to the agent setup on Linux, the native Windows installer uses the icinga2 cli commands too.

More dynamic configuration with apply for

While this is already one of the most epic things we’ve ever implemented – cli & agent – we did not stop there. Users have been asking, why custom attributes (the vars. identifier) could only hold strings, numbers and boolean values but not arrays or dictionaries. The latter could be used like groups for better apply rules expressions.
So, we added that support, and decided to dump the values as json into the existing backends, adding a new column “is_json” to allow existing interfaces to show that correctly, as seen in Icinga Web 2 already. But we did not stop here – why not extend the existing apply rules to loop over arrays/dictionaries? That way you can save yourself a lot of typing and generate new service apply rules based on host custom attributes. Sounds complicated, but once you’ve tried it, you’ll never want to go back. That gets even more interesting when you generate the host from your CMDB, Puppet, <insert #devops tool here>. It certainly provides an even more dynamic approach. Take a look at the configuration screenshots for the new “apply <objecttype> <optionalprefix> for (key => value in dict)” syntax 🙂
icinga2_custom_dict_arr_attributes_host icinga2_custom_dict_arr_attributes_services icinga2_custom_dict_arr_attributes_icingaweb2

More config magic – apply with variables

Iicinga2_cli_command_object_list_filtern a different situation – back with the agent, and its dependencies, we’ve learnt, that setting local variables in apply rules should be able to read host or service attributes in that scope. For instance, you would want to generate host and service dependencies for your vmware or cloud farm, and set the parent_host_name attribute directly in the child hosts inherited template. No need for duplicate dependency rules – control them using apply and a locally-scoped variable set. You’ll find a more telling example on the doumentation as well, and also that this feature isn’t dependency-exclusive – it can be used for all apply rules.
And while that could become tremendously complicated, the “object list” cli command allows you to filter by name or type wildcard strings. Plus, we’ve worked a lot on possible configuration errors, making them as telling as possible, even for complicated nested apply rules.

More Features: Graphite, GELF

Apart from the core feature set, we noticed at Icinga Camp San Francisco that many already use the GraphiteWriter in production – which is freaking awesome! While chatting with Grant from SpaceX, we’ve also made sure that everyone out there can configure the host and service prefix template, thus adding more statistics and making it more usable afterall. You’ll also recognize the GelfWriter feature, which was contributed by the graylog2 developers and we found it so nice to include it in 2.2. There’s a talk on this years OSMC on that topic, more details once OSMC is over.

Get Icinga 2

Last but not least – thanks everyone for their ongoing feedback. The documentation and also the example configuration has been overhauled in many places. Be it better explanations of apply rules and their expressions in general, or detailed examples on how to use the new apply for rules. We’ve even shipped in an example configuration. The old strategy with single objects just does not work that well now with ever more dynamic apply rules, and only confuses you, the user 😉
The cluster vagrant boxes ship new demo configurations for cluster and remote check execution bits, if you want to give it a try.
Package builds are running, and hopefully everyone gets 2.2.0 asap. While downloading, please be sure to read the Changelog and all the changes introduced with this release. They require your attention! 🙂
As always, thanks for using Icinga 2 and watch out for Icinga Web 2 🙂 Feedback, bugs or feature requests are always welcome!
PS: This video was taken during Icinga 2 development. Get the Windows Agent from packages.icinga.org and try it yourself!


2.2.0 Changelog

For details on the below issues see https://dev.icinga.com/versions/200


  • DB IDO schema update to version `1.12.0`
    • schema files in `lib/db_ido_{mysql,pgsql}/schema` (source)
    • Table `programstatus`: New column `program_version`
    • Table `customvariables` and `customvariablestatus`: New column `is_json` (required for custom attribute array/dictionary support)
  • New features
    • GelfWriter: Logging check results, state changes, notifications to GELF (graylog2, logstash) #7619
    • Agent/Client/Node framework #7249
    • Windows plugins for the client/agent parts #7242 #7243
  • New CLI commands #7245
    • `icinga2 feature {enable,disable}` replaces `icinga2-{enable,disable}-feature` script #7250
    • `icinga2 object list` replaces `icinga2-list-objects` script #7251
    • `icinga2 pki` replaces` icinga2-build-{ca,key}` scripts #7247
    • `icinga2 repository` manages `/etc/icinga2/repository.d` which must be included in `icinga2.conf` #7255
    • `icinga2 node` cli command provides node (master, satellite, agent) setup (wizard) and management functionality #7248
    • `icinga2 daemon` for existing daemon arguments (`-c`, `-C`). Removed `-u` and `-g` parameters in favor of [init.conf](#init-conf).
    • bash auto-completion & terminal colors #7396
  • Configuration
    • Former `localhost` example host is now defined in hosts.conf #7594
    • All example services moved into advanced apply rules in services.conf
    • Updated downtimes configuration example in downtimes.conf #7472
    • Updated notification apply example in notifications.conf #7594
    • Support for object attribute ‘zone’ #7400
    • Support setting object variables in apply rules #7479
    • Support arrays and dictionaries in custom attributes #6544 #7560
    • Add apply for rules for advanced dynamic object generation #7561
    • New attribute `accept_commands` for ApiListener #7559
    • New init.conf file included first containing new constants `RunAsUser` and `RunAsGroup`.
  • Cluster
    • Add CSR Auto-Signing support using generated ticket #7244
    • Allow to execute remote commands on endpoint clients #7559
  • Perfdata
    • PerfdataWriter: Don’t change perfdata, pass through from plugins #7268
    • GraphiteWriter: Add warn/crit/min/max perfdata and downtime_depth stats values #7366 #6946
  • Packages
    • `python-icinga2` package dropped in favor of integrated cli commands #7245
    • Windows Installer for the agent parts #7243

Please remove `conf.d/hosts/localhost*` after verifying your updated configuration!


  • Feature #6544: Support for array in custom variable.
  • Feature #6946: Add downtime depth as statistic metric for GraphiteWriter
  • Feature #7187: Document how to use multiple assign/ignore statements with logical “and” & “or”
  • Feature #7199: Cli commands: add filter capability to ‘object list’
  • Feature #7241: Windows Wizard
  • Feature #7242: Windows plugins
  • Feature #7243: Windows installer
  • Feature #7244: CSR auto-signing
  • Feature #7245: Cli commands
  • Feature #7246: Cli command framework
  • Feature #7247: Cli command: pki
  • Feature #7248: Cli command: Node
  • Feature #7249: Node Repository
  • Feature #7250: Cli command: Feature
  • Feature #7251: Cli command: Object
  • Feature #7252: Cli command: SCM
  • Feature #7253: Cli Commands: Node Repository Blacklist & Whitelist
  • Feature #7254: Documentation: Agent/Satellite Setup
  • Feature #7255: Cli command: Repository
  • Feature #7262: macro processor needs an array printer
  • Feature #7319: Documentation: Add support for locally-scoped variables for host/service in applied Dependency
  • Feature #7334: GraphiteWriter: Add support for customized metric prefix names
  • Feature #7356: Documentation: Cli Commands
  • Feature #7366: GraphiteWriter: Add warn/crit/min/max perfdata values if existing
  • Feature #7370: CLI command: variable
  • Feature #7391: Add program_version column to programstatus table
  • Feature #7396: Implement generic color support for terminals
  • Feature #7400: Remove zone keyword and allow to use object attribute ‘zone’
  • Feature #7415: CLI: List disabled features in feature list too
  • Feature #7421: Add -h next to –help
  • Feature #7423: Cli command: Node Setup
  • Feature #7452: Replace cJSON with a better JSON parser
  • Feature #7465: Cli command: Node Setup Wizard (for Satellites and Agents)
  • Feature #7467: Remove virtual agent name feature for localhost
  • Feature #7472: Update downtimes.conf example config
  • Feature #7478: Documentation: Mention ‘icinga2 object list’ in config validation
  • Feature #7479: Set host/service variable in apply rules
  • Feature #7480: Documentation: Add host/services variables in apply rules
  • Feature #7504: Documentation: Revamp getting started with 1 host and multiple (service) applies
  • Feature #7514: Documentation: Move troubleshooting after the getting started chapter
  • Feature #7524: Documentation: Explain how to manage agent config in central repository
  • Feature #7543: Documentation for arrays & dictionaries in custom attributes and their usage in apply rules for
  • Feature #7559: Execute remote commands on the agent w/o local objects by passing custom attributes
  • Feature #7560: Support dictionaries in custom attributes
  • Feature #7561: Generate objects using apply with foreach in arrays or dictionaries (key => value)
  • Feature #7566: Implement support for arbitrarily complex indexers
  • Feature #7594: Revamp sample configuration: add NodeName host, move services into apply rules schema
  • Feature #7596: Plugin Check Commands: disk is missing ‘-p’, ‘x’ parameter
  • Feature #7619: Add GelfWriter for writing log events to graylog2/logstash
  • Feature #7620: Documentation: Update Icinga Web 2 installation
  • Feature #7622: Icinga 2 should use less RAM
  • Feature #7680: Conditionally enable MySQL and PostgresSQL, add support for FreeBSD and DragonFlyBSD
  • Bug #6547: delaying notifications with times.begin should postpone first notification into that window
  • Bug #7257: default value for “disable_notifications” in service dependencies is set to “false”
  • Bug #7268: Icinga2 changes perfdata order and removes maximum
  • Bug #7272: icinga2 returns exponential perfdata format with check_nt
  • Bug #7275: snmp-load checkcommand has wrong threshold syntax
  • Bug #7276: SLES (Suse Linux Enterprise Server) 11 SP3 package dependency failure
  • Bug #7302: ITL: check_procs and check_http are missing arguments
  • Bug #7324: config parser crashes on unknown attribute in assign
  • Bug #7327: Icinga2 docs: link supported operators from sections about apply rules
  • Bug #7331: Error messages for invalid imports missing
  • Bug #7338: Docs: Default command timeout is 60s not 5m
  • Bug #7339: Importing a CheckCommand in a NotificationCommand results in an exception without stacktrace.
  • Bug #7349: Documentation: Wrong check command for snmp-int(erface)
  • Bug #7351: snmp-load checkcommand has a wrong “-T” param value
  • Bug #7359: Setting snmp_v2 can cause snmp-manubulon-command derived checks to fail
  • Bug #7365: Typo for “HTTP Checks” match in groups.conf
  • Bug #7369: Fix reading perfdata in compat/checkresultreader
  • Bug #7372: custom attribute name ‘type’ causes empty vars dictionary
  • Bug #7373: Wrong usermod command for external command pipe setup
  • Bug #7378: Commands are auto-completed when they shouldn’t be
  • Bug #7379: failed en/disable feature should return error
  • Bug #7380: Debian package root permissions interfere with icinga2 cli commands as icinga user
  • Bug #7392: Schema upgrade files are missing in /usr/share/icinga2-ido-{mysql,pgsql}
  • Bug #7417: CMake warnings on OS X
  • Bug #7428: Documentation: 1-about contribute links to non-existing report a bug howto
  • Bug #7433: Unity build fails on RHEL 5
  • Bug #7446: When replaying logs the secobj attribute is ignored
  • Bug #7473: Performance data via API is broken
  • Bug #7475: can’t assign Service to Host in nested HostGroup
  • Bug #7477: Fix typos and other small corrections in documentation
  • Bug #7482: OnStateLoaded isn’t called for objects which don’t have any state
  • Bug #7483: Hosts/services should not have themselves as parents
  • Bug #7495: Utility::GetFQDN doesn’t work on OS X
  • Bug #7503: Icinga2 fails to start due to configuration errors
  • Bug #7520: Use ScriptVariable::Get for RunAsUser/RunAsGroup
  • Bug #7536: Object list dump erraneously evaluates template definitions
  • Bug #7537: Nesting an object in a template causes the template to become non-abstract
  • Bug #7538: There is no __name available to nested objects
  • Bug #7573: link missing in documentation about livestatus
  • Bug #7577: Invalid checkresult object causes Icinga 2 to crash
  • Bug #7579: only notify users on recovery which have been notified before (not-ok state)
  • Bug #7585: Nested templates do not work (anymore)
  • Bug #7586: Exception when executing check
  • Bug #7597: Compilation Error with boost 1.56 under Windows
  • Bug #7599: Plugin execution on Windows does not work
  • Bug #7617: mkclass crashes when called without arguments
  • Bug #7623: Missing state filter ‘OK’ must not prevent recovery notifications being sent
  • Bug #7624: Installation on Windows fails
  • Bug #7625: IDO module crashes on Windows
  • Bug #7646: Get rid of static boost::mutex variables
  • Bug #7648: Unit tests fail to run
  • Bug #7650: Wrong set of dependency state when a host depends on a service
  • Bug #7681: CreateProcess fails on Windows 7
  • Bug #7688: DebugInfo is missing for nested dictionaries