Now we are here, after many months of development – we proudly release Icinga 2.11 available today.
It has been an emotional ride with many changes under the hood. The most obvious change is that Icinga’s distributed cluster operates more stable, the past quirks with hanging certificate signing requests or dead-locked TLS handshakes are now gone. This required us to go an unusual route: Evaluate new libraries and programming techniques in order to replace hand-written lower layered code, with later replacing the entire code base for the network stack operations in Icinga. This is a massive effort in quality and stability where users had called out for 3.0 already.
The core components of Icinga are checks and notifications. Thanks to sponsoring from a customer, Icinga now immediately send notifications after a downtime ends whenever hosts/services are still NOT-OK. Lost notifications during restarts in HA zones also have been tackled and fixed.
In the past years, we’ve seen operation problems with the config sync inside the cluster system. This has been tailored to a more robust deployment with staged syncs as well as preventing unwanted sync loops for comments/downtimes. The local storage for runtime created objects also is more robust. Also, note additional HA capabilities for features like Graphite or Elasticsearch.
We care about security. The setup steps with TLS certificates and signing requests in addition to the trusted zone hierarchy is necessary to keep your monitoring data safe. Icinga can see many details in your environment and must not be compromised. Taking things serious, Icinga 2.11 enforces TLS 1.2 as a minimum and also hardens the available cipher lists by default. Older agent versions still work granted that they support TLS 1.2 (OpenSSL 1.x.y).
More Icing on the cake
Icinga aims to run in small environments as well as large distributed enterprise systems. With the general availability of Debian Buster on Raspberry Pi, Icinga 2.11 is ready for your home lab and IoT setups.
To all the fans of on-demand certificate signing: You’ll get more tooling to manage/revoke satellite and agent certificate requests. A small but nifty addition are environment variables which can be retrieved with the new getenv() DSL function – for sensitive data or getting the path of HOME into your command’s context.
A small but gentle addition is the “all_services” attribute when setting a host in downtime with the REST API. Aside from the more visible fixes, many changes have been applied to improve our code base for the next years and avoid future bugs. This includes more unit tests e.g. for timeperiod ranges or version comparisons in the built-in icinga check.
Coming late to the party, we’ve fixed crashes from Nessus scanners on Linux and following up on Windows. Well, this wasn’t easy – a technical monster to debug and fix. Cheers to our friends Stefan and Philipp in Linz who have helped us to reproduce the issue. The technology now used in Icinga 2.11 proves the rule for scaling monitoring for the next decade.
Icinga uses the well-known Boost library as part of the network stack rewrite, thanks to everyone who contributes to their development. Also we’d like to cordially say thanks to Niels Lohmann for the magic JSON library, and Nemanja Trifunovic for making UTF8 handling in C++ a breeze. And to you, our fellow community contributors!
In the late development stages for 2.11, we’ve also tackled a problem with systemd reloads resulting in a killed process. The only way to prevent this from happening is actually a new feature: A real umbrella process managing the reload signal. Turns out, this implements a long standing feature request for running Icinga in a (Docker) container.
Docs, docs, docs
Our community forum with many great users and contributors helping each other raised the bar for the documentation. We are addicted to making the Icinga experience as easy as possible, and for 2.11, we have improved the documentation in many ways: From the basic setup to configuration to distributed environments to service monitoring to features and addons.
This includes a clear cut specification of the plugin API, with best practices from experienced plugin developers. Speaking of development, packagers and future developers are key to Icinga’s success. The development chapter as been overhauled to motivate everyone out there. Those wanting to learn about the design decisions, the technical concepts chapter highlights new bleeding edge insights.
Get Icinga 2.11, check the Changelog & upgrading docs and say cheers to the biggest release we’ve ever done.
Some say you have to move all of your server infrastructure into the cloud. Others counter that you should keep your data safe and secure in your own datacenter. And then there are many people in between who use cloud services as an addition to their self-hosted servers. In fact, there’s no right or wrong, because as always in IT: It depends. We at Icinga always try to find a way to make everyone happy with their monitoring – be it in the cloud or on premise. Today we’re pleased to announce version 1.0 of our Icinga Module for AWS!
Automation and integrations are important parts of every Icinga setup since ever then. The Icinga Module for AWS follows this path by embedding into your existing monitoring setup. The module is an addition for the well-known Icinga Director, whose main purpose is to manage Icinga configuration at a glance – the Icinga Module for AWS is yet another Director Import Source. Import sources are a crucial factor of the Director as they enable you to import data from multiple data providers. The collected data is transformed into Icinga 2 configuration. Learn more about import sources and automation in our guide Monitoring Automation with Icinga – The Director.
This module collects data about your EC2 Instances, RDS Instances, Load Balancers and Auto Scaling Groups. The Director uses the regularly collected information to create Icinga 2 configuration out of it. Since this can be automated, you can just continue on spawning and deleting new instances in AWS without having to take care about if they’re going to be monitored or not – without any human interaction.
Import EC2 Instances
The following example demonstrates how you can use this module to import details about your currently running EC2 Instances into Icinga. Before you get started, make sure you have your AWS Access Key by hand. This is required, so you can establish a connection to your account and pick up the information we need for monitoring. Make sure to add your Key details to the modules configuration at “Configuration > Modules > aws > AWS Keys”.
Installation and Configuration
Extract or clone this module to your Icinga Web 2 module path. The directory name must fit the module name, aws. This would usually lead to
Additionally, the module requires the AWS PHP SDK v3 which you have to install manually. The easiest way to do this is by downloading it from the release page and unpacking it into
Configuring the Import Source
The first step is to create a new Director Import Source:
- Import source name: A general identifier, choose something that you will recognise later
- Source type: Select
- AWS region: You may create an import source for each region you use
- AWS access method: Select your previously configured AWS Key from the dropdown
- Object type: We use
EC2 Instances for this demo purpose, you may create multiple import jobs for each object type you want to import
Save the details and open the newly created Import Source again. We now continue with adding a Modifier.
Import only running Instances
Director Modifiers are used to handle the incoming data and transform it or filter it by certain conditions. Since you usually want to monitor only running instances, we’re using a Modifier to filter for the state of the EC2 Instances.
- Property: We refer to the
status property, in order to filter by states
- Modifier: The
Black or White-list modifier allows you to whitelist only certain instances
- Filter: We filter by
running systems only
- Policy: Here we decide to keep only matching rows, therefore only running systems are whitelisted
Hit the save button to save your Modifier.
Trigger Import Run
You’re now ready to trigger the first import run. This will not create any Icinga configuration yet, but allow you to see a preview of the collected information in the “Preview” tab. There you can now see a variety of data that was synchronised from your AWS account. All of these information can be used to create Icinga hosts for example. Additionally you may use part of the data and add it as custom variables to your monitoring objects.
Your next step is to create a Sync Rule which will use the gathered AWS data to generate Icinga configuration. If you’re not familiar with Sync Rules, you may refer either to the Director Documentation to learn about it or read our guide Monitoring Automation with Icinga – The Director
We are proud to announce that icinga.com is now also hosting integrations of hardware for monitoring environmental sensors and alerting via text message. The devices from the manufacturers listed on Icinga Integrations can easily be implemented into your Icinga monitoring by using plugins provided on Icinga Exchange.
Alerts via Text Message
Basically, there are two types of devices, which are most suitable for integration with Icinga. First group of them are gateways for sending SMS as alert messages. These gateways work with standard SIM cards and can run on bands in 4G, 3G and 2G networks. These devices also provide more functionality than only sending messages, they are also capable of converting SMS to E-Mail or the other way round or using filters and routing for sending messages only to a certain person or group of persons (phonebook listing). Furthermore, the gateways have their own monitoring server which means that they can automatically send notifications for example in case of loosing connection to the network.
In order to smoothly add these gateways into one’s productive processes, there exist for example plugins for directly connecting to your Microsoft Exchange for full usage of E-Mail to SMS conversion. Integrated with Icinga 2 you cannot only monitor the device itself but also use it in order to forward notifications generated by Icinga. The two main manufacturers of these gateways are Braintower located in Germany and SMSEagle with their headquarters in Poland.
Second type of integratable devices are mainly meant for measuring and surveillance. These consist most of the time of a base unit and several ports for connecting sensors. The base units can either directly communicate via LAN or via GSM in case of remote locations. For full environmental monitoring, there are plenty of sensor types that can be attached:
- combination of temperature and humidity
- air pressure
- air flow
- water and leakage
- smoke and Gas
- levels of liquids like pertrol
- movement, vibration and opening of doors
The main manufacturers for measuring devices are currently AKCP, GUDE, HW group and Tinkerforge. For all these you can find monitoring plugins on Icinga Exchange. The manufacturer GUDE also offers power supplies that cannot only monitor voltage, but are also capable of remote controlling plugs and devices.
Integrating Tinkerforge with Icinga
In order to have a closer look at how these devices can be integrated with Icinga 2, Tinkerforge is one of the best examples. The whole plugin project can be found at https://exchange.icinga.com/netways/check_tinkerforge and shows not only the integration but also offers an insight into the functionalities of the device.
For using the check_tinkerforge plugin it is required to run Python 2.7+ and to have the tinkerforge Python library from Pypi installed. After this, installation is done with
pip install tinkerforge and moving the plugin into the Icinga
Now let’s have a look at the options of the plugin:
$ ./check_tinkerforge.py --help
usage: check_tinkerforge.py [-h] [-V] [-v] -H HOST [-P PORT] [-S SECRET]
[-u UID] -T TYPE [-w WARNING] [-c CRITICAL]
-h, --help show this help message and exit
-V, --version show program's version number and exit
-H HOST, --host HOST The host address of the Tinkerforge device
-P PORT, --port PORT Port (default=4223)
-S SECRET, --secret SECRET Authentication secret
-u UID, --uid UID UID from Bricklet
-T TYPE, --type TYPE Bricklet type. Supported: 'temperature', 'humidity', 'ambient_light', 'ptc'
-w WARNING, --warning WARNING Warning threshold. Single value or range, e.g. '20:50'.
-c CRITICAL, --critical CRITICAL Critical threshold. Single vluae or range, e.g. '25:45'.
-t TIMEOUT, --timeout TIMEOUT Timeout in seconds
As the TinkerForge device supports sensors like temperature or humidity as well as other types, we need to indicate what sensor should be checked by using the
check_tinkerforge.py -H 10.0.10.163 -T temperature -w 23
WARNING - Tinkerforge: Temperature is 24.75 degrees celcius|'temperature'=24.75
In case you have more multiple sensors of one type, you are required to indicate the UID with the
--uid operator in order to correctly identify the sensor in question. Furthermore, thresholds can either be indicated as single values or value ranges and by using the usual operators for warning and critical.
For integration of the sensors into your Icinga 2 configuration, you need to create a
CheckCommand object like for other check plugins. The device itself will be created as
Host object and all sensors should be covered with an
apply Service. For your comfort you can find an example config for TinkerForge here. This example can directly be used, it is only necessary to change the parameters and in case you have multiple sensors of one type, to add the UIDs.
All manufacturers mentioned in this blog post can be found on the Icinga integrations website including information on the companies, the devices and the links to the corresponding monitoring plugins.
We are happy to announce a new bugfix release for Icinga Web 2. Official packages are available on packages.icinga.com. You can find all issues related to this release on our Roadmap.
Sneaky Solution for Sneaky Links
Usually we try to include only bugs in minor-releases. Sorry, bug-fixes, of course. But thanks to @winem_ we have also a little enhancement this time: Links in comments, notes, etc. are now highlighted as such.
- Highlight links in the notes of an object #3888
Nobody’s Perfect, Not Even Developers
We knew it. We saw it coming. And forgot about it. Some views, especially histories, showed an anarchic behavior since v2.7.0. The change responsible for this has been undone and history’s order is reestablished now.
- Default sort rules no longer work in 2.7.0 #3891
Restrictions Gone Wild Cagey
A fix unfortunately caused restrictions using wildcards to show no results anymore. This is now solved and such restrictions are as permissive as ever.
- Wildcard filters in chains broken #3886
Give your Icinga Web 2 a new look – with official ‘Color Blind Theme’ introduced in version 2.7.0! Not just for people with impaired colour vision – with a fresh and well rounded new palette, the new theme is a real looker!
It’s important for us that everyone has the chance to use our tools with maximum effectiveness.We took an existing module from our Icinga Partner Sol1 – icingaweb2-theme-colourblind by Andrew Foster – refined it a little and it is now part of the default Icinga Web 2 colour scheme bundle!
What’s with the change?
The colorblind theme by Sol1 was already very well made, but we saw some minor issues with the chosen colours in terms of:
- context with the icinga states
Well, let’s just have a look first:
Overview over the different themes and the effects colour deficiency has on the appearance
So in the top, there is the default Icinga colour scheme – with mocks of the three types of colour deficiency we focused on during the development for the module.
In the middle row you can see the module Sol1 provided
Last row is the new theme which is integrated into the Icinga Web 2 core now.
Icinga state context
Another minor detail to improve was the colour of the WARNING (handled) state, which didn’t really fit in with the rest:
The WARNING (handled) grey sticks out and is hard to connect to the unhandled state colour
So in order to both have the context in which the colours are used preserved and have some sort of hierarchy in severeness of the states some changes needed to be made.
This warranted for a change both in brightness and hue for the OK green as well, so it could be distinguished from the WARNING (handled) yellow:
Visual representation of changes in hue, saturation and brightness.
There have also been some changes to the font colour:
Handled states (Including OK for urgencies sake) are now displayed with black text.
This makes it a lot easier to distinguish between handled and unhandled states in the blink of an eye.
Badges with the different themes
In case you want to build your own theme, there is a helpful article on the topic from our Partner Würth Phoenix. If you have any issues, feedback or inspiration – please tell us about it here or open an issue on github.
And by the way, I will be speaking about this topic at the Icinga Camp Stockholm as well!
We are happy to announce a new release for Icinga Web 2, version 2.7.0. Official packages are available on packages.icinga.com. You can find all issues related to this release on our Roadmap.
Icinga’s Amazingness Spreads Further
All the Japanese and Ukrainian monitoring enthusiasts can now appreciate our web-frontend in their native tongue. Being so late to the party is also of their advantage, though. Because they can adjust their dashboard without worrying it gets broke with the next update. (All other admins with non-english users, please have a look at our upgrading documentation)
- Add Japanese language support #3776
- Add Ukrainian language support #3828
- Don’t translate pane and dashlet names in configs #3837
Modules – Bonus Functionality Unleashed
With this release module developers got additional ways to customize Icinga Web 2. Whether you ever wanted to hook into a configuration form’s handling, to perform your very own Ajax requests or enhance our multi-select views with fancy graphs. All is possible now.
- Allow to hook into a configuration form’s handling #3862
- Allow to fully customize click and submit handling #3767
- Integrate DetailviewExtension into multi-select views #3304
UI – Your Daily Routine and Incident Management, Enhanced
Users with color deficiencies now have a built-in theme to ease navigating within Icinga Web 2. Also, our forms got a long overdue re-design and now look less boring. Though, the best of all features is that clicking while holding the Ctrl-key now actually opens a new browser tab! Lost comments? No more. Defining an expiry date again? No more!
- Add colorblind theme #3743
- Improve the look of forms #3416
- Make ctrl-click open new tab #3723
Stay Focused – More Room for More Important Stuff
Some of you know that some checks tend to produce walls of text or measure (too) many interfaces. Now, plugin output and performance data will collapse if they exceed a certain height. If necessary they can of course be expanded and keep that way across browser restarts. The same is also true for the sidebar. (Though, this one stays collapsed)
- Persistent Collapsible Containers #3638
- Collapsible plugin output #3870
- Collapsed sidebar should stay collapsed #3682
Markdown – Tables, Lists and Emphasized Text The Easy Way
Since we now have the possibility to collapse large content dynamically, we allow you to add entire wiki pages to hosts and services. Though, if you prefer to use a real wiki to maintain those (what we’d strongly suggest) it’s now easier than ever before to link to it. Copy url, paste url, submit comment, Done.
- Make notes, comments and announcements markdown aware #3814
- Transform any URL in a Comment to a clickable Link #3441
- Support relative links in plugin output #2916
Things You Have Missed Previously
The tactical overview, our fancy pie charts, is now the very first result when you search something in the sidebar. If you’ll see two entirely green circles there, relax. Also overdue or unreachable checks are now appropriately marked in list views and the service grid now allows you to switch between everything or problems only.
- Add tactical overview to global search #3845
- Servicegrid: Add toggle to show problems only #3871
- Make overdue/unreachable checks better visible #3860
Authorization – Knowing and Controlling What’s Going On
Roles can now be even more tailored to users since the introduction of a new placeholder. This placeholder allows to use a user’s name in restrictions. Things like
_service_responsible_person=$user:local_name$ are now possible. The audit log now receives failed login-attempts, that’s been made possible since hooks can now run for anonymous users.
- Allow roles to filter for the currently logged in user #3493
- Add possibility to disable permission checks for hooks #3849
- Send failed login-attempts to the audit log #3856
See also the audit module which got an update and is required for #3856 to work.