2.11.0 unveiled a critical bug in the cluster config sync. Before you panic – upgrading the config master as documented before any other instance keeps you safe. The 2.11.1 release is for everyone upgrading their satellites/agents first. For those who cannot do a major upgrade on their masters, we also release 2.10.7 where we added an additional fix to prevent this behaviour.
The online documentation additionally covers troubleshooting topics for configuration problems with command endpoint agents and zone-in-zone synchronization.
Check the Changelog for 2.11.1 and 2.10.7 prior to upgrading. Packages are available on packages.icinga.com as always.
We are happy to announce a new bugfix release for Icinga Web 2. Official packages are available on packages.icinga.com. You can find all issues related to this release on our Roadmap.
Less Smoky Database Servers
The release of v2.7.1 introduced a change which revealed an inefficient part of our database queries. We made some
general optimizations on our queries and changed the way we utilize them in some views. The result are faster
response times by less work for the database server.
- Consuming more CPU resources since upgraded to 2.7.1 #3928
Anarchism Infested Dashboards
Recent history already showed signs of anarchism. (Pun intended) A similar mindset now infested default dashboards
which appeared in a different way than before v2.7.0. We taught their dashlets a lesson and order has been reestablished
- Recently Recovered Services in dashboard “Current Incidents” seems out of order #3931
We improved the host and service distinction with v2.7.0. The downtimes list however got confused by this and didn’t
knew anymore how to combine multiple downtimes. If you now instruct the list to select multiple downtimes this works
again as we removed the confusing parts.
- Selection of multiple downtimes fails #3920
Over the last four years, the Icinga Director has grown from an optional configuration add-on to a mature Software product with lot‘s of features. Most Icinga installations are now driven by the Director, no matter whether they are small or huge, manually curated or fully automated.
But it will not stop here. Many cool ideas are eager to finally become reality. Director v1.7 is a huge step in that direction, as it lays the foundation for a completely new type of features. We are now able to delegate complex tasks to a dedicated background daemon that has been introduced with this version. New library modules have been published, allowing us to share cool bleeding edge funtionality among different modules in a more efficient way.
The UI comes with new features like cloning a Service to a different Host, Downtime-related features or autosuggestion for fields based on Data Lists. Please check our changelog for more details.
Some artificial limits have been increased. Host address length does no longer limit cloud people who want to fire their checks against super long hostnames instead of IPs. For large setups with tens of thousands of hosts in a single zone the maximum size for generated single configuration files has been increased.
Import and Sync
Ever wondered what WOULD happen when pressing “Trigger this Sync”? Sync now offers a preview, showing what to expect. This works fine even if thousands of objects might be subject to change. New Property Modifiers have been added, and a generic REST API Import Source has been implemented. Import Sources will now provide detailled errors hinting right to the problem in case of erraneous data.
Available since v1.6.0, configuration baskets got a lot of interest. They became a valuable tool when it goes to exchange Service Sets or Automation Pipelines, either with the Community, when asking an Icinga Partner for support or just between your very own testing/staging and production systems. In this release many related issues have been addressed, and more object types are now supported.
Extensions and Hooks
Icinga Partners and Community members are building products based on Icinga and the Icinga Director. To satisfy their needs a couple of new Hooks have been introduced. For example it’s now possible to run custom code at deployment time. Experienced developers are allowed to do black magic by extending many of our web forms.
Last but not least the German translation has been refreshed, and we have now a Japanese translation. Many thanks to Chisato Hashimoto!
Get it now
As always, please check our Upgrading Documentation and then have fun with the shiny new Icinga Director v1.7.0.
Now we are here, after many months of development – we proudly release Icinga 2.11 available today.
It has been an emotional ride with many changes under the hood. The most obvious change is that Icinga’s distributed cluster operates more stable, the past quirks with hanging certificate signing requests or dead-locked TLS handshakes are now gone. This required us to go an unusual route: Evaluate new libraries and programming techniques in order to replace hand-written lower layered code, with later replacing the entire code base for the network stack operations in Icinga. This is a massive effort in quality and stability where users had called out for 3.0 already.
The core components of Icinga are checks and notifications. Thanks to sponsoring from a customer, Icinga now immediately send notifications after a downtime ends whenever hosts/services are still NOT-OK. Lost notifications during restarts in HA zones also have been tackled and fixed.
In the past years, we’ve seen operation problems with the config sync inside the cluster system. This has been tailored to a more robust deployment with staged syncs as well as preventing unwanted sync loops for comments/downtimes. The local storage for runtime created objects also is more robust. Also, note additional HA capabilities for features like Graphite or Elasticsearch.
We care about security. The setup steps with TLS certificates and signing requests in addition to the trusted zone hierarchy is necessary to keep your monitoring data safe. Icinga can see many details in your environment and must not be compromised. Taking things serious, Icinga 2.11 enforces TLS 1.2 as a minimum and also hardens the available cipher lists by default. Older agent versions still work granted that they support TLS 1.2 (OpenSSL 1.x.y).
More Icing on the cake
Icinga aims to run in small environments as well as large distributed enterprise systems. With the general availability of Debian Buster on Raspberry Pi, Icinga 2.11 is ready for your home lab and IoT setups.
To all the fans of on-demand certificate signing: You’ll get more tooling to manage/revoke satellite and agent certificate requests. A small but nifty addition are environment variables which can be retrieved with the new getenv() DSL function – for sensitive data or getting the path of HOME into your command’s context.
A small but gentle addition is the “all_services” attribute when setting a host in downtime with the REST API. Aside from the more visible fixes, many changes have been applied to improve our code base for the next years and avoid future bugs. This includes more unit tests e.g. for timeperiod ranges or version comparisons in the built-in icinga check.
Coming late to the party, we’ve fixed crashes from Nessus scanners on Linux and following up on Windows. Well, this wasn’t easy – a technical monster to debug and fix. Cheers to our friends Stefan and Philipp in Linz who have helped us to reproduce the issue. The technology now used in Icinga 2.11 proves the rule for scaling monitoring for the next decade.
Icinga uses the well-known Boost library as part of the network stack rewrite, thanks to everyone who contributes to their development. Also we’d like to cordially say thanks to Niels Lohmann for the magic JSON library, and Nemanja Trifunovic for making UTF8 handling in C++ a breeze. And to you, our fellow community contributors!
In the late development stages for 2.11, we’ve also tackled a problem with systemd reloads resulting in a killed process. The only way to prevent this from happening is actually a new feature: A real umbrella process managing the reload signal. Turns out, this implements a long standing feature request for running Icinga in a (Docker) container.
Docs, docs, docs
Our community forum with many great users and contributors helping each other raised the bar for the documentation. We are addicted to making the Icinga experience as easy as possible, and for 2.11, we have improved the documentation in many ways: From the basic setup to configuration to distributed environments to service monitoring to features and addons.
This includes a clear cut specification of the plugin API, with best practices from experienced plugin developers. Speaking of development, packagers and future developers are key to Icinga’s success. The development chapter as been overhauled to motivate everyone out there. Those wanting to learn about the design decisions, the technical concepts chapter highlights new bleeding edge insights.
Get Icinga 2.11, check the Changelog & upgrading docs and say cheers to the biggest release we’ve ever done.
Some say you have to move all of your server infrastructure into the cloud. Others counter that you should keep your data safe and secure in your own datacenter. And then there are many people in between who use cloud services as an addition to their self-hosted servers. In fact, there’s no right or wrong, because as always in IT: It depends. We at Icinga always try to find a way to make everyone happy with their monitoring – be it in the cloud or on premise. Today we’re pleased to announce version 1.0 of our Icinga Module for AWS!
Automation and integrations are important parts of every Icinga setup since ever then. The Icinga Module for AWS follows this path by embedding into your existing monitoring setup. The module is an addition for the well-known Icinga Director, whose main purpose is to manage Icinga configuration at a glance – the Icinga Module for AWS is yet another Director Import Source. Import sources are a crucial factor of the Director as they enable you to import data from multiple data providers. The collected data is transformed into Icinga 2 configuration. Learn more about import sources and automation in our guide Monitoring Automation with Icinga – The Director.
This module collects data about your EC2 Instances, RDS Instances, Load Balancers and Auto Scaling Groups. The Director uses the regularly collected information to create Icinga 2 configuration out of it. Since this can be automated, you can just continue on spawning and deleting new instances in AWS without having to take care about if they’re going to be monitored or not – without any human interaction.
Import EC2 Instances
The following example demonstrates how you can use this module to import details about your currently running EC2 Instances into Icinga. Before you get started, make sure you have your AWS Access Key by hand. This is required, so you can establish a connection to your account and pick up the information we need for monitoring. Make sure to add your Key details to the modules configuration at “Configuration > Modules > aws > AWS Keys”.
Installation and Configuration
Extract or clone this module to your Icinga Web 2 module path. The directory name must fit the module name, aws. This would usually lead to
Additionally, the module requires the AWS PHP SDK v3 which you have to install manually. The easiest way to do this is by downloading it from the release page and unpacking it into
Configuring the Import Source
The first step is to create a new Director Import Source:
- Import source name: A general identifier, choose something that you will recognise later
- Source type: Select
- AWS region: You may create an import source for each region you use
- AWS access method: Select your previously configured AWS Key from the dropdown
- Object type: We use
EC2 Instances for this demo purpose, you may create multiple import jobs for each object type you want to import
Save the details and open the newly created Import Source again. We now continue with adding a Modifier.
Import only running Instances
Director Modifiers are used to handle the incoming data and transform it or filter it by certain conditions. Since you usually want to monitor only running instances, we’re using a Modifier to filter for the state of the EC2 Instances.
- Property: We refer to the
status property, in order to filter by states
- Modifier: The
Black or White-list modifier allows you to whitelist only certain instances
- Filter: We filter by
running systems only
- Policy: Here we decide to keep only matching rows, therefore only running systems are whitelisted
Hit the save button to save your Modifier.
Trigger Import Run
You’re now ready to trigger the first import run. This will not create any Icinga configuration yet, but allow you to see a preview of the collected information in the “Preview” tab. There you can now see a variety of data that was synchronised from your AWS account. All of these information can be used to create Icinga hosts for example. Additionally you may use part of the data and add it as custom variables to your monitoring objects.
Your next step is to create a Sync Rule which will use the gathered AWS data to generate Icinga configuration. If you’re not familiar with Sync Rules, you may refer either to the Director Documentation to learn about it or read our guide Monitoring Automation with Icinga – The Director
We are proud to announce that icinga.com is now also hosting integrations of hardware for monitoring environmental sensors and alerting via text message. The devices from the manufacturers listed on Icinga Integrations can easily be implemented into your Icinga monitoring by using plugins provided on Icinga Exchange.
Alerts via Text Message
Basically, there are two types of devices, which are most suitable for integration with Icinga. First group of them are gateways for sending SMS as alert messages. These gateways work with standard SIM cards and can run on bands in 4G, 3G and 2G networks. These devices also provide more functionality than only sending messages, they are also capable of converting SMS to E-Mail or the other way round or using filters and routing for sending messages only to a certain person or group of persons (phonebook listing). Furthermore, the gateways have their own monitoring server which means that they can automatically send notifications for example in case of loosing connection to the network.
In order to smoothly add these gateways into one’s productive processes, there exist for example plugins for directly connecting to your Microsoft Exchange for full usage of E-Mail to SMS conversion. Integrated with Icinga 2 you cannot only monitor the device itself but also use it in order to forward notifications generated by Icinga. The two main manufacturers of these gateways are Braintower located in Germany and SMSEagle with their headquarters in Poland.
Second type of integratable devices are mainly meant for measuring and surveillance. These consist most of the time of a base unit and several ports for connecting sensors. The base units can either directly communicate via LAN or via GSM in case of remote locations. For full environmental monitoring, there are plenty of sensor types that can be attached:
- combination of temperature and humidity
- air pressure
- air flow
- water and leakage
- smoke and Gas
- levels of liquids like pertrol
- movement, vibration and opening of doors
The main manufacturers for measuring devices are currently AKCP, GUDE, HW group and Tinkerforge. For all these you can find monitoring plugins on Icinga Exchange. The manufacturer GUDE also offers power supplies that cannot only monitor voltage, but are also capable of remote controlling plugs and devices.
Integrating Tinkerforge with Icinga
In order to have a closer look at how these devices can be integrated with Icinga 2, Tinkerforge is one of the best examples. The whole plugin project can be found at https://exchange.icinga.com/netways/check_tinkerforge and shows not only the integration but also offers an insight into the functionalities of the device.
For using the check_tinkerforge plugin it is required to run Python 2.7+ and to have the tinkerforge Python library from Pypi installed. After this, installation is done with
pip install tinkerforge and moving the plugin into the Icinga
Now let’s have a look at the options of the plugin:
$ ./check_tinkerforge.py --help
usage: check_tinkerforge.py [-h] [-V] [-v] -H HOST [-P PORT] [-S SECRET]
[-u UID] -T TYPE [-w WARNING] [-c CRITICAL]
-h, --help show this help message and exit
-V, --version show program's version number and exit
-H HOST, --host HOST The host address of the Tinkerforge device
-P PORT, --port PORT Port (default=4223)
-S SECRET, --secret SECRET Authentication secret
-u UID, --uid UID UID from Bricklet
-T TYPE, --type TYPE Bricklet type. Supported: 'temperature', 'humidity', 'ambient_light', 'ptc'
-w WARNING, --warning WARNING Warning threshold. Single value or range, e.g. '20:50'.
-c CRITICAL, --critical CRITICAL Critical threshold. Single vluae or range, e.g. '25:45'.
-t TIMEOUT, --timeout TIMEOUT Timeout in seconds
As the TinkerForge device supports sensors like temperature or humidity as well as other types, we need to indicate what sensor should be checked by using the
check_tinkerforge.py -H 10.0.10.163 -T temperature -w 23
WARNING - Tinkerforge: Temperature is 24.75 degrees celcius|'temperature'=24.75
In case you have more multiple sensors of one type, you are required to indicate the UID with the
--uid operator in order to correctly identify the sensor in question. Furthermore, thresholds can either be indicated as single values or value ranges and by using the usual operators for warning and critical.
For integration of the sensors into your Icinga 2 configuration, you need to create a
CheckCommand object like for other check plugins. The device itself will be created as
Host object and all sensors should be covered with an
apply Service. For your comfort you can find an example config for TinkerForge here. This example can directly be used, it is only necessary to change the parameters and in case you have multiple sensors of one type, to add the UIDs.
All manufacturers mentioned in this blog post can be found on the Icinga integrations website including information on the companies, the devices and the links to the corresponding monitoring plugins.