In our ongoing efforts to make it easier to automate monitoring environments we recently introduced a new module for Icinga Web 2.
Icinga Certificate Monitoring
This module is first and foremost a platform which lets you have an overview over all the certificates you are using in your environment to prove the identity of your devices. You can take a quick glance or a very detailed look at them. It will help you to know exactly how your certificates are distributed based on the signing certificate authority, the used algorithms and key strengths as well as which certificate expires next.
It helps with automation
You don’t need to register each device or certificate by hand. The module will scan the networks you’ll provide it with and harvest any certificates it encounters. Whether it does this regularly or on demand is fully up to you.
Networks are provided by setting up jobs. These jobs define several IP ranges in CIDR notation and ports. Schedules in CRON format may also be set for jobs so that this module’s daemon runs them regularly.
Integrates well with your environment
Cloud hosting and virtual machines are on the rise for a long time now and with SNI (Server Name Indication) a single host may easily present different certificates on the same endpoint. In order to facilitate this, the module can be told to scan an endpoint multiple times by setting up SNI maps.
Installed alongside the monitoring module, Icinga Certificate Monitoring even accesses its database backend to fetch SNI information.¹ This will help to match results found in the scan process to already known hostnames in your monitoring environment.
Don’t miss to roll out new certificates
Let’s be honest, everyone has sometimes missed to re-new or replace expired certificates. The module provides detailed views showing you exactly which certificates require your attention.
Certificate Chain Health
Take advantage of your favorite monitoring tool
Though, if you’re not proactively looking at the user interface the check command shipped with this module may help with setting up notifications in Icinga.
Monitoring Service List
Bridging the gap with the Director
With all this talk about automation one has to wonder how to establish a link between this module’s knowledge about certificates and Icinga’s configuration. You’re right if you think of the Director’s import and synchronization functionality now.
The module lets you easily import known hosts or certificates with its own import sources. By setting this up you only have to define jobs for it and all the rest is handled automatically.
¹Available with Icinga Web v2.7 (Scheduled for release mid 2019)
This release fixes a regression introduced with namespaces in 2.10 where advanced permission filters could result in a crash with many concurrent requests. It also fixes the problem that permission filters are sometimes not applied correctly. Thanks VSHN for sponsoring the development time!
Another problem hidden in the dark was introduced in 2.9 with changed reload signals. This caused the reload shutdown handler to deactivate host and hostgroup objects in the IDO database backend, with later activating them on startup reconnect. Large scale environments would suffer from seconds to minutes not seeing objects in Icinga Web 2 then.
Last but not least, our road to better code quality unveiled that certain compilers don’t treat us well with shared pointers references and resulting crashes. This release fixes crashes with logrotate signals and other timer related problems, e.g. state file handling.
We’ve also added some documentation improvements for the REST API, object types, feature overview and technical concepts with JSON-RPC cluster messages – inspired by the feedback from our Icinga Discourse 🙂
Checkout the full changelog for 2.10.5 here. Please note that SLES11 and Ubuntu14 packages are not available anymore.
Special thanks goes out to Elias Ohm from novomind AG 🙂 He’s been debugging and fixing the many issues with us together, magnificent team work!
Note: API and cluster/agent related problems will be fixed in 2.11, requiring us to rewrite large parts of the code base. You can help test the snapshot packages and provide feedback. Thanks in advance!
We are happy to announce a new bugfix release for Icinga Web 2. Official packages are available on packages.icinga.com. Community repositories might need a while to catch up.
You can find issues related to this release on our Roadmap.
LDAP – Community contributions, that’s the spirit
With the help of our users we’ve finally fixed the issue that defining multiple hostnames and enabling STARTTLS has never properly worked. Also, they’ve identified that defining multiple hostnames caused a customized port not being utilized and fixed it themselves.
There has also a rare case been fixed that caused no group members being found in case object classes had a different casing than what we expected. (Good news for all the non-OpenLdap and non-MSActiveDirectory users)
- LDAP connection fails with multiple servers using STARTTLS #3639
- LDAPS authentication ignores custom port setting #3713
- LDAP group members not found #3650
We take care about your data even better now
With this are newlines and HTML entities (such as ) in plugin output and custom variables meant. Sorry if I’ve teased some data security folks now.
- Newlines in plugin output disappear #3662
- Windows path separators are converted to newlines in custom variables #3636
- HTML entities in plugin output are not resolved if no other HTML is there #3707
You’ve wondered how you got into a famous blue police box?
Don’t worry, not only you and the european union are sometimes unsure what’s the correct time.
- Set client timezone on DB connection #3525
- Ensure a valid default timezone is set in any case #3747
- Fix that the event detail view is not showing times in correct timezone #3660
UI – The portal to your monitoring environment, improved
The collapsible sidebar introduced with v2.5 has been plagued by some issues since then. They’re now fixed. Also, the UI should now flicker less and properly preserve the scroll position when interacting with action links. (This also allows the business process module to behave more stable when using drag and drop in large configurations.)
- Collapsible Sidebar Issues #3187
- Fix title when closing right column #3654
- Preserve scroll position upon form submits #3661
Corrected things we’ve broke recently
That’s due to preemptive changes to protect you from bad individuals. Unfortunately this meant that some unforeseen side-effects appeared after the release of v2.6.2. These are now fixed.
- Multiline values in ini files broken #3705
- PHP ini parser doesn’t strip trailing whitespace #3733
- Escaped characters in INI values are not unescaped #3648
Though, if you’ve faced issue #3705 you still need to take manual action (if not already done) as the provided fix does only prevent further occurrences of the resulting error. The required changes involve the transformation of all real newlines in Icinga Web 2’s INI files to literal \n or \r\n sequences. (Files likely having such are the roles.ini and announcements.ini)
I’m not going to list all benefits of automating your monitoring system. If you’re here and reading this, you are most likely very aware that maintaining a large infrastructure is a big challenge.
Automating the monitoring process for a huge amount of servers, virtual machines, applications, services, private and public clouds was a main driver for us when we decided to build Icinga 2. In fact, monitoring large environments is not a new demand for us at all. We experienced this challenge in tandem with many corporations for many years. Finally, it lead us to build features like our rule based configuration, Icinga’s REST API and various modules, cookbooks, roles and playbooks for different configuration management tools.
All these methods make it easier to automate monitoring in their own particular way. We have built multiple ways to automate monitoring because there is not only one way to do it right. As usual in the IT field “it depends”. Depending on your infrastructure either one way or another may be the right way for you to automate your monitoring.
Beyond the Static
With all this in mind we have created the Icinga Director 3 years ago. Director is a module for Icinga that enables users to create Icinga configuration within a web interface. The Director utilizes Icinga’s API to create, modify and delete monitoring objects. Besides the plain configuration functionality, the Director has a strong focus on automating these tasks. It provides a different approach that goes far beyond a well-known static configuration management. With thoughtful features the Director empowers operators to manage massive amounts of monitoring objects. In this article I put my attention on the automation functionality of the Director, even though creating single hosts and services manually is also possible of course.
Step 1: Import
In almost every company a database exists, or something similar, that holds information about the currently running servers and applications. Some use dedicated software to manage this information. Others use tools they have built themselves or rely on their config management tools.
While most people refer to this database as a CMDB there are also other common terms. In theory only one database exists within an organization, allowing everyone involved to manage the information in a single place. In practice, most organizations spread the data across a number of different tools. While there are some professional approaches, there are also some not-so-professional ways of maintaining this data. Have you ever seen someone keep their IP addresses and locations of servers in an Excel file? We’ve seen it all.
Creating the source
The Director contains a feature called Import Source. It allows you to import all kinds of data from many different data graves. The data does not have to be in the Icinga configuration format, the Director will take care of that later. For a start, you only need some kind of data.
In my very simple example, I’m using a MySQL database, which is a common storage for this type of information. My database
cmdb contains only one table
hosts that holds everything I know about my servers. For demo purposes, this is perfect.
Add Import Source
Import Source Preview
Creating the import source requires access to the database server. The credentials are stored in an Icinga Web 2 resource, therefore they are re-usable. After triggering a Check for changes you can preview the data set in the Preview tab. If everything that you need is included, you can trigger the import run which actually imports the data.
Starting from here your data is generally available and you can create Icinga configuration out of it. The properties may also be modified during the import, but I leave them as they are for now. Learn more about available modifiers
Using SQL, LDAP or else
For my import source I used the source type SQL which is built-in and available by default. You can use other source types as well, for example LDAP. That allows you to import not only objects that have to be monitored, but also users from your LDAP or Active Directory and use the contact information for alerts.
Of course, you can also use other import sources, such as plain text files, PuppetDB, vSphere or AWS. New import sources are added to the Director as Modules, which you could also write yourself. Our lovely community is continuously extending the Director with new import sources as well, for example with import sources for Microsoft’s Azure or Proxmox VE.
Step 2: Synchronise
After a successful import I am able to continue with basic config synchronisation. Syncing configuration means, that you use the imported data to generate Icinga configuration out of it. Generally speaking, you map your data fields into Icinga objects and properties.
Some of the data I imported is easy to map, such as hostname and IP address. Icinga has pre-defined config parameters for those. Others, like the location and environment of the servers are mapped to custom variables. Custom variables are something like tags, but on steroids. They accept plain strings as values but also booleans, arrays and even dictionaries (hashes). I know, this sounds crazy. Custom variables are usually used to store meta information about your objects. This information you might later want to use to create rules which in turn define what should be monitored or who should receive alerts.
But first things first.
Creating a new Sync requires some input and some decisions. You define what kind of monitoring objects you’re going to create out of your imported data. This can be Hosts, Endpoints, Services, User, Groups and so on. In my example I simply choose to create some host objects.
Then you decide what should happen with existing monitoring configuration. Shall it be replaced with the new one, merged or ignored? Once the Sync is created you can finally start to tell the Director how to map your data to Icinga configuration attributes. This is done within the Properties type by creating new sync property. For every column in my table I create a sync property. Have a look at the images for details.
Add Sync Rule
Add Sync Property
Sync Properties Overview
Checking for changes will only do a dry run to tell you if there are any changes available. Triggering the Sync actually synchronizes the new configuration with the existing one and automatically creates new entries in Director’s activity log. At this point, everything is ready to be deployed to Icinga.
Step 3: Deploy
The Director’s activity log shows precisely what changes are waiting to be deployed to Icinga. Clicking on each element displays the exact diff between old and new configuration. This diff format may be familiar to you from Git for example. Another point that may be familiar is the whole history of deployments which you can see within the activity log.
Travelling in time
It contains every configuration change you ever made with the Director. It lets you travel back in time and deploy old configs if necessary. The Director’s history of deployments basically works like your Git history: You can do a diff between certain deployments to track the changes and see which user deployed which configuration at what time.
The simplest way of deploying your new configuration is by just clicking Deploy pending changes. Your config will be pushed to Icinga and validated. If everything is fine it will be in production within seconds. If there’s anything wrong, you will receive a log with the details and your running configuration remains untouched.
Fully Activity Log
Config Version Diff
As I mentioned in the beginning, automation is a key aspect of the Director. The steps described so far (Import, Synchronise and Deploy) can all be automated once they are fully configured. The automation is done by creating Jobs, with each Job running one certain task in a specified period.
The frequency of a Job is freely configurable. It may run every minute or only once a day. You create a Job for each task or only for certain tasks.
Hint: You can use a simple cronjob to run the Jobs. To fully leverage the Director’s Job functionality in the web interface, you have to start a separate daemon as described in the documentation.
The workflow described above was only a sneak peek into the capabilities of the Director. This scenario however demonstrates very well the basic functionality and what you can do with it. There are many more features left out in this article like modifying imported data, merging data from multiple data sources, filtering or creating and using custom data sets.
Managing and maintaining large server infrastructures is a very complex thing to do and there’s a good reason why whole teams are required to do so. I would lie if I told you that monitoring of such large setups is easy. But with the right toolset it is definitely doable! Check out the full Director Documentation to get started with your monitoring automation project.
We will discuss the challenges of monitoring large infrastructure at the upcoming Icinga Camps as well. Join us for a full day filled with talks and discussions about and around Icinga. Meet new people and get to know others from the same field.
We’re happy to announce that we released an early version of Icinga Reporting today! With this release we create the foundation for an overall reporting functionality for Icinga by introducing a new way to work with collected data. At the same time we are also publishing the first use case of Icinga Reporting which enables you to calculate, display and export SLA reports for your hosts and services.
Icinga Reporting is the framework and foundation we created to handle data collected by Icinga 2 and other data providers. By definition Icinga Reporting does not collect or calculate any data. The framework processes usable data from data providers such as Icinga’s IDO or Icinga Web 2 modules and makes them available in different formats. The first version can display the data directly within the Icinga web interface or export it to PDF, JSON or CSV format. With scheduled reports you can receive the prepared data periodically via email.
Create Host Report
Create Scheduled Report
The IDO is the database where Icinga 2 stores all the status data it collects. It is also the first data provider for Icinga Reporting. We calculate the availability of your hosts and services over a certain amount of time and return a percentage value. This allows you to evaluate and compare the accessibility of you applications and network devices. You can use the data to check if you’re meeting your SLA (Service-Level-Agreement) and share it with your team and managers.
Open Source Projects
Icinga Reporting consists of multiple projects. We’re continuously working on updating and extending our existing Modules to provide data for Icinga Reporting. This release is at a very early stage and your feedback is very welcome and appreciated.
Join our community on community.icinga.com and have a chat with us about your reporting use cases and challenges! We will discuss Icinga Reporting on our upcoming Icinga Camps as well. The CfP for Stockholm, Milan and Zürich is still open for those who are interested in speaking at these events.
This release provides fixes for the InfluxDB and Elasticsearch metric writers. If you’re using TLS connections, the latter were not closed correctly. In addition to these fixes, we’ve also backported fixes for delayed and one-time notifications. Special thanks to mdetrano for being patient and testing this one.
Additionally, the Windows wizard has been updated and check_perfmon now supports non-localized performance counters. One of our customers sponsored improving mass-creation of downtimes via the REST API in HA enabled clusters, thank you.
Official packages are available on packages.icinga.com, have been pushed to Chocolatey and Raspbian will follow soon. Meanwhile check the changelog for v2.10.4.