In our ongoing efforts to make it easier to automate monitoring environments we recently introduced a new module for Icinga Web 2.
Icinga Certificate Monitoring
This module is first and foremost a platform which lets you have an overview over all the certificates you are using in your environment to prove the identity of your devices. You can take a quick glance or a very detailed look at them. It will help you to know exactly how your certificates are distributed based on the signing certificate authority, the used algorithms and key strengths as well as which certificate expires next.
It helps with automation
You don’t need to register each device or certificate by hand. The module will scan the networks you’ll provide it with and harvest any certificates it encounters. Whether it does this regularly or on demand is fully up to you.
Networks are provided by setting up jobs. These jobs define several IP ranges in CIDR notation and ports. Schedules in CRON format may also be set for jobs so that this module’s daemon runs them regularly.
Integrates well with your environment
Cloud hosting and virtual machines are on the rise for a long time now and with SNI (Server Name Indication) a single host may easily present different certificates on the same endpoint. In order to facilitate this, the module can be told to scan an endpoint multiple times by setting up SNI maps.
Installed alongside the monitoring module, Icinga Certificate Monitoring even accesses its database backend to fetch SNI information.¹ This will help to match results found in the scan process to already known hostnames in your monitoring environment.
Don’t miss to roll out new certificates
Let’s be honest, everyone has sometimes missed to re-new or replace expired certificates. The module provides detailed views showing you exactly which certificates require your attention.
Certificate Chain Health
Take advantage of your favorite monitoring tool
Though, if you’re not proactively looking at the user interface the check command shipped with this module may help with setting up notifications in Icinga.
Monitoring Service List
Bridging the gap with the Director
With all this talk about automation one has to wonder how to establish a link between this module’s knowledge about certificates and Icinga’s configuration. You’re right if you think of the Director’s import and synchronization functionality now.
The module lets you easily import known hosts or certificates with its own import sources. By setting this up you only have to define jobs for it and all the rest is handled automatically.
¹Available with Icinga Web v2.7 (Scheduled for release mid 2019)
This release fixes a regression introduced with namespaces in 2.10 where advanced permission filters could result in a crash with many concurrent requests. It also fixes the problem that permission filters are sometimes not applied correctly. Thanks VSHN for sponsoring the development time!
Another problem hidden in the dark was introduced in 2.9 with changed reload signals. This caused the reload shutdown handler to deactivate host and hostgroup objects in the IDO database backend, with later activating them on startup reconnect. Large scale environments would suffer from seconds to minutes not seeing objects in Icinga Web 2 then.
Last but not least, our road to better code quality unveiled that certain compilers don’t treat us well with shared pointers references and resulting crashes. This release fixes crashes with logrotate signals and other timer related problems, e.g. state file handling.
We’ve also added some documentation improvements for the REST API, object types, feature overview and technical concepts with JSON-RPC cluster messages – inspired by the feedback from our Icinga Discourse 🙂
Checkout the full changelog for 2.10.5 here. Please note that SLES11 and Ubuntu14 packages are not available anymore.
Special thanks goes out to Elias Ohm from novomind AG 🙂 He’s been debugging and fixing the many issues with us together, magnificent team work!
Note: API and cluster/agent related problems will be fixed in 2.11, requiring us to rewrite large parts of the code base. You can help test the snapshot packages and provide feedback. Thanks in advance!
Last week I told Blerim that Icinga came to life on 6.5.2009 … really, 10 years already?
A small group of Nagios community members stood up and wanted to create more than a monitoring core with an enhanced web interface. APIs, integrations, backends, scaling, metrics, … A lot of passion and love was received from the community and actually, the Nagios ecosystem became more active again. We tried many things to improve the existing code base and limitations in both backends and frontends. TL;DR – at some point we decided to throw it away and start fresh. The rest can be read on our blog 🙂
service icinga restart
Back in 2012, Gunnar, Bernd and Marius paid a visit to Vienna. They had something with them what is known to be the first prototype of Icinga 2, codename “Strawberry“. Moving from C into C++ was a whole new world, Golang didn’t really exist back then. Oh, it can run on Windows too. Oh, such speed and performance. Oh, a new playground for our crazy ideas.
Before its first 2.0 release five years ago we had many design rounds with service sets on the Host object, virtual host checks from services and so on. This changed quite a bit after strong opinionated discussions in early 2014. Those are also the origin of the magic apply rules, with mimicking the “put a service on a hostgroup, each host member inherits it” object trick from Icinga 1. Nowadays you can generate monitoring configuration and make things easier.
“Monitoring as a code” became a thing when we had created our own DSL (domain specific language) which added functions, conditions, even loops from community feedback. We’ve also learned a lesson here with the Director released after 2.4 – not everything from inline DSL code is suitable for a configuration frontend. Also that we now have 3 APIs – Core, Web, Director – users might be confused. This is a task we will be working on in the future to unite the product stack better.
Speaking of web and modules, Icinga Web 2 started with the requirement of a general framework including web and CLI based on the Zend framework. IIRC this was 1,5 to 2 years after Icinga 2 kicked off, also explaining the version difference – 2.7 vs 2.11. Web version 1 had many short-comings and was complicated with extensions – unless you’re an experienced developer. With the power of making everything a moduie, even monitoring, the Icinga Web framework inspired the many of you.
Community work is a passion
One of the first community modules I’ve seen was the Globe module which I had shown at Icinga Camp Berlin 2017. Nicolai was so inspired to start the development of the map module. I was extremely happy that I’ve tested it quite extensively and sent a docs PR. Literally the same happened with Carsten creating the famous Grafana module – a gentle README, release hints and troubleshooting – the first impression counts. Did I mention the awesome themes already?
Pushing community ideas and spirit forward is one of the most important things in the way we have become Icinga. Without you and your shared love and criticism, we wouldn’t be here and still be sitting somewhere around the globe ranting about tools.
Back then, our community was a bit split, and not everyone had the time to answer questions on IRC, the mailing lists and the community forum at Monitoring Portal. While we want to be transparent with our development workflow, release dates and design ideas, it’s always time consuming and hard to catch up. With Icinga now supporting distributed environments with zones, endpoints, and web views and configuration forms, we needed something where text formatting and screenshots really integrate well. In late 2018, we’ve decided to create our own Icinga Discourse inviting the team, partners and users to discuss. I just love it, helping others really is a good feeling.
As you may know, the majority of the Icinga development is and was funded by NETWAYS, up until 2012 the University of Vienna sponsored my time. This is paid time and helps with creating the foundation for new features and maintenance. We also raised awareness on why funding and sponsoring is a good thing – if there is a bug biting you, or a missing feature you’d like to use in production, you can speed things up by requesting a quote, or even, by getting a support subscription through one of our partners. Sounds expensive? Maybe. Helps funding the future? Definitely.
Still, we love community contributions and value your (spare) time a lot. Sometimes contributions are hard to achieve as the code is not clear, or you actually don’t know where to start. In the past, we had our own Git server and ticket system. It worked somehow – with the move to GitHub and social coding with Pull Requests, reviews, discussions and a sound platform our daily work on Icinga has become a breeze. The help of issue templates, continuous integration (CI) on when a PR is created, and inline code comments puts the focus back on development.
Lastly, we’ve started documenting our core technical concepts and improved the development guidelines. This now also includes instructions for nightly snapshot builds.
Turning back the time a bit
Icinga 1 didn’t have packages at first glance. You had to call configure/make and pass some parameters for development libraries and headers, sometimes it didn’t work. The core was to compile and then it broke on your system, but the developer’s system worked fine. Ok, let’s create a VM for SuSE in Virtualbox … well, where is my yum?
One of the first ideas was to invite upstream packagers onto the team and learn from each other. Our many releases couldn’t reach stable distributions that fast, so we needed additional repositories. There’s Debian Backports, custom Ubuntu PPAs, EPEL for RHEL/CentOS and lastly, the SuSE OBS repos – not to forget all the other distributions in the Linux/Unix world, FreeBSD, OpenBSD, NetBSD, Gentoo, ArchLinux, … For the main officially supported platforms, we decided to create our own repository at https://packages.icinga.com.
Creating and maintaining packages next to the build infrastructure for each project, language and dependencies is a huge often underestimated effort. Our build server based on Jenkins made several iterations over the years, lately we decided to replace it with GitLab CI for less security risks and better configuration. CI and CD is just great, and with Docker containers nearly everything can be built – requiring your self-hosted infrastructure, maintained by the awesome managed service team at NETWAYS. This includes deployments with Foreman and Puppet, and of course monitoring with Icinga just like “eat your own dogfood” 🙂
Another thing which really was hard to achieve – ARM based architecture on Raspberry Pi. Sounds easy to just compile the binary packages on a different architecture. Actually we had to tackle compiler bugs and long running builds – we started a thing at the OSMC hackathon 2018 and Nicolai’s employer sponsored the last missing days on the road to official packages.
We can always improve
Looking back where we started 10 years ago, we’re in a very comfortable situation. This also includes making the initial setup a breeze, adding lots of documentation and helping on the community channels when things are on fire. With partners providing support, trainings and consultancy, the enterprise requirements are fulfilled even better.
In terms of new projects and ideas, it always turns out that we want to create 4 things, then 10 different customer issues arise, the proof of concepts don’t work out well and obviously, there’s only resources for one or two things being released later. Right on, many have asked about notification managers or mobile apps – that’s on hold for now, finishing the ongoing tasks with Core and Web major versions, Icinga DB and Icinga Reporting, and additional new module version for certificates (x509), Vsphere, Director and Business Process. Again, focus is required – you cannot manage 10 things in parallel. That being said, the monthly snap blog will be moved into a combined effort with the enhanced newsletter telling you about the latest and greatest developments. We also continue to have bi-yearly strategy workshops to refine future roadmaps and discuss changes in our ecosystem.
Those things are great to read, but even more, they are great to talk about. Our idea of meeting at Icinga Camps worked out pretty well, Berlin is by far the biggest one thus far. OSMC with the “State of Icinga” talk by Bernd is great and funny way to share the latest stories too. Our first Icinga team meeting was at OSMC 2009 with many more to come at Icinga Camps all over the world, where you’ll meet, chat and laugh. Our first Icinga Camp San Francisco 2014 happened at GitHub HQ. This is an incredible memory and kicked off the many community events after. In the past months, our community in Germany and Austria have been going strong with regular meetups, that’s truly inspiring. Not been there, or want to create your own meetup group? It is not too late.
We are also aiming for something bigger for everyone planning for next year: IcingaConf in Amsterdam in 2020.
Our way of not re-inventing the wheel but building our integrations with proven tools will continue. Today we are proud to enhance Elastic Stack with adding Icingabeat, Logstash outputs/filters and metric writers even. Adding monitoring as first class citizen in your DevOps lifecycle has become reality with official modules for Puppet, Ansible and Chef and a deeper integration into Foreman. If this isn’t enough, there’s more cloud provider integrations such as Terraform, AWS, Azure and whatnot. Enhancing metrics with Graphite and InfluxDB, provided with example Grafana dashboards and web module integrations. Dashing is still a thing, and reporting has come in its first early version for more SLA and visualization magic.
This is something we couldn’t imagine 10 years ago. We are also proud to work with many professionals and open source lovers all over the world, spreading the #monitoringlove.
The people behind, around and with Icinga
That’s not only me, I hear that often when people find my name on Google. This is a group of passionate people, each with their own character, ideas and responsibilities. Over the years, fellow friends went with us on our journey and left again for their new projects and private life.
Karo, who created the initial designs, blog posts and social media for Icinga. Amanda, who tackled technical problems and spread the love for Icinga on social media and press releases. Gunnar, who is the main architect of Icinga 2, and a passionate coder, the brain behind the DSL and many more. Hendrik who forked Icinga 1 as core and taught us how to use Git and debug the IDO backend.
Michael who worked many hours on Icinga Reports with Jasper, creating fancy web views. Thomas who took over the Oracle backend improvements for Core 1.x. Christian who tackled the Oracle backend in Icinga Web 1, followed by Jannis who enhanced version 1, created mobile 1 and helped create Icinga Web 2 back then. Scott who joined the Skype sessions from Australia, changing our German project into an English one.
Lara and Wolfgang who tackled the version 1 documentation in docbook xml, making our later decision for Markdown very easy. Christoph who created the first Icinga Core RPM packages, Shawn who tried to bring Icinga packages into EPEL, Sam helping here. Carl who’s been working on Solaris packages for 1.x. Nick and Tom paving the way for the first Puppet module. David who created the first VM appliances. Franz and Mike took care of test scripts, Carlos created the vim/nano syntax highlighting for the Icinga DSL, Jordan who’s been working on Docker containers.
Matthew who created the header status bar for Classic UI, Ricardo who took over improving the old CGIs with many enhancements. Rune who did everything testing and created the term “release fucker” back in the 1.x days – when developers didn’t test enough. Massimo helped with 1.x core issues back then. Alex created and maintained the Debian/Ubuntu packages for many years, Tim who did the same for SuSE. Gerd who worked on Icinga 2 reload and threading techniques. Simon who created the InfluxDB feature for Icinga 2.
Thank you deeply.
Many thanks also to Thomas Krenn who sponsored hardware for the initial build server, JetBrains, Navicat, Atlassian & Activestate for granting us open source project licenses back then.
Bernd, Marius and Julian are the remaining co-founders of Icinga. I was a little late to the party in May 2009, with the rest of us enjoying Icinga with our friends, partners and users every day. You’ll recognize Eric for our technical strategy and everything web and core, Blerim improving and managing all the products, helping partners, maintaining integrations, Tom pushing forward with designs and architecture and the shiny Director, VSphere, etc. modules, Markus taking care of QA, tests and the heavy build infrastructure, followed by Director development lately.
Jean moving along from Icinga 2 core into IcingaDB, followed by Noah with passion for Golang. Our UX designer Florian shaping the web in all its beauty, Johannes taking the lead for Icinga Web and many new modules, followed by Feu with her passion for CSS and better frontend widgets. Alex being the Graphite module developer who recently joined the core devs with the major network stack rewrite, helping my duties on leading the core development whilst sharing all the knowledge with our trainee Henrik. Thomas taking care of Icinga support and improving everything around it. Julia joined for spreading the love on social media and events.
Michael (mcktr) with his passion for the unforeseen Windows issue, Assaf taking care of automation with the official Ansible module, Lennart spreading the love for Puppet while Dirk is our SELinux and RedHat professional. Christian whose day job is in Sales, codes Windows PS at night, sharing his deep knowledge. Lars does a magnificent job on the FreeBSD ports, Matthew dives deep into Gentoo. Bodo is well known for his love for Ruby and Docker containers, appreciated by the community.
Carsten builds awesome Icinga Web modules, Nicolai does so too and jumps right into Director import source development as well. Kevin created a Python API library and knows Ansible&Director, Tobias did so too whilst testing quite a lot. Marianne spreads the Icinga love in her blog, OSMC talks and all over the Internet, even iX articles. Elias analysed, debugged and fixed critical bugs for Icinga 2.11. Virender creating and shaping the Chef cookbook. Our many friends at RedHat helping with the Stackguard Kernel problems.
Timo and Matthias believing in us in trouble times, and letting us test in production, Emanuel who did so too with a large cluster at Contintental. Brad who brought passion and fun from the US, What the Heck from the Netherlands as well. Dave always coming from Australia, and spreading the love (and TimTams). Marcel joining fresh and creating howtos. Jörg who believes in Open Source and still maintains PNP. Michael who develops NSClient++ in his spare time, running on hundreds of thousands Windows clients. Philipp talking things Elastic with Icinga, Jens sharing the love from Müller to the world. Sven & Michael for caring about Livestatus. Moritz spreading the love in Austria at meetups, Bernd taking care of Graylog Vagrant boxes.
Bas maintaining the Icinga packages in Debian. Rico convincing his boss to early adopt Icinga 2. Marco and Max finding all the nasty bugs in larger environments. Claudio creates plugins for ESXI monitoring and shares many howtos. Marc tackles core notifications and Director integrations. Morgan maintains the fancy Grafana notification scripts. Kris thought of the DevOps culture and believes in Icinga. The many of you improving the Icinga Template Library (ITL) commands, sending Graphite templates, documentation updates, issues, pull requests, etc.
Our many friends at customers and users who convince their managers to use and support Icinga. And to share the love – publicly talk about it. Whenever a new Audi or VW is built, Icinga watches. Whenever you buy things online, do bank transactions, go shopping, watch TV, use your mobile phone or enjoy life – Icinga takes care that it works. And when you look up the sky, maybe you can spot the ISS with Icinga exploring space.
There are many more great Icinga users, you know who you are. Thank you from our hearts for an incredible journey.
You know, we are party people celebrating after hard work. Say cheers with your favourite drink, I’ll have – obviously – a G&T.
I’m not going to list all benefits of automating your monitoring system. If you’re here and reading this, you are most likely very aware that maintaining a large infrastructure is a big challenge.
Automating the monitoring process for a huge amount of servers, virtual machines, applications, services, private and public clouds was a main driver for us when we decided to build Icinga 2. In fact, monitoring large environments is not a new demand for us at all. We experienced this challenge in tandem with many corporations for many years. Finally, it lead us to build features like our rule based configuration, Icinga’s REST API and various modules, cookbooks, roles and playbooks for different configuration management tools.
All these methods make it easier to automate monitoring in their own particular way. We have built multiple ways to automate monitoring because there is not only one way to do it right. As usual in the IT field “it depends”. Depending on your infrastructure either one way or another may be the right way for you to automate your monitoring.
Beyond the Static
With all this in mind we have created the Icinga Director 3 years ago. Director is a module for Icinga that enables users to create Icinga configuration within a web interface. The Director utilizes Icinga’s API to create, modify and delete monitoring objects. Besides the plain configuration functionality, the Director has a strong focus on automating these tasks. It provides a different approach that goes far beyond a well-known static configuration management. With thoughtful features the Director empowers operators to manage massive amounts of monitoring objects. In this article I put my attention on the automation functionality of the Director, even though creating single hosts and services manually is also possible of course.
Step 1: Import
In almost every company a database exists, or something similar, that holds information about the currently running servers and applications. Some use dedicated software to manage this information. Others use tools they have built themselves or rely on their config management tools.
While most people refer to this database as a CMDB there are also other common terms. In theory only one database exists within an organization, allowing everyone involved to manage the information in a single place. In practice, most organizations spread the data across a number of different tools. While there are some professional approaches, there are also some not-so-professional ways of maintaining this data. Have you ever seen someone keep their IP addresses and locations of servers in an Excel file? We’ve seen it all.
Creating the source
The Director contains a feature called Import Source. It allows you to import all kinds of data from many different data graves. The data does not have to be in the Icinga configuration format, the Director will take care of that later. For a start, you only need some kind of data.
In my very simple example, I’m using a MySQL database, which is a common storage for this type of information. My database
cmdb contains only one table
hosts that holds everything I know about my servers. For demo purposes, this is perfect.
Add Import Source
Import Source Preview
Creating the import source requires access to the database server. The credentials are stored in an Icinga Web 2 resource, therefore they are re-usable. After triggering a Check for changes you can preview the data set in the Preview tab. If everything that you need is included, you can trigger the import run which actually imports the data.
Starting from here your data is generally available and you can create Icinga configuration out of it. The properties may also be modified during the import, but I leave them as they are for now. Learn more about available modifiers
Using SQL, LDAP or else
For my import source I used the source type SQL which is built-in and available by default. You can use other source types as well, for example LDAP. That allows you to import not only objects that have to be monitored, but also users from your LDAP or Active Directory and use the contact information for alerts.
Of course, you can also use other import sources, such as plain text files, PuppetDB, vSphere or AWS. New import sources are added to the Director as Modules, which you could also write yourself. Our lovely community is continuously extending the Director with new import sources as well, for example with import sources for Microsoft’s Azure or Proxmox VE.
Step 2: Synchronise
After a successful import I am able to continue with basic config synchronisation. Syncing configuration means, that you use the imported data to generate Icinga configuration out of it. Generally speaking, you map your data fields into Icinga objects and properties.
Some of the data I imported is easy to map, such as hostname and IP address. Icinga has pre-defined config parameters for those. Others, like the location and environment of the servers are mapped to custom variables. Custom variables are something like tags, but on steroids. They accept plain strings as values but also booleans, arrays and even dictionaries (hashes). I know, this sounds crazy. Custom variables are usually used to store meta information about your objects. This information you might later want to use to create rules which in turn define what should be monitored or who should receive alerts.
But first things first.
Creating a new Sync requires some input and some decisions. You define what kind of monitoring objects you’re going to create out of your imported data. This can be Hosts, Endpoints, Services, User, Groups and so on. In my example I simply choose to create some host objects.
Then you decide what should happen with existing monitoring configuration. Shall it be replaced with the new one, merged or ignored? Once the Sync is created you can finally start to tell the Director how to map your data to Icinga configuration attributes. This is done within the Properties type by creating new sync property. For every column in my table I create a sync property. Have a look at the images for details.
Add Sync Rule
Add Sync Property
Sync Properties Overview
Checking for changes will only do a dry run to tell you if there are any changes available. Triggering the Sync actually synchronizes the new configuration with the existing one and automatically creates new entries in Director’s activity log. At this point, everything is ready to be deployed to Icinga.
Step 3: Deploy
The Director’s activity log shows precisely what changes are waiting to be deployed to Icinga. Clicking on each element displays the exact diff between old and new configuration. This diff format may be familiar to you from Git for example. Another point that may be familiar is the whole history of deployments which you can see within the activity log.
Travelling in time
It contains every configuration change you ever made with the Director. It lets you travel back in time and deploy old configs if necessary. The Director’s history of deployments basically works like your Git history: You can do a diff between certain deployments to track the changes and see which user deployed which configuration at what time.
The simplest way of deploying your new configuration is by just clicking Deploy pending changes. Your config will be pushed to Icinga and validated. If everything is fine it will be in production within seconds. If there’s anything wrong, you will receive a log with the details and your running configuration remains untouched.
Fully Activity Log
Config Version Diff
As I mentioned in the beginning, automation is a key aspect of the Director. The steps described so far (Import, Synchronise and Deploy) can all be automated once they are fully configured. The automation is done by creating Jobs, with each Job running one certain task in a specified period.
The frequency of a Job is freely configurable. It may run every minute or only once a day. You create a Job for each task or only for certain tasks.
Hint: You can use a simple cronjob to run the Jobs. To fully leverage the Director’s Job functionality in the web interface, you have to start a separate daemon as described in the documentation.
The workflow described above was only a sneak peek into the capabilities of the Director. This scenario however demonstrates very well the basic functionality and what you can do with it. There are many more features left out in this article like modifying imported data, merging data from multiple data sources, filtering or creating and using custom data sets.
Managing and maintaining large server infrastructures is a very complex thing to do and there’s a good reason why whole teams are required to do so. I would lie if I told you that monitoring of such large setups is easy. But with the right toolset it is definitely doable! Check out the full Director Documentation to get started with your monitoring automation project.
We will discuss the challenges of monitoring large infrastructure at the upcoming Icinga Camps as well. Join us for a full day filled with talks and discussions about and around Icinga. Meet new people and get to know others from the same field.