Last week I told Blerim that Icinga came to life on 6.5.2009 … really, 10 years already?

A small group of Nagios community members stood up and wanted to create more than a monitoring core with an enhanced web interface. APIs, integrations, backends, scaling, metrics, … A lot of passion and love was received from the community and actually, the Nagios ecosystem became more active again. We tried many things to improve the existing code base and limitations in both backends and frontends. TL;DR – at some point we decided to throw it away and start fresh. The rest can be read on our blog 🙂


service icinga restart

Back in 2012, Gunnar, Bernd and Marius paid a visit to Vienna. They had something with them what is known to be the first prototype of Icinga 2, codename “Strawberry“. Moving from C into C++ was a whole new world, Golang didn’t really exist back then. Oh, it can run on Windows too. Oh, such speed and performance. Oh, a new playground for our crazy ideas.

Before its first 2.0 release five years ago we had many design rounds with service sets on the Host object, virtual host checks from services and so on. This changed quite a bit after strong opinionated discussions in early 2014. Those are also the origin of the magic apply rules, with mimicking the “put a service on a hostgroup, each host member inherits it” object trick from Icinga 1. Nowadays you can generate monitoring configuration and make things easier.

“Monitoring as a code” became a thing when we had created our own DSL (domain specific language) which added functions, conditions, even loops from community feedback. We’ve also learned a lesson here with the Director released after 2.4 – not everything from inline DSL code is suitable for a configuration frontend. Also that we now have 3 APIs – Core, Web, Director – users might be confused. This is a task we will be working on in the future to unite the product stack better.

Speaking of web and modules, Icinga Web 2 started with the requirement of a general framework including web and CLI based on the Zend framework. IIRC this was 1,5 to 2 years after Icinga 2 kicked off, also explaining the version difference – 2.7 vs 2.11. Web version 1 had many short-comings and was complicated with extensions – unless you’re an experienced developer. With the power of making everything a moduie, even monitoring, the Icinga Web framework inspired the many of you.


Community work is a passion

One of the first community modules I’ve seen was the Globe module which I had shown at Icinga Camp Berlin 2017. Nicolai was so inspired to start the development of the map module. I was extremely happy that I’ve tested it quite extensively and sent a docs PR. Literally the same happened with Carsten creating the famous Grafana module – a gentle README, release hints and troubleshooting – the first impression counts. Did I mention the awesome themes already?

Pushing community ideas and spirit forward is one of the most important things in the way we have become Icinga. Without you and your shared love and criticism, we wouldn’t be here and still be sitting somewhere around the globe ranting about tools.

Back then, our community was a bit split, and not everyone had the time to answer questions on IRC, the mailing lists and the community forum at Monitoring Portal. While we want to be transparent with our development workflow, release dates and design ideas, it’s always time consuming and hard to catch up. With Icinga now supporting distributed environments with zones, endpoints, and web views and configuration forms, we needed something where text formatting and screenshots really integrate well. In late 2018, we’ve decided to create our own Icinga Discourse inviting the team, partners and users to discuss. I just love it, helping others really is a good feeling.

As you may know, the majority of the Icinga development is and was funded by NETWAYS, up until 2012 the University of Vienna sponsored my time. This is paid time and helps with creating the foundation for new features and maintenance. We also raised awareness on why funding and sponsoring is a good thing – if there is a bug biting you, or a missing feature you’d like to use in production, you can speed things up by requesting a quote, or even, by getting a support subscription through one of our partners. Sounds expensive? Maybe. Helps funding the future? Definitely.

Still, we love community contributions and value your (spare) time a lot. Sometimes contributions are hard to achieve as the code is not clear, or you actually don’t know where to start. In the past, we had our own Git server and ticket system. It worked somehow – with the move to GitHub and social coding with Pull Requests, reviews, discussions and a sound platform our daily work on Icinga has become a breeze. The help of issue templates, continuous integration (CI) on when a PR is created, and inline code comments puts the focus back on development.

Lastly, we’ve started documenting our core technical concepts and improved the development guidelines. This now also includes instructions for nightly snapshot builds.


Turning back the time a bit

Icinga 1 didn’t have packages at first glance. You had to call configure/make and pass some parameters for development libraries and headers, sometimes it didn’t work. The core was to compile and then it broke on your system, but the developer’s system worked fine. Ok, let’s create a VM for SuSE in Virtualbox … well, where is my yum?

One of the first ideas was to invite upstream packagers onto the team and learn from each other. Our many releases couldn’t reach stable distributions that fast, so we needed additional repositories. There’s Debian Backports, custom Ubuntu PPAs, EPEL for RHEL/CentOS and lastly, the SuSE OBS repos – not to forget all the other distributions in the Linux/Unix world, FreeBSD, OpenBSD, NetBSD, Gentoo, ArchLinux, … For the main officially supported platforms, we decided to create our own repository at https://packages.icinga.com.

Creating and maintaining packages next to the build infrastructure for each project, language and dependencies is a huge often underestimated effort. Our build server based on Jenkins made several iterations over the years, lately we decided to replace it with GitLab CI for less security risks and better configuration. CI and CD is just great, and with Docker containers nearly everything can be built – requiring your self-hosted infrastructure, maintained by the awesome managed service team at NETWAYS. This includes deployments with Foreman and Puppet, and of course monitoring with Icinga just like “eat your own dogfood” 🙂

Another thing which really was hard to achieve – ARM based architecture on Raspberry Pi. Sounds easy to just compile the binary packages on a different architecture. Actually we had to tackle compiler bugs and long running builds – we started a thing at the OSMC hackathon 2018 and Nicolai’s employer sponsored the last missing days on the road to official packages.


We can always improve

Looking back where we started 10 years ago, we’re in a very comfortable situation. This also includes making the initial setup a breeze, adding lots of documentation and helping on the community channels when things are on fire. With partners providing support, trainings and consultancy, the enterprise requirements are fulfilled even better.

In terms of new projects and ideas, it always turns out that we want to create 4 things, then 10 different customer issues arise, the proof of concepts don’t work out well and obviously, there’s only resources for one or two things being released later. Right on, many have asked about notification managers or mobile apps – that’s on hold for now, finishing the ongoing tasks with Core and Web major versions, Icinga DB and Icinga Reporting, and additional new module version for certificates (x509), Vsphere, Director and Business Process. Again, focus is required – you cannot manage 10 things in parallel. That being said, the monthly snap blog will be moved into a combined effort with the enhanced newsletter telling you about the latest and greatest developments. We also continue to have bi-yearly strategy workshops to refine future roadmaps and discuss changes in our ecosystem.

Those things are great to read, but even more, they are great to talk about. Our idea of meeting at Icinga Camps worked out pretty well, Berlin is by far the biggest one thus far. OSMC with the “State of Icinga” talk by Bernd is great and funny way to share the latest stories too. Our first Icinga team meeting was at OSMC 2009 with many more to come at Icinga Camps all over the world, where you’ll meet, chat and laugh. Our first Icinga Camp San Francisco 2014 happened at GitHub HQ. This is an incredible memory and kicked off the many community events after. In the past months, our community in Germany and Austria have been going strong with regular meetups, that’s truly inspiring. Not been there, or want to create your own meetup group? It is not too late.

We are also aiming for something bigger for everyone planning for next year: IcingaConf in Amsterdam in 2020.

Our way of not re-inventing the wheel but building our integrations with proven tools will continue. Today we are proud to enhance Elastic Stack with adding Icingabeat, Logstash outputs/filters and metric writers even. Adding monitoring as first class citizen in your DevOps lifecycle has become reality with official modules for Puppet, Ansible and Chef and a deeper integration into Foreman. If this isn’t enough, there’s more cloud provider integrations such as Terraform, AWS, Azure and whatnot. Enhancing metrics with Graphite and InfluxDB, provided with example Grafana dashboards and web module integrations. Dashing is still a thing, and reporting has come in its first early version for more SLA and visualization magic.

This is something we couldn’t imagine 10 years ago. We are also proud to work with many professionals and open source lovers all over the world, spreading the #monitoringlove.


The people behind, around and with Icinga

That’s not only me, I hear that often when people find my name on Google. This is a group of passionate people, each with their own character, ideas and responsibilities. Over the years, fellow friends went with us on our journey and left again for their new projects and private life.

Karo, who created the initial designs, blog posts and social media for Icinga. Amanda, who tackled technical problems and spread the love for Icinga on social media and press releases. Gunnar, who is the main architect of Icinga 2, and a passionate coder, the brain behind the DSL and many more. Hendrik who forked Icinga 1 as core and taught us how to use Git and debug the IDO backend.

Michael who worked many hours on Icinga Reports with Jasper, creating fancy web views. Thomas who took over the Oracle backend improvements for Core 1.x. Christian who tackled the Oracle backend in Icinga Web 1, followed by Jannis who enhanced version 1, created mobile 1 and helped create Icinga Web 2 back then. Scott who joined the Skype sessions from Australia, changing our German project into an English one.

Lara and Wolfgang who tackled the version 1 documentation in docbook xml, making our later decision for Markdown very easy. Christoph who created the first Icinga Core RPM packages, Shawn who tried to bring Icinga packages into EPEL, Sam helping here. Carl who’s been working on Solaris packages for 1.x. Nick and Tom paving the way for the first Puppet module. David who created the first VM appliances. Franz and Mike took care of test scripts, Carlos created the vim/nano syntax highlighting for the Icinga DSL, Jordan who’s been working on Docker containers.

Matthew who created the header status bar for Classic UI, Ricardo who took over improving the old CGIs with many enhancements. Rune who did everything testing and created the term “release fucker” back in the 1.x days – when developers didn’t test enough. Massimo helped with 1.x core issues back then. Alex created and maintained the Debian/Ubuntu packages for many years, Tim who did the same for SuSE. Gerd who worked on Icinga 2 reload and threading techniques. Simon who created the InfluxDB feature for Icinga 2.

Thank you deeply.

Many thanks also to Thomas Krenn who sponsored hardware for the initial build server, JetBrains, Navicat, Atlassian & Activestate for granting us open source project licenses back then.

Bernd, Marius and Julian are the remaining co-founders of Icinga. I was a little late to the party in May 2009, with the rest of us enjoying Icinga with our friends, partners and users every day. You’ll recognize Eric for our technical strategy and everything web and core, Blerim improving and managing all the products, helping partners, maintaining integrations, Tom pushing forward with designs and architecture and the shiny Director, VSphere, etc. modules, Markus taking care of QA, tests and the heavy build infrastructure, followed by Director development lately.

Jean moving along from Icinga 2 core into IcingaDB, followed by Noah with passion for Golang. Our UX designer Florian shaping the web in all its beauty, Johannes taking the lead for Icinga Web and many new modules, followed by Feu with her passion for CSS and better frontend widgets. Alex being the Graphite module developer who recently joined the core devs with the major network stack rewrite, helping my duties on leading the core development whilst sharing all the knowledge with our trainee Henrik. Thomas taking care of Icinga support and improving everything around it. Julia joined for spreading the love on social media and events.

Michael (mcktr) with his passion for the unforeseen Windows issue, Assaf taking care of automation with the official Ansible module, Lennart spreading the love for Puppet while Dirk is our SELinux and RedHat professional. Christian whose day job is in Sales, codes Windows PS at night, sharing his deep knowledge. Lars does a magnificent job on the FreeBSD ports, Matthew dives deep into Gentoo. Bodo is well known for his love for Ruby and Docker containers, appreciated by the community.

Carsten builds awesome Icinga Web modules, Nicolai does so too and jumps right into Director import source development as well. Kevin created a Python API library and knows Ansible&Director, Tobias did so too whilst testing quite a lot. Marianne spreads the Icinga love in her blog, OSMC talks and all over the Internet, even iX articles. Elias analysed, debugged and fixed critical bugs for Icinga 2.11. Virender creating and shaping the Chef cookbook. Our many friends at RedHat helping with the Stackguard Kernel problems.

Timo and Matthias believing in us in trouble times, and letting us test in production, Emanuel who did so too with a large cluster at Contintental. Brad who brought passion and fun from the US, What the Heck from the Netherlands as well. Dave always coming from Australia, and spreading the love (and TimTams). Marcel joining fresh and creating howtos. Jörg who believes in Open Source and still maintains PNP. Michael who develops NSClient++ in his spare time, running on hundreds of thousands Windows clients. Philipp talking things Elastic with Icinga, Jens sharing the love from Müller to the world. Sven & Michael for caring about Livestatus. Moritz spreading the love in Austria at meetups, Bernd taking care of Graylog Vagrant boxes.

Bas maintaining the Icinga packages in Debian. Rico convincing his boss to early adopt Icinga 2. Marco and Max finding all the nasty bugs in larger environments. Claudio creates plugins for ESXI monitoring and shares many howtos. Marc tackles core notifications and Director integrations. Morgan maintains the fancy Grafana notification scripts. Kris thought of the DevOps culture and believes in Icinga. The many of you improving the Icinga Template Library (ITL) commands, sending Graphite templates, documentation updates, issues, pull requests, etc.

Our many friends at customers and users who convince their managers to use and support Icinga. And to share the love – publicly talk about it. Whenever a new Audi or VW is built, Icinga watches. Whenever you buy things online, do bank transactions, go shopping, watch TV, use your mobile phone or enjoy life – Icinga takes care that it works. And when you look up the sky, maybe you can spot the ISS with Icinga exploring space.

There are many more great Icinga users, you know who you are. Thank you from our hearts for an incredible journey.

You know, we are party people celebrating after hard work. Say cheers with your favourite drink, I’ll have – obviously – a G&T.



We've come a long way together … Icinga turns 9

Born to be wild

We’ve been busy with bringing Elasticstack integration into full shape with the Icinga 2 feature and the corresponding Icinga Web 2 module. Both of them were thankfully funded and sponsored by VW. Elasticsearch 6 support is sponsored by a customer too, this will hit 2.9 later in June. Earlier in 2017 we’ve also released the Logstash output for Icinga.
Another integration is Graphite. Shiny metrics available for problem analysis and correlation directly integrated into the host/service detail view in Icinga Web 2 are now a breeze. There’s also a feature rich template language available which makes customization a breeze. And in case you’ll need an overview dashboard, that’s the icing on the cake.
Want to kick off your own module? Look into the existing modules and peek into Eric’s showcase module. Expect more insights into Icinga Web 2 modules and also the theming in the future 🙂

Icinga Camp and DevOpsDays Amsterdam

Team Icinga is going to Amsterdam next week – join us at Icinga Camp Amsterdam on 28.6.2016 and DevOpsDays Amsterdam from 29.6.-1.7.2016!

Icinga Camp Amsterdam

amsterdam_campA full day of monitoring madness – getting started with Icinga monitoring, Icinga project updates and hands-on with Icinga Web 2. Diving further into beautiful graphs with Graphite and Grafana and Icinga 2 API for managing your monitoring stack. Automation and configuration management with Puppet and Ansible is also covered during the camp talks. And keep your fingers crossed – Tom will talk about the Icinga Director providing latest insights into the upcoming 1.1.0 release.
Register now if you haven’t done so already!

devopsdays_amsterdamDevOpsDays Amsterdam

We’ll be having our Icinga 2 Workshop on 29.6.2016 on the first day of DevOpsDays Amsterdam. Basic installation of Icinga 2 and Icinga Web 2 diving deeper in fancy dashboards, configuration hands-on and the Icinga 2 API. Join us there – we’re also sponsoring the event, just look for the guys wearing the Icinga logo!
See you in Amsterdam 🙂

Team Icinga welcomes …

It’s been a while that we’ve written about new team members, but you may have seen new faces on community events and also names inside the git commits recently. Icinga is a community project after all, but some of us want to do more. Getting access to Icinga’s infrastructure, working on existing tasks and finally tagging a release being proud of your work is enjoyable & fun at once. Contact us if you feel the same!

Team Core

Carlos CesarioCarlos Cesario | Core Developer
A sys admin of 8+ years, Carlos’ joins us from Brazil to bring much virtualisation and monitoring experience. The man behind Icinga 2 Vim and Nano syntax highlighting, Carlos has been an avid Icinga user since Icinga 1.2. On board since June 2014.
tkroneTobias von der Krone | Core Developer, QA & Support
Tobias is a long-term Icinga user managing distributed setups with Icinga 2 clusters & Saltstack formulas at ProfitBricks. Whilst learning C++ he takes care of QA, documentation and community support. On the team since January 2015.

Team Web

jmeyerJohannes Meyer | Icinga Web 2 Developer
Johannes started working on Icinga Web 1.x and is now responsible for Icinga Web 2 development. One of his babies is the Icinga Web 2 Setup Wizard paving the way for all the exciting modules and views he’s also been building. On the team since June 2013.
mjentschMatthias Jentsch | Icinga Web 2 Developer
Using the newest web technologies & frameworks, Matthias has proven his expertise in many ways. He’s the main author of NETRP and contributes to Icinga Web 2 development with a love of charts for reporting & auth backend integration. On the team since June 2013.
afuhrAlexander Fuhr | Icinga Web 2 Developer
With more than 10 years of web development experience, Alexander knows how to deal with ExtJS in Icinga Web 1.x and LConf. His main contribution area is the all new shiny Icinga Web 2 and its responsiveness. On the team since August 2014.

Team Packages and Tools

vgagliardiValentino Gagliardi | Ansible Role Developer
A freelance system administrator, in love with monitoring and logging, Valentino uses Icinga to monitor hundreds of Linux servers. IT automation fanatic, is responsible for Icinga 2 Ansible role. Onboard since June 2014.
nchappellNick Chappell | Puppet Module Developer
Nick is located in Portland, OR and working as a sysadmin. Icinga is his first open source project and so he was brave and chose a hard one: Nick is the creator and maintainer of the official puppet-icinga2 Puppet module.  Onboard since August 2014.
bahlersBernd Ahlers | Graylog2 Vagrant Maintainer
Bernd is a Graylog developer and is located in Hamburg. During his journey of log processing and connecting monitoring cores like Icinga 2, he recently joined the Icinga team helping in maintaining the Vagrant boxes, specifically the icinga2x-graylog2 demo box. Onboard since October 2014.
vkhatriVirender Khatri | Chef Cookbook Developer
Virender works as Senior DevOps Engineer at BSB and has 7+ years of experience with AWS, Open Source technologies and automation tooling. He is the lead maintainer of the official Icinga 2 Chef Cookbook paving the way for config management tool intergration. On the team since January 2015.
tdvylderTom de Vylder | Puppet Module Developer
Tom is working at Inuits and is an experienced Puppet developerincluding officially approved modules. He’s putting his effort into thePuppet module for Icinga Web 2, but also ensures that the Icinga 2 module passes Q&A and unit tests. On the team since January 2015.
dgoetzDirk Götz | RPM Packager & SELinux Master
Dirk is working as Senior Consultant at NETWAYS. His main tasks are monitoring with and around Icinga as well as configuration management with Puppet. He’s an RHCE and helps mastering the Icinga RPMs especially focusing on SELinux. On the team since February 2015.

Thank You!

Every little or big contribution counts – but if there is no time left for Icinga, and other things in life become more important, it’s truly understandable.
You know Jannis as the brain behind Icinga Mobile whilst developing Icinga Web 1.x and its modules. He also kicked off the development of Icinga Web 2 together with Tom. Thank you Jannis for remarkable 5 years making Icinga great! All the best for your future – salute to him on Twitter, if you like 🙂

From FOSDEM to CfgMgmtCamp and beyond

fosdem_2015_logoIcinga team members are spread all around the globe, and so are the events we’re attending or organizing 🙂
We’ve never been to FOSDEM, so it was really nice to catch up with community members and speakers knowing Icinga very well, as well as talking about the hot stuff with Icinga 2 and Icinga Web 2. Bruxelles & Waffels, Gin & Beer, and also meeting with the Inuits guys talking Puppet.
FOSDEM took place on Saturday & Sunday where we mostly attended the configuration management dev room, next to distributions (systemd, etc) and visiting the booths. Numbers said 3000 to 5000 users, and certain rooms were just full any time – FOSDEM was renamed to “Fulldem” on Twitter. Sunday evening we joined the Foreman developers and contributors for dinner, moving to Ghent afterwards.
cfgmgmtcamp_logoWith nearly ~500 registered users CfgMgmtCamp has a full house, kicked off by the keynote by Mitchell Hashimoto (creator of Vagrant), providing a vision where configuration management will head. Next up was a panel on the future of configuration management with the brains behind Chef, Ansible, Saltstack and Puppet – awesome! We were moving further into the Foreman track with Katello as content life-cycle management system, and a nifty talk on how to deploy CoreOS with Foreman.
Last but not least – James Turnbull on Monitoring Sucks. While certain aspects are definitely true such as using the configuration management tool to discover and register services, we do not agree on a bad monolithic model behind Icinga 2.
Second day started with a great talk by Jez Humble from Chef on Lean Configuration Management. Next up: Writing Foreman plugins, using autodiscovery and even further – there certainly were many tracks and talks to choose from, Puppet, Ansible, Chef, Salt, Juju, Foreman, … and the ones who discovered the already developed modules for integrating Icinga2 with Puppet, Ansible & Chef doing their home exercises already (have a Saltstack recipe? Contact us!).
Afterall, CfgMgmtCamp was an awesome event – big thank you to the organizers! #cfgmgmtcamp – see you 2016 🙂
PS: Keep monitoring all the things! 😉