Last week I told Blerim that Icinga came to life on 6.5.2009 … really, 10 years already?
A small group of Nagios community members stood up and wanted to create more than a monitoring core with an enhanced web interface. APIs, integrations, backends, scaling, metrics, … A lot of passion and love was received from the community and actually, the Nagios ecosystem became more active again. We tried many things to improve the existing code base and limitations in both backends and frontends. TL;DR – at some point we decided to throw it away and start fresh. The rest can be read on our blog 🙂
service icinga restart
Back in 2012, Gunnar, Bernd and Marius paid a visit to Vienna. They had something with them what is known to be the first prototype of Icinga 2, codename “Strawberry“. Moving from C into C++ was a whole new world, Golang didn’t really exist back then. Oh, it can run on Windows too. Oh, such speed and performance. Oh, a new playground for our crazy ideas.
Before its first 2.0 release five years ago we had many design rounds with service sets on the Host object, virtual host checks from services and so on. This changed quite a bit after strong opinionated discussions in early 2014. Those are also the origin of the magic apply rules, with mimicking the “put a service on a hostgroup, each host member inherits it” object trick from Icinga 1. Nowadays you can generate monitoring configuration and make things easier.
“Monitoring as a code” became a thing when we had created our own DSL (domain specific language) which added functions, conditions, even loops from community feedback. We’ve also learned a lesson here with the Director released after 2.4 – not everything from inline DSL code is suitable for a configuration frontend. Also that we now have 3 APIs – Core, Web, Director – users might be confused. This is a task we will be working on in the future to unite the product stack better.
Speaking of web and modules, Icinga Web 2 started with the requirement of a general framework including web and CLI based on the Zend framework. IIRC this was 1,5 to 2 years after Icinga 2 kicked off, also explaining the version difference – 2.7 vs 2.11. Web version 1 had many short-comings and was complicated with extensions – unless you’re an experienced developer. With the power of making everything a moduie, even monitoring, the Icinga Web framework inspired the many of you.
Community work is a passion
One of the first community modules I’ve seen was the Globe module which I had shown at Icinga Camp Berlin 2017. Nicolai was so inspired to start the development of the map module. I was extremely happy that I’ve tested it quite extensively and sent a docs PR. Literally the same happened with Carsten creating the famous Grafana module – a gentle README, release hints and troubleshooting – the first impression counts. Did I mention the awesome themes already?
Pushing community ideas and spirit forward is one of the most important things in the way we have become Icinga. Without you and your shared love and criticism, we wouldn’t be here and still be sitting somewhere around the globe ranting about tools.
Back then, our community was a bit split, and not everyone had the time to answer questions on IRC, the mailing lists and the community forum at Monitoring Portal. While we want to be transparent with our development workflow, release dates and design ideas, it’s always time consuming and hard to catch up. With Icinga now supporting distributed environments with zones, endpoints, and web views and configuration forms, we needed something where text formatting and screenshots really integrate well. In late 2018, we’ve decided to create our own Icinga Discourse inviting the team, partners and users to discuss. I just love it, helping others really is a good feeling.
As you may know, the majority of the Icinga development is and was funded by NETWAYS, up until 2012 the University of Vienna sponsored my time. This is paid time and helps with creating the foundation for new features and maintenance. We also raised awareness on why funding and sponsoring is a good thing – if there is a bug biting you, or a missing feature you’d like to use in production, you can speed things up by requesting a quote, or even, by getting a support subscription through one of our partners. Sounds expensive? Maybe. Helps funding the future? Definitely.
Still, we love community contributions and value your (spare) time a lot. Sometimes contributions are hard to achieve as the code is not clear, or you actually don’t know where to start. In the past, we had our own Git server and ticket system. It worked somehow – with the move to GitHub and social coding with Pull Requests, reviews, discussions and a sound platform our daily work on Icinga has become a breeze. The help of issue templates, continuous integration (CI) on when a PR is created, and inline code comments puts the focus back on development.
Lastly, we’ve started documenting our core technical concepts and improved the development guidelines. This now also includes instructions for nightly snapshot builds.
Turning back the time a bit
Icinga 1 didn’t have packages at first glance. You had to call configure/make and pass some parameters for development libraries and headers, sometimes it didn’t work. The core was to compile and then it broke on your system, but the developer’s system worked fine. Ok, let’s create a VM for SuSE in Virtualbox … well, where is my yum?
One of the first ideas was to invite upstream packagers onto the team and learn from each other. Our many releases couldn’t reach stable distributions that fast, so we needed additional repositories. There’s Debian Backports, custom Ubuntu PPAs, EPEL for RHEL/CentOS and lastly, the SuSE OBS repos – not to forget all the other distributions in the Linux/Unix world, FreeBSD, OpenBSD, NetBSD, Gentoo, ArchLinux, … For the main officially supported platforms, we decided to create our own repository at https://packages.icinga.com.
Creating and maintaining packages next to the build infrastructure for each project, language and dependencies is a huge often underestimated effort. Our build server based on Jenkins made several iterations over the years, lately we decided to replace it with GitLab CI for less security risks and better configuration. CI and CD is just great, and with Docker containers nearly everything can be built – requiring your self-hosted infrastructure, maintained by the awesome managed service team at NETWAYS. This includes deployments with Foreman and Puppet, and of course monitoring with Icinga just like “eat your own dogfood” 🙂
Another thing which really was hard to achieve – ARM based architecture on Raspberry Pi. Sounds easy to just compile the binary packages on a different architecture. Actually we had to tackle compiler bugs and long running builds – we started a thing at the OSMC hackathon 2018 and Nicolai’s employer sponsored the last missing days on the road to official packages.
We can always improve
Looking back where we started 10 years ago, we’re in a very comfortable situation. This also includes making the initial setup a breeze, adding lots of documentation and helping on the community channels when things are on fire. With partners providing support, trainings and consultancy, the enterprise requirements are fulfilled even better.
In terms of new projects and ideas, it always turns out that we want to create 4 things, then 10 different customer issues arise, the proof of concepts don’t work out well and obviously, there’s only resources for one or two things being released later. Right on, many have asked about notification managers or mobile apps – that’s on hold for now, finishing the ongoing tasks with Core and Web major versions, Icinga DB and Icinga Reporting, and additional new module version for certificates (x509), Vsphere, Director and Business Process. Again, focus is required – you cannot manage 10 things in parallel. That being said, the monthly snap blog will be moved into a combined effort with the enhanced newsletter telling you about the latest and greatest developments. We also continue to have bi-yearly strategy workshops to refine future roadmaps and discuss changes in our ecosystem.
Those things are great to read, but even more, they are great to talk about. Our idea of meeting at Icinga Camps worked out pretty well, Berlin is by far the biggest one thus far. OSMC with the “State of Icinga” talk by Bernd is great and funny way to share the latest stories too. Our first Icinga team meeting was at OSMC 2009 with many more to come at Icinga Camps all over the world, where you’ll meet, chat and laugh. Our first Icinga Camp San Francisco 2014 happened at GitHub HQ. This is an incredible memory and kicked off the many community events after. In the past months, our community in Germany and Austria have been going strong with regular meetups, that’s truly inspiring. Not been there, or want to create your own meetup group? It is not too late.
We are also aiming for something bigger for everyone planning for next year: IcingaConf in Amsterdam in 2020.
Our way of not re-inventing the wheel but building our integrations with proven tools will continue. Today we are proud to enhance Elastic Stack with adding Icingabeat, Logstash outputs/filters and metric writers even. Adding monitoring as first class citizen in your DevOps lifecycle has become reality with official modules for Puppet, Ansible and Chef and a deeper integration into Foreman. If this isn’t enough, there’s more cloud provider integrations such as Terraform, AWS, Azure and whatnot. Enhancing metrics with Graphite and InfluxDB, provided with example Grafana dashboards and web module integrations. Dashing is still a thing, and reporting has come in its first early version for more SLA and visualization magic.
This is something we couldn’t imagine 10 years ago. We are also proud to work with many professionals and open source lovers all over the world, spreading the #monitoringlove.
The people behind, around and with Icinga
That’s not only me, I hear that often when people find my name on Google. This is a group of passionate people, each with their own character, ideas and responsibilities. Over the years, fellow friends went with us on our journey and left again for their new projects and private life.
Karo, who created the initial designs, blog posts and social media for Icinga. Amanda, who tackled technical problems and spread the love for Icinga on social media and press releases. Gunnar, who is the main architect of Icinga 2, and a passionate coder, the brain behind the DSL and many more. Hendrik who forked Icinga 1 as core and taught us how to use Git and debug the IDO backend.
Michael who worked many hours on Icinga Reports with Jasper, creating fancy web views. Thomas who took over the Oracle backend improvements for Core 1.x. Christian who tackled the Oracle backend in Icinga Web 1, followed by Jannis who enhanced version 1, created mobile 1 and helped create Icinga Web 2 back then. Scott who joined the Skype sessions from Australia, changing our German project into an English one.
Lara and Wolfgang who tackled the version 1 documentation in docbook xml, making our later decision for Markdown very easy. Christoph who created the first Icinga Core RPM packages, Shawn who tried to bring Icinga packages into EPEL, Sam helping here. Carl who’s been working on Solaris packages for 1.x. Nick and Tom paving the way for the first Puppet module. David who created the first VM appliances. Franz and Mike took care of test scripts, Carlos created the vim/nano syntax highlighting for the Icinga DSL, Jordan who’s been working on Docker containers.
Matthew who created the header status bar for Classic UI, Ricardo who took over improving the old CGIs with many enhancements. Rune who did everything testing and created the term “release fucker” back in the 1.x days – when developers didn’t test enough. Massimo helped with 1.x core issues back then. Alex created and maintained the Debian/Ubuntu packages for many years, Tim who did the same for SuSE. Gerd who worked on Icinga 2 reload and threading techniques. Simon who created the InfluxDB feature for Icinga 2.
Thank you deeply.
Many thanks also to Thomas Krenn who sponsored hardware for the initial build server, JetBrains, Navicat, Atlassian & Activestate for granting us open source project licenses back then.
Bernd, Marius and Julian are the remaining co-founders of Icinga. I was a little late to the party in May 2009, with the rest of us enjoying Icinga with our friends, partners and users every day. You’ll recognize Eric for our technical strategy and everything web and core, Blerim improving and managing all the products, helping partners, maintaining integrations, Tom pushing forward with designs and architecture and the shiny Director, VSphere, etc. modules, Markus taking care of QA, tests and the heavy build infrastructure, followed by Director development lately.
Jean moving along from Icinga 2 core into IcingaDB, followed by Noah with passion for Golang. Our UX designer Florian shaping the web in all its beauty, Johannes taking the lead for Icinga Web and many new modules, followed by Feu with her passion for CSS and better frontend widgets. Alex being the Graphite module developer who recently joined the core devs with the major network stack rewrite, helping my duties on leading the core development whilst sharing all the knowledge with our trainee Henrik. Thomas taking care of Icinga support and improving everything around it. Julia joined for spreading the love on social media and events.
Michael (mcktr) with his passion for the unforeseen Windows issue, Assaf taking care of automation with the official Ansible module, Lennart spreading the love for Puppet while Dirk is our SELinux and RedHat professional. Christian whose day job is in Sales, codes Windows PS at night, sharing his deep knowledge. Lars does a magnificent job on the FreeBSD ports, Matthew dives deep into Gentoo. Bodo is well known for his love for Ruby and Docker containers, appreciated by the community.
Carsten builds awesome Icinga Web modules, Nicolai does so too and jumps right into Director import source development as well. Kevin created a Python API library and knows Ansible&Director, Tobias did so too whilst testing quite a lot. Marianne spreads the Icinga love in her blog, OSMC talks and all over the Internet, even iX articles. Elias analysed, debugged and fixed critical bugs for Icinga 2.11. Virender creating and shaping the Chef cookbook. Our many friends at RedHat helping with the Stackguard Kernel problems.
Timo and Matthias believing in us in trouble times, and letting us test in production, Emanuel who did so too with a large cluster at Contintental. Brad who brought passion and fun from the US, What the Heck from the Netherlands as well. Dave always coming from Australia, and spreading the love (and TimTams). Marcel joining fresh and creating howtos. Jörg who believes in Open Source and still maintains PNP. Michael who develops NSClient++ in his spare time, running on hundreds of thousands Windows clients. Philipp talking things Elastic with Icinga, Jens sharing the love from Müller to the world. Sven & Michael for caring about Livestatus. Moritz spreading the love in Austria at meetups, Bernd taking care of Graylog Vagrant boxes.
Bas maintaining the Icinga packages in Debian. Rico convincing his boss to early adopt Icinga 2. Marco and Max finding all the nasty bugs in larger environments. Claudio creates plugins for ESXI monitoring and shares many howtos. Marc tackles core notifications and Director integrations. Morgan maintains the fancy Grafana notification scripts. Kris thought of the DevOps culture and believes in Icinga. The many of you improving the Icinga Template Library (ITL) commands, sending Graphite templates, documentation updates, issues, pull requests, etc.
Our many friends at customers and users who convince their managers to use and support Icinga. And to share the love – publicly talk about it. Whenever a new Audi or VW is built, Icinga watches. Whenever you buy things online, do bank transactions, go shopping, watch TV, use your mobile phone or enjoy life – Icinga takes care that it works. And when you look up the sky, maybe you can spot the ISS with Icinga exploring space.
There are many more great Icinga users, you know who you are. Thank you from our hearts for an incredible journey.
You know, we are party people celebrating after hard work. Say cheers with your favourite drink, I’ll have – obviously – a G&T.
