Icinga 2.11 Release Candidate

Icinga 2.11 Release Candidate

For months we have been working on one of the biggest and most important releases since the creation of Icinga 2. Icinga 2 version 2.11 includes improvements for performance, stability and scalability. After many changes of lines of code, additions and deletions we believe we’re finally there. But we want to have you on board, so we decided to go with a Release Candidate first. Today we’re happy to announce the general availability of this release!

As you may know, Icinga’s core and network stack is written from scratch in C++, specifically the REST API and network event handling. They caused problems, and the attempts the past releases couldn’t reliably make them go away. We’ve therefore decided to do something unusual and resource intensive: Rewrite the whole network stack by using modern programming techniques and throw away the old implementation. This is a huge step forward and some may say, this would be Icinga 3.

Coming late to the party, fixes for reload handling and unwanted notifications also turned into a long-awaited feature: Run Icinga 2 in foreground and let the new umbrella process handle restarts and reloads. Additionally to the issues we’re aiming to fix, this comes in very handy for (Docker) containers as well!

Lean back and enjoy more changes, fixes and features – our GitHub milestone counts 280+ issues and PRs. With all the changes in mind, we want to make sure that 2.11 will be a great release. Snapshot packages have been tested in the past, now it is time to provide you with a release candidate. Please put it into your test stages and let us know about fixed problems or things we need to fix prior the final release!

Scroll down for installation instructions from our testing repository.

 

Rewritten Network Stack

The low level network I/O with JSON-RPC and HTTP on top was written from scratch many years ago. Recent times have shown problems with hanging TLS connections, stalled HTTP requests and resource consuming operations with the Icinga network stack. Since our old code did not allow us to fix all these bugs, we decided to create a few PoCs with different libraries and technologies. This resulted in a rewrite based on the well-known and tested Boost library. The price to a stable REST API and cluster environment are updated Boost packages which we provide on packages.icinga.com for your convenience.

 

Improved Cluster Sync & High Availability for Features

The cluster config sync and the possibility to just deploy config files into “zones.d” is a key feature in Icinga. With 2.11, we’re adding a long-wanted feature to it: Staging for received configuration. This avoids deploying broken configuration directly into production.

Another thing tackled is the storage for created objects during runtime. In the past, we received reports that downtimes were missing after restart. The underlaying problems have now been fixed. 2.11 heals itself whenever things don’t turn out well, additionally the troubleshooting docs help you during your debugging sessions.

Last but not least, we added our HA mechanism (you know it from the IDO feature) to other features as well. When enabling features like Graphite or Elasticsearch you can now set the setting enable_ha=true on both sides. The feature will then run only on one node and the other will take over in the case of failure.

 

Enhanced Core

The JSON library has been replaced and modernized for full UTF8 support. This comes in preparation for the IcingaDB backend and also solves possible crashes we’ve seen on GitHub.

Notifications which would be triggered during the reload phase are now properly sent. Furthermore, when a host/service is still NOT-OK after a downtime ends, a problem notification is sent immediately.

There’s a bunch of smaller enhancements too: all_services for the schedule-downtime host action, on-demand CSR signing CLI tools, getenv() as DSL function and much more.

 

Documentation

For this release, we’re making a dedicated documentation available which includes all changes and updates. Please note that all the documentation links point to “v2.11-RC1” and not “latest” in the doc URLs. Icinga 2.11 comes with numerous additions and changes, and for the first time we’ve written an extensive upgrading documentation. Don’t be afraid, you don’t need to migrate things manually. Look around, there are many improvements for service monitoring, distributed monitoring, features, technical concepts, development and the upgrading chapter.

As always, prior to installing things, make sure to check the Changelog for v2.11.0-rc1!

 

Installation

These details follow the instruction for testing daily snapshot packages with the main difference that v2.11.0-rc1 is available in another repository on packages.icinga.com. 2.11 requires newer Boost packages provided in the release repository, therefore this is needed for testing the release candidate as well.

 

Debian

apt-get update
apt-get -y install apt-transport-https wget gnupg

wget -O - https://packages.icinga.com/icinga.key | apt-key add -

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
 echo "deb https://packages.icinga.com/debian icinga-${DIST} main" > \
 /etc/apt/sources.list.d/${DIST}-icinga.list
 echo "deb-src https://packages.icinga.com/debian icinga-${DIST} main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga.list

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
 echo "deb http://packages.icinga.com/debian icinga-${DIST}-testing main" > \
 /etc/apt/sources.list.d/${DIST}-icinga-testing.list
 echo "deb-src http://packages.icinga.com/debian icinga-${DIST}-testing main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga-testing.list

apt-get update

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
apt-get install -t icinga-${DIST}-testing icinga2

On Debian Stretch, you also need to add the Backports repository before installation:

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
 echo "deb https://deb.debian.org/debian ${DIST}-backports main" > \
 /etc/apt/sources.list.d/${DIST}-backports.list

apt-get update

 

Ubuntu

apt-get update
apt-get -y install apt-transport-https wget gnupg

wget -O - https://packages.icinga.com/icinga.key | apt-key add -

. /etc/os-release; if [ ! -z ${UBUNTU_CODENAME+x} ]; then DIST="${UBUNTU_CODENAME}"; else DIST="$(lsb_release -c| awk '{print $2}')"; fi; \
 echo "deb https://packages.icinga.com/ubuntu icinga-${DIST} main" > \
 /etc/apt/sources.list.d/${DIST}-icinga.list
 echo "deb-src https://packages.icinga.com/ubuntu icinga-${DIST} main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga.list

. /etc/os-release; if [ ! -z ${UBUNTU_CODENAME+x} ]; then DIST="${UBUNTU_CODENAME}"; else DIST="$(lsb_release -c| awk '{print $2}')"; fi; \
 echo "deb https://packages.icinga.com/ubuntu icinga-${DIST}-testing main" > \
 /etc/apt/sources.list.d/${DIST}-icinga-testing.list
 echo "deb-src https://packages.icinga.com/ubuntu icinga-${DIST}-testing main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga-testing.list

. /etc/os-release; if [ ! -z ${UBUNTU_CODENAME+x} ]; then DIST="${UBUNTU_CODENAME}"; else DIST="$(lsb_release -c| awk '{print $2}')"; fi; \
apt-get install -t icinga-${DIST}-testing icinga2

 

RHEL/CentOS 7

yum -y install https://packages.icinga.com/epel/icinga-rpm-release-7-latest.noarch.rpm
yum -y install epel-release
yum makecache

cat >/etc/yum.repos.d/icinga-testing.repo <<EOF

[icinga-testing-builds]
name=ICINGA (testing builds for epel)
baseurl=http://packages.icinga.com/epel/\$releasever/testing/
enabled=1
gpgcheck=1
gpgkey=https://packages.icinga.com/icinga.key
metadata_expire=1d

EOF

yum makecache

yum install -t icinga-testing-builds icinga2

 

SLES

rpm --import https://packages.icinga.com/icinga.key

zypper ar https://packages.icinga.com/SUSE/ICINGA-release.repo
zypper ref -fs

cat >/etc/zypp/repos.d/icinga-testing.repo <<EOF

[icinga-testing-builds]
name=ICINGA (testing builds for SUSE \$releasever)
baseurl=https://packages.icinga.com/SUSE/\$releasever/testing/
enabled=1
gpgcheck=1
gpgkey=https://packages.icinga.com/icinga.key
metadata_expire=1d

EOF

zypper ref -fs

zypper in icinga2

 

Windows

x64 and x86.

If you need help with testing the release candidate, or want to discuss new things, hop onto our community forums.

Update 2019-08-01: In case you’re encountering “no shared cipher” errors with 2.10.x agents on RHEL7 or Windows connecting to parent 2.11 RC1 instances, please adjust the “cipher_list” configuration as shown in this issue. More troubleshooting hints can be found here.

Update 2019-08-06: Added backports repo for Debian Stretch. All RC feedback is collected and smaller fixes are applied, nothing which harms testing the RC. Thanks for your ongoing feedback!

 

 

Icinga 2.10.5

Icinga 2.10.5

This release fixes a regression introduced with namespaces in 2.10 where advanced permission filters could result in a crash with many concurrent requests. It also fixes the problem that permission filters are sometimes not applied correctly. Thanks VSHN for sponsoring the development time!

Another problem hidden in the dark was introduced in 2.9 with changed reload signals. This caused the reload shutdown handler to deactivate host and hostgroup objects in the IDO database backend, with later activating them on startup reconnect. Large scale environments would suffer from seconds to minutes not seeing objects in Icinga Web 2 then.

Last but not least, our road to better code quality unveiled that certain compilers don’t treat us well with shared pointers references and resulting crashes. This release fixes crashes with logrotate signals and other timer related problems, e.g. state file handling.

We’ve also added some documentation improvements for the REST API, object types, feature overview and technical concepts with JSON-RPC cluster messages – inspired by the feedback from our Icinga Discourse 🙂

Checkout the full changelog for 2.10.5 here. Please note that SLES11 and Ubuntu14 packages are not available anymore. 

Special thanks goes out to Elias Ohm from novomind AG 🙂 He’s been debugging and fixing the many issues with us together, magnificent team work!

Note: API and cluster/agent related problems will be fixed in 2.11, requiring us to rewrite large parts of the code base. You can help test the snapshot packages and provide feedback. Thanks in advance!

10

10

Last week I told Blerim that Icinga came to life on 6.5.2009 … really, 10 years already?

A small group of Nagios community members stood up and wanted to create more than a monitoring core with an enhanced web interface. APIs, integrations, backends, scaling, metrics, … A lot of passion and love was received from the community and actually, the Nagios ecosystem became more active again. We tried many things to improve the existing code base and limitations in both backends and frontends. TL;DR – at some point we decided to throw it away and start fresh. The rest can be read on our blog 🙂

 

service icinga restart

Back in 2012, Gunnar, Bernd and Marius paid a visit to Vienna. They had something with them what is known to be the first prototype of Icinga 2, codename “Strawberry“. Moving from C into C++ was a whole new world, Golang didn’t really exist back then. Oh, it can run on Windows too. Oh, such speed and performance. Oh, a new playground for our crazy ideas.

Before its first 2.0 release five years ago we had many design rounds with service sets on the Host object, virtual host checks from services and so on. This changed quite a bit after strong opinionated discussions in early 2014. Those are also the origin of the magic apply rules, with mimicking the “put a service on a hostgroup, each host member inherits it” object trick from Icinga 1. Nowadays you can generate monitoring configuration and make things easier.

“Monitoring as a code” became a thing when we had created our own DSL (domain specific language) which added functions, conditions, even loops from community feedback. We’ve also learned a lesson here with the Director released after 2.4 – not everything from inline DSL code is suitable for a configuration frontend. Also that we now have 3 APIs – Core, Web, Director – users might be confused. This is a task we will be working on in the future to unite the product stack better.

Speaking of web and modules, Icinga Web 2 started with the requirement of a general framework including web and CLI based on the Zend framework. IIRC this was 1,5 to 2 years after Icinga 2 kicked off, also explaining the version difference – 2.7 vs 2.11. Web version 1 had many short-comings and was complicated with extensions – unless you’re an experienced developer. With the power of making everything a moduie, even monitoring, the Icinga Web framework inspired the many of you.

 

Community work is a passion

One of the first community modules I’ve seen was the Globe module which I had shown at Icinga Camp Berlin 2017. Nicolai was so inspired to start the development of the map module. I was extremely happy that I’ve tested it quite extensively and sent a docs PR. Literally the same happened with Carsten creating the famous Grafana module – a gentle README, release hints and troubleshooting – the first impression counts. Did I mention the awesome themes already?

Pushing community ideas and spirit forward is one of the most important things in the way we have become Icinga. Without you and your shared love and criticism, we wouldn’t be here and still be sitting somewhere around the globe ranting about tools.

Back then, our community was a bit split, and not everyone had the time to answer questions on IRC, the mailing lists and the community forum at Monitoring Portal. While we want to be transparent with our development workflow, release dates and design ideas, it’s always time consuming and hard to catch up. With Icinga now supporting distributed environments with zones, endpoints, and web views and configuration forms, we needed something where text formatting and screenshots really integrate well. In late 2018, we’ve decided to create our own Icinga Discourse inviting the team, partners and users to discuss. I just love it, helping others really is a good feeling.

As you may know, the majority of the Icinga development is and was funded by NETWAYS, up until 2012 the University of Vienna sponsored my time. This is paid time and helps with creating the foundation for new features and maintenance. We also raised awareness on why funding and sponsoring is a good thing – if there is a bug biting you, or a missing feature you’d like to use in production, you can speed things up by requesting a quote, or even, by getting a support subscription through one of our partners. Sounds expensive? Maybe. Helps funding the future? Definitely.

Still, we love community contributions and value your (spare) time a lot. Sometimes contributions are hard to achieve as the code is not clear, or you actually don’t know where to start. In the past, we had our own Git server and ticket system. It worked somehow – with the move to GitHub and social coding with Pull Requests, reviews, discussions and a sound platform our daily work on Icinga has become a breeze. The help of issue templates, continuous integration (CI) on when a PR is created, and inline code comments puts the focus back on development.

Lastly, we’ve started documenting our core technical concepts and improved the development guidelines. This now also includes instructions for nightly snapshot builds.

 

Turning back the time a bit

Icinga 1 didn’t have packages at first glance. You had to call configure/make and pass some parameters for development libraries and headers, sometimes it didn’t work. The core was to compile and then it broke on your system, but the developer’s system worked fine. Ok, let’s create a VM for SuSE in Virtualbox … well, where is my yum?

One of the first ideas was to invite upstream packagers onto the team and learn from each other. Our many releases couldn’t reach stable distributions that fast, so we needed additional repositories. There’s Debian Backports, custom Ubuntu PPAs, EPEL for RHEL/CentOS and lastly, the SuSE OBS repos – not to forget all the other distributions in the Linux/Unix world, FreeBSD, OpenBSD, NetBSD, Gentoo, ArchLinux, … For the main officially supported platforms, we decided to create our own repository at https://packages.icinga.com.

Creating and maintaining packages next to the build infrastructure for each project, language and dependencies is a huge often underestimated effort. Our build server based on Jenkins made several iterations over the years, lately we decided to replace it with GitLab CI for less security risks and better configuration. CI and CD is just great, and with Docker containers nearly everything can be built – requiring your self-hosted infrastructure, maintained by the awesome managed service team at NETWAYS. This includes deployments with Foreman and Puppet, and of course monitoring with Icinga just like “eat your own dogfood” 🙂

Another thing which really was hard to achieve – ARM based architecture on Raspberry Pi. Sounds easy to just compile the binary packages on a different architecture. Actually we had to tackle compiler bugs and long running builds – we started a thing at the OSMC hackathon 2018 and Nicolai’s employer sponsored the last missing days on the road to official packages.

 

We can always improve

Looking back where we started 10 years ago, we’re in a very comfortable situation. This also includes making the initial setup a breeze, adding lots of documentation and helping on the community channels when things are on fire. With partners providing support, trainings and consultancy, the enterprise requirements are fulfilled even better.

In terms of new projects and ideas, it always turns out that we want to create 4 things, then 10 different customer issues arise, the proof of concepts don’t work out well and obviously, there’s only resources for one or two things being released later. Right on, many have asked about notification managers or mobile apps – that’s on hold for now, finishing the ongoing tasks with Core and Web major versions, Icinga DB and Icinga Reporting, and additional new module version for certificates (x509), Vsphere, Director and Business Process. Again, focus is required – you cannot manage 10 things in parallel. That being said, the monthly snap blog will be moved into a combined effort with the enhanced newsletter telling you about the latest and greatest developments. We also continue to have bi-yearly strategy workshops to refine future roadmaps and discuss changes in our ecosystem.

Those things are great to read, but even more, they are great to talk about. Our idea of meeting at Icinga Camps worked out pretty well, Berlin is by far the biggest one thus far. OSMC with the “State of Icinga” talk by Bernd is great and funny way to share the latest stories too. Our first Icinga team meeting was at OSMC 2009 with many more to come at Icinga Camps all over the world, where you’ll meet, chat and laugh. Our first Icinga Camp San Francisco 2014 happened at GitHub HQ. This is an incredible memory and kicked off the many community events after. In the past months, our community in Germany and Austria have been going strong with regular meetups, that’s truly inspiring. Not been there, or want to create your own meetup group? It is not too late.

We are also aiming for something bigger for everyone planning for next year: IcingaConf in Amsterdam in 2020.

Our way of not re-inventing the wheel but building our integrations with proven tools will continue. Today we are proud to enhance Elastic Stack with adding Icingabeat, Logstash outputs/filters and metric writers even. Adding monitoring as first class citizen in your DevOps lifecycle has become reality with official modules for Puppet, Ansible and Chef and a deeper integration into Foreman. If this isn’t enough, there’s more cloud provider integrations such as Terraform, AWS, Azure and whatnot. Enhancing metrics with Graphite and InfluxDB, provided with example Grafana dashboards and web module integrations. Dashing is still a thing, and reporting has come in its first early version for more SLA and visualization magic.

This is something we couldn’t imagine 10 years ago. We are also proud to work with many professionals and open source lovers all over the world, spreading the #monitoringlove.

 

The people behind, around and with Icinga

That’s not only me, I hear that often when people find my name on Google. This is a group of passionate people, each with their own character, ideas and responsibilities. Over the years, fellow friends went with us on our journey and left again for their new projects and private life.

Karo, who created the initial designs, blog posts and social media for Icinga. Amanda, who tackled technical problems and spread the love for Icinga on social media and press releases. Gunnar, who is the main architect of Icinga 2, and a passionate coder, the brain behind the DSL and many more. Hendrik who forked Icinga 1 as core and taught us how to use Git and debug the IDO backend.

Michael who worked many hours on Icinga Reports with Jasper, creating fancy web views. Thomas who took over the Oracle backend improvements for Core 1.x. Christian who tackled the Oracle backend in Icinga Web 1, followed by Jannis who enhanced version 1, created mobile 1 and helped create Icinga Web 2 back then. Scott who joined the Skype sessions from Australia, changing our German project into an English one.

Lara and Wolfgang who tackled the version 1 documentation in docbook xml, making our later decision for Markdown very easy. Christoph who created the first Icinga Core RPM packages, Shawn who tried to bring Icinga packages into EPEL, Sam helping here. Carl who’s been working on Solaris packages for 1.x. Nick and Tom paving the way for the first Puppet module. David who created the first VM appliances. Franz and Mike took care of test scripts, Carlos created the vim/nano syntax highlighting for the Icinga DSL, Jordan who’s been working on Docker containers.

Matthew who created the header status bar for Classic UI, Ricardo who took over improving the old CGIs with many enhancements. Rune who did everything testing and created the term “release fucker” back in the 1.x days – when developers didn’t test enough. Massimo helped with 1.x core issues back then. Alex created and maintained the Debian/Ubuntu packages for many years, Tim who did the same for SuSE. Gerd who worked on Icinga 2 reload and threading techniques. Simon who created the InfluxDB feature for Icinga 2.

Thank you deeply.

Many thanks also to Thomas Krenn who sponsored hardware for the initial build server, JetBrains, Navicat, Atlassian & Activestate for granting us open source project licenses back then.

Bernd, Marius and Julian are the remaining co-founders of Icinga. I was a little late to the party in May 2009, with the rest of us enjoying Icinga with our friends, partners and users every day. You’ll recognize Eric for our technical strategy and everything web and core, Blerim improving and managing all the products, helping partners, maintaining integrations, Tom pushing forward with designs and architecture and the shiny Director, VSphere, etc. modules, Markus taking care of QA, tests and the heavy build infrastructure, followed by Director development lately.

Jean moving along from Icinga 2 core into IcingaDB, followed by Noah with passion for Golang. Our UX designer Florian shaping the web in all its beauty, Johannes taking the lead for Icinga Web and many new modules, followed by Feu with her passion for CSS and better frontend widgets. Alex being the Graphite module developer who recently joined the core devs with the major network stack rewrite, helping my duties on leading the core development whilst sharing all the knowledge with our trainee Henrik. Thomas taking care of Icinga support and improving everything around it. Julia joined for spreading the love on social media and events.

Michael (mcktr) with his passion for the unforeseen Windows issue, Assaf taking care of automation with the official Ansible module, Lennart spreading the love for Puppet while Dirk is our SELinux and RedHat professional. Christian whose day job is in Sales, codes Windows PS at night, sharing his deep knowledge. Lars does a magnificent job on the FreeBSD ports, Matthew dives deep into Gentoo. Bodo is well known for his love for Ruby and Docker containers, appreciated by the community.

Carsten builds awesome Icinga Web modules, Nicolai does so too and jumps right into Director import source development as well. Kevin created a Python API library and knows Ansible&Director, Tobias did so too whilst testing quite a lot. Marianne spreads the Icinga love in her blog, OSMC talks and all over the Internet, even iX articles. Elias analysed, debugged and fixed critical bugs for Icinga 2.11. Virender creating and shaping the Chef cookbook. Our many friends at RedHat helping with the Stackguard Kernel problems.

Timo and Matthias believing in us in trouble times, and letting us test in production, Emanuel who did so too with a large cluster at Contintental. Brad who brought passion and fun from the US, What the Heck from the Netherlands as well. Dave always coming from Australia, and spreading the love (and TimTams). Marcel joining fresh and creating howtos. Jörg who believes in Open Source and still maintains PNP. Michael who develops NSClient++ in his spare time, running on hundreds of thousands Windows clients. Philipp talking things Elastic with Icinga, Jens sharing the love from Müller to the world. Sven & Michael for caring about Livestatus. Moritz spreading the love in Austria at meetups, Bernd taking care of Graylog Vagrant boxes.

Bas maintaining the Icinga packages in Debian. Rico convincing his boss to early adopt Icinga 2. Marco and Max finding all the nasty bugs in larger environments. Claudio creates plugins for ESXI monitoring and shares many howtos. Marc tackles core notifications and Director integrations. Morgan maintains the fancy Grafana notification scripts. Kris thought of the DevOps culture and believes in Icinga. The many of you improving the Icinga Template Library (ITL) commands, sending Graphite templates, documentation updates, issues, pull requests, etc.

Our many friends at customers and users who convince their managers to use and support Icinga. And to share the love – publicly talk about it. Whenever a new Audi or VW is built, Icinga watches. Whenever you buy things online, do bank transactions, go shopping, watch TV, use your mobile phone or enjoy life – Icinga takes care that it works. And when you look up the sky, maybe you can spot the ISS with Icinga exploring space.

There are many more great Icinga users, you know who you are. Thank you from our hearts for an incredible journey.

You know, we are party people celebrating after hard work. Say cheers with your favourite drink, I’ll have – obviously – a G&T.

 

 

Icinga 2.10.4 bugfix release

Icinga 2.10.4 bugfix release

This release provides fixes for the InfluxDB and Elasticsearch metric writers. If you’re using TLS connections, the latter were not closed correctly. In addition to these fixes, we’ve also backported fixes for delayed and one-time notifications. Special thanks to mdetrano for being patient and testing this one.

Additionally, the Windows wizard has been updated and check_perfmon now supports non-localized performance counters. One of our customers sponsored improving mass-creation of downtimes via the REST API in HA enabled clusters, thank you.

Official packages are available on packages.icinga.com, have been pushed to Chocolatey and Raspbian will follow soon. Meanwhile check the changelog for v2.10.4.

Icinga 2.10.3 bugfix release

2.10.3 fixes TLS connections with masters and agents on reload and “Connection: close” headers with Ruby clients. We’ve also tackled long-term problems with (scheduled) downtimes in HA-enabled cluster environments. 2.10.3 also fixes a problem with time offsets and check results from the future.

Next to the long list of bugfixes, we’ve also improved the documentation with technical concepts on the check scheduler, and a complete overhaul of our development docs for contributors and packagers. We’re currently working on an improved network stack and cluster synchronisation for 2.11, stay tuned.

Thanks to our contributors Edgar, Sven, Leon, Michael, Alex and Max! 2.10.3 is available on packages.icinga.com, Raspbian packages will follow soon. Meanwhile check the changelog for 2.10.3.