Icinga 2.11.1 release

Icinga 2.11.1 release

2.11.0 unveiled a critical bug in the cluster config sync. Before you panic – upgrading the config master as documented before any other instance keeps you safe. The 2.11.1 release is for everyone upgrading their satellites/agents first. For those who cannot do a major upgrade on their masters, we also release 2.10.7 where we added an additional fix to prevent this behaviour.

The online documentation additionally covers troubleshooting topics for configuration problems with command endpoint agents and zone-in-zone synchronization.

Check the Changelog for 2.11.1 and 2.10.7 prior to upgrading. Packages are available on packages.icinga.com as always.

Icinga 2.11

Icinga 2.11

Now we are here, after many months of development – we proudly release Icinga 2.11 available today.

 

Bleeding edge

It has been an emotional ride with many changes under the hood. The most obvious change is that Icinga’s distributed cluster operates more stable, the past quirks with hanging certificate signing requests or dead-locked TLS handshakes are now gone. This required us to go an unusual route: Evaluate new libraries and programming techniques in order to replace hand-written lower layered code, with later replacing the entire code base for the network stack operations in Icinga. This is a massive effort in quality and stability where users had called out for 3.0 already.

The core components of Icinga are checks and notifications. Thanks to sponsoring from a customer, Icinga now immediately send notifications after a downtime ends whenever hosts/services are still NOT-OK. Lost notifications during restarts in HA zones also have been tackled and fixed.

In the past years, we’ve seen operation problems with the config sync inside the cluster system. This has been tailored to a more robust deployment with staged syncs as well as preventing unwanted sync loops for comments/downtimes. The local storage for runtime created objects also is more robust. Also, note additional HA capabilities for features like Graphite or Elasticsearch.

We care about security. The setup steps with TLS certificates and signing requests in addition to the trusted zone hierarchy is necessary to keep your monitoring data safe. Icinga can see many details in your environment and must not be compromised. Taking things serious, Icinga 2.11 enforces TLS 1.2 as a minimum and also hardens the available cipher lists by default. Older agent versions still work granted that they support TLS 1.2 (OpenSSL 1.x.y).

 

More Icing on the cake

Icinga aims to run in small environments as well as large distributed enterprise systems. With the general availability of Debian Buster on Raspberry Pi, Icinga 2.11 is ready for your home lab and IoT setups.

To all the fans of on-demand certificate signing: You’ll get more tooling to manage/revoke satellite and agent certificate requests. A small but nifty addition are environment variables which can be retrieved with the new getenv() DSL function – for sensitive data or getting the path of HOME into your command’s context.

A small but gentle addition is the “all_services” attribute when setting a host in downtime with the REST API. Aside from the more visible fixes, many changes have been applied to improve our code base for the next years and avoid future bugs. This includes more unit tests e.g. for timeperiod ranges or version comparisons in the built-in icinga check.

Coming late to the party, we’ve fixed crashes from Nessus scanners on Linux and following up on Windows. Well, this wasn’t easy – a technical monster to debug and fix. Cheers to our friends Stefan and Philipp in Linz who have helped us to reproduce the issue. The technology now used in Icinga 2.11 proves the rule for scaling monitoring for the next decade.

Icinga uses the well-known Boost library as part of the network stack rewrite, thanks to everyone who contributes to their development. Also we’d like to cordially say thanks to Niels Lohmann for the magic JSON library, and Nemanja Trifunovic for making UTF8 handling in C++ a breeze. And to you, our fellow community contributors!

In the late development stages for 2.11, we’ve also tackled a problem with systemd reloads resulting in a killed process. The only way to prevent this from happening is actually a new feature: A real umbrella process managing the reload signal. Turns out, this implements a long standing feature request for running Icinga in a (Docker) container.

 

Docs, docs, docs

Our community forum with many great users and contributors helping each other raised the bar for the documentation. We are addicted to making the Icinga experience as easy as possible, and for 2.11, we have improved the documentation in many ways: From the basic setup to configuration to distributed environments to service monitoring to features and addons.

This includes a clear cut specification of the plugin API, with best practices from experienced plugin developers. Speaking of development, packagers and future developers are key to Icinga’s success. The development chapter as been overhauled to motivate everyone out there. Those wanting to learn about the design decisions, the technical concepts chapter highlights new bleeding edge insights.

 

Get Icinga 2.11, check the Changelog & upgrading docs and say cheers to the biggest release we’ve ever done.

Color Blind theme for Icinga Web 2

Color Blind theme for Icinga Web 2

Give your Icinga Web 2 a new look – with official ‘Color Blind Theme’ introduced in version 2.7.0! Not just for people with impaired colour vision – with a fresh and well rounded new palette, the new theme is a real looker!

It’s important for us that everyone has the chance to use our tools with maximum effectiveness.We took an existing module from our Icinga Partner Sol1icingaweb2-theme-colourblind by Andrew Foster – refined it a little and it is now part of the default Icinga Web 2 colour scheme bundle!

 

What’s with the change?

The colorblind theme by Sol1 was already very well made, but we saw some minor issues with the chosen colours in terms of:

  • context with the icinga states
  • readability

Well, let’s just have a look first:

Overview over the different themes and the effects colour deficiency has on the appearance

Overview over the different themes and the effects colour deficiency has on the appearance

So in the top, there is the default Icinga colour scheme – with mocks of the three types of colour deficiency we focused on during the development for the module.
In the middle row you can see the module Sol1 provided
Last row is the new theme which is integrated into the Icinga Web 2 core now.

Icinga state context

Another minor detail to improve was the colour of the WARNING (handled) state, which didn’t really fit in with the rest:

Comparison of colours for the states + handled states

The WARNING (handled) grey sticks out and is hard to connect to the unhandled state colour

So in order to both have the context in which the colours are used preserved and have some sort of hierarchy in severeness of the states some changes needed to be made.

This warranted for a change both in brightness and hue for the OK green as well, so it could be distinguished from the WARNING (handled) yellow:

Visual representation of a colour with changes in hue, saturation and brightness

Visual representation of changes in hue, saturation and brightness.

 

Readability

There have also been some changes to the font colour:
Handled states (Including OK for urgencies sake) are now displayed with black text.

This makes it a lot easier to distinguish between handled and unhandled states in the blink of an eye.

Badges with the different themes

Badges with the different themes

 

 

In case you want to build your own theme, there is a helpful article on the topic from our Partner Würth Phoenix. If you have any issues, feedback or inspiration – please tell us about it here or open an issue on github.

And by the way, I will be speaking about this topic at the Icinga Camp Stockholm as well!

Icinga 2.11 Release Candidate

Icinga 2.11 Release Candidate

For months we have been working on one of the biggest and most important releases since the creation of Icinga 2. Icinga 2 version 2.11 includes improvements for performance, stability and scalability. After many changes of lines of code, additions and deletions we believe we’re finally there. But we want to have you on board, so we decided to go with a Release Candidate first. Today we’re happy to announce the general availability of this release!

As you may know, Icinga’s core and network stack is written from scratch in C++, specifically the REST API and network event handling. They caused problems, and the attempts the past releases couldn’t reliably make them go away. We’ve therefore decided to do something unusual and resource intensive: Rewrite the whole network stack by using modern programming techniques and throw away the old implementation. This is a huge step forward and some may say, this would be Icinga 3.

Coming late to the party, fixes for reload handling and unwanted notifications also turned into a long-awaited feature: Run Icinga 2 in foreground and let the new umbrella process handle restarts and reloads. Additionally to the issues we’re aiming to fix, this comes in very handy for (Docker) containers as well!

Lean back and enjoy more changes, fixes and features – our GitHub milestone counts 280+ issues and PRs. With all the changes in mind, we want to make sure that 2.11 will be a great release. Snapshot packages have been tested in the past, now it is time to provide you with a release candidate. Please put it into your test stages and let us know about fixed problems or things we need to fix prior the final release!

Scroll down for installation instructions from our testing repository.

 

Rewritten Network Stack

The low level network I/O with JSON-RPC and HTTP on top was written from scratch many years ago. Recent times have shown problems with hanging TLS connections, stalled HTTP requests and resource consuming operations with the Icinga network stack. Since our old code did not allow us to fix all these bugs, we decided to create a few PoCs with different libraries and technologies. This resulted in a rewrite based on the well-known and tested Boost library. The price to a stable REST API and cluster environment are updated Boost packages which we provide on packages.icinga.com for your convenience.

 

Improved Cluster Sync & High Availability for Features

The cluster config sync and the possibility to just deploy config files into “zones.d” is a key feature in Icinga. With 2.11, we’re adding a long-wanted feature to it: Staging for received configuration. This avoids deploying broken configuration directly into production.

Another thing tackled is the storage for created objects during runtime. In the past, we received reports that downtimes were missing after restart. The underlaying problems have now been fixed. 2.11 heals itself whenever things don’t turn out well, additionally the troubleshooting docs help you during your debugging sessions.

Last but not least, we added our HA mechanism (you know it from the IDO feature) to other features as well. When enabling features like Graphite or Elasticsearch you can now set the setting enable_ha=true on both sides. The feature will then run only on one node and the other will take over in the case of failure.

 

Enhanced Core

The JSON library has been replaced and modernized for full UTF8 support. This comes in preparation for the IcingaDB backend and also solves possible crashes we’ve seen on GitHub.

Notifications which would be triggered during the reload phase are now properly sent. Furthermore, when a host/service is still NOT-OK after a downtime ends, a problem notification is sent immediately.

There’s a bunch of smaller enhancements too: all_services for the schedule-downtime host action, on-demand CSR signing CLI tools, getenv() as DSL function and much more.

 

Documentation

For this release, we’re making a dedicated documentation available which includes all changes and updates. Please note that all the documentation links point to “v2.11-RC1” and not “latest” in the doc URLs. Icinga 2.11 comes with numerous additions and changes, and for the first time we’ve written an extensive upgrading documentation. Don’t be afraid, you don’t need to migrate things manually. Look around, there are many improvements for service monitoring, distributed monitoring, features, technical concepts, development and the upgrading chapter.

As always, prior to installing things, make sure to check the Changelog for v2.11.0-rc1!

 

Installation

These details follow the instruction for testing daily snapshot packages with the main difference that v2.11.0-rc1 is available in another repository on packages.icinga.com. 2.11 requires newer Boost packages provided in the release repository, therefore this is needed for testing the release candidate as well.

 

Debian

apt-get update
apt-get -y install apt-transport-https wget gnupg

wget -O - https://packages.icinga.com/icinga.key | apt-key add -

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
 echo "deb https://packages.icinga.com/debian icinga-${DIST} main" > \
 /etc/apt/sources.list.d/${DIST}-icinga.list
 echo "deb-src https://packages.icinga.com/debian icinga-${DIST} main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga.list

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
 echo "deb http://packages.icinga.com/debian icinga-${DIST}-testing main" > \
 /etc/apt/sources.list.d/${DIST}-icinga-testing.list
 echo "deb-src http://packages.icinga.com/debian icinga-${DIST}-testing main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga-testing.list

apt-get update

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
apt-get install -t icinga-${DIST}-testing icinga2

On Debian Stretch, you also need to add the Backports repository before installation:

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
 echo "deb https://deb.debian.org/debian ${DIST}-backports main" > \
 /etc/apt/sources.list.d/${DIST}-backports.list

apt-get update

 

Ubuntu

apt-get update
apt-get -y install apt-transport-https wget gnupg

wget -O - https://packages.icinga.com/icinga.key | apt-key add -

. /etc/os-release; if [ ! -z ${UBUNTU_CODENAME+x} ]; then DIST="${UBUNTU_CODENAME}"; else DIST="$(lsb_release -c| awk '{print $2}')"; fi; \
 echo "deb https://packages.icinga.com/ubuntu icinga-${DIST} main" > \
 /etc/apt/sources.list.d/${DIST}-icinga.list
 echo "deb-src https://packages.icinga.com/ubuntu icinga-${DIST} main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga.list

. /etc/os-release; if [ ! -z ${UBUNTU_CODENAME+x} ]; then DIST="${UBUNTU_CODENAME}"; else DIST="$(lsb_release -c| awk '{print $2}')"; fi; \
 echo "deb https://packages.icinga.com/ubuntu icinga-${DIST}-testing main" > \
 /etc/apt/sources.list.d/${DIST}-icinga-testing.list
 echo "deb-src https://packages.icinga.com/ubuntu icinga-${DIST}-testing main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga-testing.list

. /etc/os-release; if [ ! -z ${UBUNTU_CODENAME+x} ]; then DIST="${UBUNTU_CODENAME}"; else DIST="$(lsb_release -c| awk '{print $2}')"; fi; \
apt-get install -t icinga-${DIST}-testing icinga2

 

RHEL/CentOS 7

yum -y install https://packages.icinga.com/epel/icinga-rpm-release-7-latest.noarch.rpm
yum -y install epel-release
yum makecache

cat >/etc/yum.repos.d/icinga-testing.repo <<EOF

[icinga-testing-builds]
name=ICINGA (testing builds for epel)
baseurl=http://packages.icinga.com/epel/\$releasever/testing/
enabled=1
gpgcheck=1
gpgkey=https://packages.icinga.com/icinga.key
metadata_expire=1d

EOF

yum makecache

yum install -t icinga-testing-builds icinga2

 

SLES

rpm --import https://packages.icinga.com/icinga.key

zypper ar https://packages.icinga.com/SUSE/ICINGA-release.repo
zypper ref -fs

cat >/etc/zypp/repos.d/icinga-testing.repo <<EOF

[icinga-testing-builds]
name=ICINGA (testing builds for SUSE \$releasever)
baseurl=https://packages.icinga.com/SUSE/\$releasever/testing/
enabled=1
gpgcheck=1
gpgkey=https://packages.icinga.com/icinga.key
metadata_expire=1d

EOF

zypper ref -fs

zypper in icinga2

 

Windows

x64 and x86.

If you need help with testing the release candidate, or want to discuss new things, hop onto our community forums.

Update 2019-08-01: In case you’re encountering “no shared cipher” errors with 2.10.x agents on RHEL7 or Windows connecting to parent 2.11 RC1 instances, please adjust the “cipher_list” configuration as shown in this issue. More troubleshooting hints can be found here.

Update 2019-08-06: Added backports repo for Debian Stretch. All RC feedback is collected and smaller fixes are applied, nothing which harms testing the RC. Thanks for your ongoing feedback!

 

 

Icinga 2.10.5

Icinga 2.10.5

This release fixes a regression introduced with namespaces in 2.10 where advanced permission filters could result in a crash with many concurrent requests. It also fixes the problem that permission filters are sometimes not applied correctly. Thanks VSHN for sponsoring the development time!

Another problem hidden in the dark was introduced in 2.9 with changed reload signals. This caused the reload shutdown handler to deactivate host and hostgroup objects in the IDO database backend, with later activating them on startup reconnect. Large scale environments would suffer from seconds to minutes not seeing objects in Icinga Web 2 then.

Last but not least, our road to better code quality unveiled that certain compilers don’t treat us well with shared pointers references and resulting crashes. This release fixes crashes with logrotate signals and other timer related problems, e.g. state file handling.

We’ve also added some documentation improvements for the REST API, object types, feature overview and technical concepts with JSON-RPC cluster messages – inspired by the feedback from our Icinga Discourse 🙂

Checkout the full changelog for 2.10.5 here. Please note that SLES11 and Ubuntu14 packages are not available anymore. 

Special thanks goes out to Elias Ohm from novomind AG 🙂 He’s been debugging and fixing the many issues with us together, magnificent team work!

Note: API and cluster/agent related problems will be fixed in 2.11, requiring us to rewrite large parts of the code base. You can help test the snapshot packages and provide feedback. Thanks in advance!

Icinga 2.10.4 bugfix release

Icinga 2.10.4 bugfix release

This release provides fixes for the InfluxDB and Elasticsearch metric writers. If you’re using TLS connections, the latter were not closed correctly. In addition to these fixes, we’ve also backported fixes for delayed and one-time notifications. Special thanks to mdetrano for being patient and testing this one.

Additionally, the Windows wizard has been updated and check_perfmon now supports non-localized performance counters. One of our customers sponsored improving mass-creation of downtimes via the REST API in HA enabled clusters, thank you.

Official packages are available on packages.icinga.com, have been pushed to Chocolatey and Raspbian will follow soon. Meanwhile check the changelog for v2.10.4.