Announcing Icinga 2.16.0 and 2.15.3

We are happy to announce the release of two new versions of Icinga 2 today, 2.16.0 and 2.15.3. The first one includes some new features highlighted below, as well as a number of bug fixes and other improvements. The latter one is a small bug fix release that brings some of the other fixes included in 2.16.0 to the 2.15.x branch as well.

OpenTelemetry Metrics Writer

In the last year, we have received surprisingly many requests for a single feature: OpenTelemetry support. As the name already suggests, OpenTelemetry is an open standard for telemetry, which includes, among many other things, a wire protocol for sending metrics. Icinga 2 now features a new writer that sends the performance data metrics collected by the individual check commands to compatible receivers using the OpenTelemetry Protocol (OTLP).

For one, following a vendor-neutral standard also allows integrating Icinga 2 metrics into more metrics backends than before. For example, it is now also possible to write metrics directly to Prometheus as it has an OpenTelemetry receiver as well.

Additionally, it is supposed to replace the existing ElasticsearchWriter. We considered fixes to the dedicated writer’s index template and support for the Elasticsearch data stream feature in the 2.16 development cycle, but then we realized that Elasticsearch supports ingesting OpenTelemetry metrics as well, including support for data streams on newer Elasticsearch versions. Thus, we focused on the OpenTelemetry writer and also recommend using it instead. As a result, we are deprecating the existing Elasticsearch writer.

The OpenTelemetry writer also has the potential to replace other dedicated writers in the long term as well, but no final decisions on that were made yet.

License Switch to GPL Version 3 or later

Starting with version 2.16, Icinga 2 is now licensed under the GPL version 3 or later, changed from GPL version 2 or later.

This change is required because in order to integrate OpenTelemetry, we had to include new dependencies that are licensed under the Apache 2.0 license. This license is incompatible with version 2 of the GPL, but is compatible with version 3.

HTTP Response Streaming

If you have a large installation, are a heavy user of the Icinga 2 REST API, and have had trouble with high RAM utilization by Icinga 2, this release is for you!

So far, one of the factors that could heavily bloat up the memory usage of the Icinga 2 process was that it always allocated the full HTTP response in memory before sending it to the client. For one, with enough host and service objects, responses of tens or even hundreds of megabytes aren’t uncommon, which in itself increases the memory requirements while processing the API request. In addition, that memory allocation pattern is quite demanding for the system memory allocator and can easily result in memory fragmentation, growing the memory usage of the process over time.

In contrast, starting with v2.16, where applicable, Icinga 2 will now generate the response JSON on the fly while writing the response to the client. With this, it only needs to keep a few kilobytes in memory at a time, greatly reducing memory usage. With this change, some responses are now sent using chunked transfer encoding, which is handled transparently by standard-compliant HTTP clients, so the change should be transparent for users.

Updated Container Images

We have completely reworked the build process of our Docker container images. The changes should be mostly transparent to users and existing setups continue to work without changes.

One important change is that we now let the Docker daemon initialize the contents of the volume. This now allows to mount additional configuration files below the /data volume without those breaking the volume initialization. However, this comes with the drawback that if you now bind mount a host directory to /data, it will not be initialized properly anymore and you should use a named volume instead. If you use such a bind mount for the volume and it was already initialized in the past, it will continue to work.

Deprecated Features Scheduled for Removal in v2.18

We believe it’s finally time to cut the ties and shed some ballast. This means that for some old features that were left in limbo for quite some time, there is now a concrete plan for removal. The full list of affected features, including some that are newly deprecated, is:

• New deprecation: ElasticsearchWriter:
  Please use the new OTLPMetricsWriter instead.
• New deprecation: User declared namespace objects like namespace foo {}:
  Please define your functions or variables outside of namespaces.
• Deprecated since v2.14: IDO (IdoMySqlConnection and IdoPgsqlConnection):
  Please switch to Icinga DB.
• Deprecated since v2.12: CompatLogger, ExternalCommandListener, and LivestatusListener:
  Please switch to using the Icinga 2 API.
• Deprecated since v2.12: Windows check plugins bundled with Icinga 2 (check_*.exe) and the corresponding CheckCommand definitions:
  Please use our PowerShell Plugins coming with Icinga for Windows instead.

All of these are due to be removed in version 2.18.0. Deprecation warnings were added or updated to notify users about the scheduled removal.

And Much More

In addition to what is highlighted above, v2.16.0 includes many other smaller fixes and improvements. Please refer to the full change log for all of that.

Icinga 2 v2.15.3

Together with v2.16.0, we are also releasing a small bugfix release for the v2.15.x maintenance branch, v2.15.3, which includes some of the fixes from v2.16.0 as well. Some possible crashes are fixed, the OpenSSL version included in the Windows builds is updated, and error reporting in the InfluxDB writers is improved. If you are interested in the details, please check the change log for that version.