Icinga Performance Monitoring and Analysis

Icinga 2 is a feature “monster”. You can do so much more with it than just “check” and “notify”. Forward your performance data into metric systems such as Graphite or InfluxDB, add the IDO database backend for beautiful dashboards in Icinga Web 2 or connect to the REST API and have Dashing present the latest stats in your office.
After all, Icinga 2 runs as an application on your server and will suffer from outages, full disks, load and memory issues and what not. It shouldn’t happen but what if?
(more…)

Monthly Snap March: Icinga Camp, Partners, Trainings, Icingabeat, approved Puppet module

March was all about our lovely community. We’ve had Icinga Camp Berlin and San Francisco, and also joined FLOSSUK. You’ll also recognize that our Puppet module for Icinga 2 was officially approved by Puppet. Blerim released icingabeat and blogged about it at the Elastic blog. And many more things happened …
We’ve also thought about 1st of April, but hey – we have so many great things to share and work on, we’ll skip it for 2017 😉
(more…)

Icinga Camp San Diego is waiting for you

I’m super excited for next week already – third time flying to the US joining community members at an Icinga Camp, this time San Diego after Portland and San Francisco.
“The Unrealized Role of Monitoring & Alerting” – Jason Hand from VictorOps takes the challenge to dive deep into metrics, alerts and learning from failures. We’ve seen such discussions at Devopsdays Amsterdam in June too and are super excited that Jason joins us in San Diego.
Screen Shot 2016-08-29 at 16.11.23Blerim is also all about metrics – monitor all the things, retrieve metrics and present them in beautiful graphs on your dashboard. Be it Graphite or InfluxDB natively integrated with Icinga 2 or the all new shiny Grafana dashboard we’ve shared on grafana.net.
Rumors do tell that Icinga Web 2 added support for sending commands using the Icinga 2 REST API. Eric will prove that in his talk about new features coming with Icinga Web 2 v2.4.
Yet you’ve been eagerly awaiting a new Icinga Director release. Tom is busy coding and improving it and will showcase the latest and greatest additions – probably firing the last git commit and then starting the talk, as always 😉
vagrant_icinga2_dashingAnything else all around the Icinga 2 API including live demos of existing tools and a revamped Dashing interface are waiting for you in Michael’s presentation.
Last but not least we’ll also have the latest project news and future outlooks with us – Bernd will kick off the Icinga Camp as always.
Still not convinced? We’ll be sharing the venue with PuppetConf 2016 on next Tuesday. If you love automation, you’ll surely want to learn about Icinga and Puppet all week long 🙂
Take a peek into all talks and make sure to join us – it is free, a full day of #monitoringlove in lovely San Diego.
(more…)

Icinga 2 v2.4: New Graphite Schema

icinga2_graphite_web_treeThe Graphite feature in Icinga 2 is pretty cool and simple – just enable the feature and point it to your Graphite Carbon Cache listening address. Graphite 0.9.14 was released just a few days ago 🙂
We’ve received lots of nice feedback on this but also several feature requests which involve breaking changes. Be it the default escaping of metrics (the dash becoming an underscore), turning off additional meta data or a different tree schema.
While working on a Graphite module for Icinga Web 2 we’ve evaluated all these changes and came to the conclusion that we need to change the Graphite tree and layout.
 

Schema

Prefix for hosts:

icinga2.$host.name$.host.$host.check_command$

Prefix for services:

icinga2.$host.name$.services.$service.name$.$service.check_command$

Metrics are written as follows underneath the perfdata level.

<prefix>.perfdata.<perfdata-label>.value

With enable_send_thresholds = true (default is false) Icinga 2 will add the following threshold values.

<prefix>.perfdata.<perfdata-label>.min
<prefix>.perfdata.<perfdata-label>.max
<prefix>.perfdata.<perfdata-label>.warn
<prefix>.perfdata.<perfdata-label>.crit

With enable_send_metadata = true (default is false) Icinga 2 will add

<prefix>.metadata.execution_time
<prefix>.metadata.latency
<prefix>.metadata.state

Sending thresholds and meta data must be enabled as this additional data caused problems in the past in large scale environments.
 

Templates

Time-series databases generally store metrics but no additional meta data as for example PNP4Nagios does to select specific templates for graph representation. PNP uses additional XML files generated on each update. We’ve come up with a solution for that by changing the host and service prefix and adding the CheckCommand for proper template selection by user interfaces.
Furthermore, all services are located underneath “services” on the host to allow easier selection from the applications pulling the data from the Graphite Web API.
 

Metric escaping

The following characters are escaped with an underscore in prefix labels: whitespace, dots, /, \. Performance data labels won’t escape the dot allowing a more selective representation of multiple metric levels returned by the plugin. If your host or service name contains dots, they still will be escaped in order to forbid multiple tree levels and ensure that external applications may properly select the objects.
 

Migration

There is no direct migration path although you can still use the old schema. To prevent unwanted data corruption the new schema is located underneath “icinga2” while the old schema is using “icinga”. In order to restore the old legacy schema, you’ll need to adopt the GraphiteWriter configuration:

object GraphiteWriter "graphite" {
  enable_legacy_mode = true
  host_name_template = "icinga.$host.name$"
  service_name_template = "icinga.$host.name$.$service.name$"
}

Note: The legacy mode will be removed in future feature releases.
 

Test-drive Icinga 2 and Graphite

The easiest way is to use 2 Docker Containers: One for Graphite and icinga/icinga2. I’ve found docker-graphite-statsd which works like a charm also for Icinga 2 development tests.

docker run -d --name graphite --restart=always -p 9090:80 -p 2003:2003 hopsoft/graphite-statsd

The most recent icinga2 Docker container build provides additional options to enable and configure the Graphite feature by passing these environment variables:

  • ICINGA2_FEATURE_GRAPHITE
  • ICINGA2_FEATURE_GRAPHITE_HOST
  • ICINGA2_FEATURE_GRAPHITE_PORT

These two containers must be linked together in order to let Icinga 2 write to port 2003 and Graphite read from port 2003 in two different containers. graphite is the name of the previously started Graphite container.
The following example uses the Docker IP address assigned on OSX (192.168.99.100). Adjust this for your needs where the Graphite Container is actually listening on.

docker run -d -ti --name icinga2 -p 3080:80 --link graphite:graphite -e ICINGA2_FEATURE_GRAPHITE=1 -e ICINGA2_FEATURE_GRAPHITE_HOST="192.168.99.100" -e ICINGA2_FEATURE_GRAPHITE_PORT=2003 icinga/icinga2

Navigate to https://192.168.99.100:9090 (adjust for your container) and select “icinga2” from the tree navigation. You can watch the screencast over here 🙂
icinga2_graphite_web_new_schema