We are proud to announce that icinga.com is now also hosting integrations of hardware for monitoring environmental sensors and alerting via text message. The devices from the manufacturers listed on Icinga Integrations can easily be implemented into your Icinga monitoring by using plugins provided on Icinga Exchange.
Alerts via Text Message
Basically, there are two types of devices, which are most suitable for integration with Icinga. First group of them are gateways for sending SMS as alert messages. These gateways work with standard SIM cards and can run on bands in 4G, 3G and 2G networks. These devices also provide more functionality than only sending messages, they are also capable of converting SMS to E-Mail or the other way round or using filters and routing for sending messages only to a certain person or group of persons (phonebook listing). Furthermore, the gateways have their own monitoring server which means that they can automatically send notifications for example in case of loosing connection to the network.
In order to smoothly add these gateways into one’s productive processes, there exist for example plugins for directly connecting to your Microsoft Exchange for full usage of E-Mail to SMS conversion. Integrated with Icinga 2 you cannot only monitor the device itself but also use it in order to forward notifications generated by Icinga. The two main manufacturers of these gateways are Braintower located in Germany and SMSEagle with their headquarters in Poland.
Second type of integratable devices are mainly meant for measuring and surveillance. These consist most of the time of a base unit and several ports for connecting sensors. The base units can either directly communicate via LAN or via GSM in case of remote locations. For full environmental monitoring, there are plenty of sensor types that can be attached:
- combination of temperature and humidity
- air pressure
- air flow
- water and leakage
- smoke and Gas
- levels of liquids like pertrol
- movement, vibration and opening of doors
The main manufacturers for measuring devices are currently AKCP, GUDE, HW group and Tinkerforge. For all these you can find monitoring plugins on Icinga Exchange. The manufacturer GUDE also offers power supplies that cannot only monitor voltage, but are also capable of remote controlling plugs and devices.
Integrating Tinkerforge with Icinga
In order to have a closer look at how these devices can be integrated with Icinga 2, Tinkerforge is one of the best examples. The whole plugin project can be found at https://exchange.icinga.com/netways/check_tinkerforge and shows not only the integration but also offers an insight into the functionalities of the device.
For using the check_tinkerforge plugin it is required to run Python 2.7+ and to have the tinkerforge Python library from Pypi installed. After this, installation is done with
pip install tinkerforge and moving the plugin into the Icinga
Now let’s have a look at the options of the plugin:
$ ./check_tinkerforge.py --help
usage: check_tinkerforge.py [-h] [-V] [-v] -H HOST [-P PORT] [-S SECRET]
[-u UID] -T TYPE [-w WARNING] [-c CRITICAL]
-h, --help show this help message and exit
-V, --version show program's version number and exit
-H HOST, --host HOST The host address of the Tinkerforge device
-P PORT, --port PORT Port (default=4223)
-S SECRET, --secret SECRET Authentication secret
-u UID, --uid UID UID from Bricklet
-T TYPE, --type TYPE Bricklet type. Supported: 'temperature', 'humidity', 'ambient_light', 'ptc'
-w WARNING, --warning WARNING Warning threshold. Single value or range, e.g. '20:50'.
-c CRITICAL, --critical CRITICAL Critical threshold. Single vluae or range, e.g. '25:45'.
-t TIMEOUT, --timeout TIMEOUT Timeout in seconds
As the TinkerForge device supports sensors like temperature or humidity as well as other types, we need to indicate what sensor should be checked by using the
check_tinkerforge.py -H 10.0.10.163 -T temperature -w 23
WARNING - Tinkerforge: Temperature is 24.75 degrees celcius|'temperature'=24.75
In case you have more multiple sensors of one type, you are required to indicate the UID with the
--uid operator in order to correctly identify the sensor in question. Furthermore, thresholds can either be indicated as single values or value ranges and by using the usual operators for warning and critical.
For integration of the sensors into your Icinga 2 configuration, you need to create a
CheckCommand object like for other check plugins. The device itself will be created as
Host object and all sensors should be covered with an
apply Service. For your comfort you can find an example config for TinkerForge here. This example can directly be used, it is only necessary to change the parameters and in case you have multiple sensors of one type, to add the UIDs.
All manufacturers mentioned in this blog post can be found on the Icinga integrations website including information on the companies, the devices and the links to the corresponding monitoring plugins.
When we hunt down problems in Icinga setups we ask for logs most of the time. While you get used to sifting through logs and collect some bash magic during the process there’s always the wish for this routine to be easier and especially faster. If you get logfiles from several days where each of the nodes produces millions of logfiles per day, every time you start your grep’s over and over get’s you madder and madder. So I started searching for a solution.
Having experience with Elastic Stack setups I always wanted to have an easier way of parsing Icinga logs with Logstash. I built some setups with some basic rules before but they were just starting points. The
icinga module in Filebeat helps with that as well but it also just parses the metadata of the logfiles and not the logs itself.
Like every decent IT person ( 😉 ) I run my own installation of Icinga and Elastic Stack for my personal systems so one day I started building filters that would parse the Icinga logfiles as much as possible. When this project matured I decided that I wanted to open source it so everyone running the same combination could benefit. I tried making it as easily usable as possible so I added every bit of configuration you would need to the repository on GitHub. It’s all said in detail in the Readme of the project but what you basically need is to check out the repository in a directory, use this directory as configuration for a Logstash pipeline and use Redis (with predefined keys) to get the data into and out of this pipeline.
Meanwhile the project got a lot of attention from Alexander Stoll, an employee at Netways, one of our Icinga partners. This means, the project grows even faster and having a second pair of eyes looking at your code is always a great help. Thank you very much!
What you get if you apply this pipeline:
- More and more of all possible log events from Icinga get parsed. So you will get fields holding the information about the host/service/object an event refers to, queue lengths, pids, timestamps and so on. You can build your own dashboards out of this information or just use the example ones the project is providing. They are great for filtering your events, too.
- The pipeline tries to follow the Elastic Common Schema as closely as possible. So you will be able to use your logs for upcoming features like SIEM
- The pipeline adds a field called
eventtype to every event, so you can filter more easily and there’s even the possibility of having a knowledgebase to all the events when there’s enough interest (and help) from the community
- You get information like the host writing the log or the Icinga facility providing the information. This can be used for filtering more easily for the information you really need. Of course the severity has also its own field and filterable
What this ruleset brings is not only the possibility of users parsing their own logs but one big benefit for Icinga partners providing support. Netways has set up a centralized Elastic Stack running in Netways Web Services where their support squad can access logs we have been sent by their customers. To get the logs into this centralized system they use a dockerized Filebeat installation which reads some extra information (customer id, ticket id) from an .ini file and adds them to every event.
The Elastic Stack uses this information not only as part of the events being written into Elasticsearch but also to create the index name where the information is written, too. This way they stray from the default where the index name has a timestamp (or increasing counter when using ILM). This leads to indices having very different sizes which doesn’t matter because they don’t need a tuned high performance stack in this case. But they get the benefit that they can cleanly delete all logevents about one specific ticket when they are done.
As an example what you can do with this setup: There was a ticket about a problem with notifications being triggered despite an object being in downtime. So the customer sent all logs from their Icinga masters and the agent involved. All of them where sent to the centralized setup (with the filename of the log representing the server name) and they started digging. Narrowing down the window of events being shown to the timeframe from when the downtime was created and when the notification was triggered helped in the first place. Searching for the name of the agent being involved showed the downtime being created and the notification being sent.
They could easily change between several levels of information and get more or less information for debugging. Since the information came in more than 40GB split into several logfiles it would have taken them quite a long time to refine their search every time they wanted to change something. Using the Elastic Stack they could make changes within seconds.
As with every open source project feedback is always very welcome. Be it mails, issues on GitHub or Pull Requests. Things like this exist for the community and through the help of the community. By now there are not all possible events being parsed but the ruleset get more and more complete over time. If you find events with the eventtype
undefined please consider writing rules and issuing a pull request. By the way – during the process of working on these rules we discovered one or another way to improve logging in Icinga itself, and created issues or pull requests.
If you want to meet and have a chat about combining monitoring with logmanagement you can catch me at the upcoming Icinga Meetup in Linz, Austria or listen to my talk about this very topic at the Icinga Camp in Stockholm, Sweden.
Last week I told Blerim that Icinga came to life on 6.5.2009 … really, 10 years already?
A small group of Nagios community members stood up and wanted to create more than a monitoring core with an enhanced web interface. APIs, integrations, backends, scaling, metrics, … A lot of passion and love was received from the community and actually, the Nagios ecosystem became more active again. We tried many things to improve the existing code base and limitations in both backends and frontends. TL;DR – at some point we decided to throw it away and start fresh. The rest can be read on our blog 🙂
service icinga restart
Back in 2012, Gunnar, Bernd and Marius paid a visit to Vienna. They had something with them what is known to be the first prototype of Icinga 2, codename “Strawberry“. Moving from C into C++ was a whole new world, Golang didn’t really exist back then. Oh, it can run on Windows too. Oh, such speed and performance. Oh, a new playground for our crazy ideas.
Before its first 2.0 release five years ago we had many design rounds with service sets on the Host object, virtual host checks from services and so on. This changed quite a bit after strong opinionated discussions in early 2014. Those are also the origin of the magic apply rules, with mimicking the “put a service on a hostgroup, each host member inherits it” object trick from Icinga 1. Nowadays you can generate monitoring configuration and make things easier.
“Monitoring as a code” became a thing when we had created our own DSL (domain specific language) which added functions, conditions, even loops from community feedback. We’ve also learned a lesson here with the Director released after 2.4 – not everything from inline DSL code is suitable for a configuration frontend. Also that we now have 3 APIs – Core, Web, Director – users might be confused. This is a task we will be working on in the future to unite the product stack better.
Speaking of web and modules, Icinga Web 2 started with the requirement of a general framework including web and CLI based on the Zend framework. IIRC this was 1,5 to 2 years after Icinga 2 kicked off, also explaining the version difference – 2.7 vs 2.11. Web version 1 had many short-comings and was complicated with extensions – unless you’re an experienced developer. With the power of making everything a moduie, even monitoring, the Icinga Web framework inspired the many of you.
Community work is a passion
One of the first community modules I’ve seen was the Globe module which I had shown at Icinga Camp Berlin 2017. Nicolai was so inspired to start the development of the map module. I was extremely happy that I’ve tested it quite extensively and sent a docs PR. Literally the same happened with Carsten creating the famous Grafana module – a gentle README, release hints and troubleshooting – the first impression counts. Did I mention the awesome themes already?
Pushing community ideas and spirit forward is one of the most important things in the way we have become Icinga. Without you and your shared love and criticism, we wouldn’t be here and still be sitting somewhere around the globe ranting about tools.
Back then, our community was a bit split, and not everyone had the time to answer questions on IRC, the mailing lists and the community forum at Monitoring Portal. While we want to be transparent with our development workflow, release dates and design ideas, it’s always time consuming and hard to catch up. With Icinga now supporting distributed environments with zones, endpoints, and web views and configuration forms, we needed something where text formatting and screenshots really integrate well. In late 2018, we’ve decided to create our own Icinga Discourse inviting the team, partners and users to discuss. I just love it, helping others really is a good feeling.
As you may know, the majority of the Icinga development is and was funded by NETWAYS, up until 2012 the University of Vienna sponsored my time. This is paid time and helps with creating the foundation for new features and maintenance. We also raised awareness on why funding and sponsoring is a good thing – if there is a bug biting you, or a missing feature you’d like to use in production, you can speed things up by requesting a quote, or even, by getting a support subscription through one of our partners. Sounds expensive? Maybe. Helps funding the future? Definitely.
Still, we love community contributions and value your (spare) time a lot. Sometimes contributions are hard to achieve as the code is not clear, or you actually don’t know where to start. In the past, we had our own Git server and ticket system. It worked somehow – with the move to GitHub and social coding with Pull Requests, reviews, discussions and a sound platform our daily work on Icinga has become a breeze. The help of issue templates, continuous integration (CI) on when a PR is created, and inline code comments puts the focus back on development.
Lastly, we’ve started documenting our core technical concepts and improved the development guidelines. This now also includes instructions for nightly snapshot builds.
Turning back the time a bit
Icinga 1 didn’t have packages at first glance. You had to call configure/make and pass some parameters for development libraries and headers, sometimes it didn’t work. The core was to compile and then it broke on your system, but the developer’s system worked fine. Ok, let’s create a VM for SuSE in Virtualbox … well, where is my yum?
One of the first ideas was to invite upstream packagers onto the team and learn from each other. Our many releases couldn’t reach stable distributions that fast, so we needed additional repositories. There’s Debian Backports, custom Ubuntu PPAs, EPEL for RHEL/CentOS and lastly, the SuSE OBS repos – not to forget all the other distributions in the Linux/Unix world, FreeBSD, OpenBSD, NetBSD, Gentoo, ArchLinux, … For the main officially supported platforms, we decided to create our own repository at https://packages.icinga.com.
Creating and maintaining packages next to the build infrastructure for each project, language and dependencies is a huge often underestimated effort. Our build server based on Jenkins made several iterations over the years, lately we decided to replace it with GitLab CI for less security risks and better configuration. CI and CD is just great, and with Docker containers nearly everything can be built – requiring your self-hosted infrastructure, maintained by the awesome managed service team at NETWAYS. This includes deployments with Foreman and Puppet, and of course monitoring with Icinga just like “eat your own dogfood” 🙂
Another thing which really was hard to achieve – ARM based architecture on Raspberry Pi. Sounds easy to just compile the binary packages on a different architecture. Actually we had to tackle compiler bugs and long running builds – we started a thing at the OSMC hackathon 2018 and Nicolai’s employer sponsored the last missing days on the road to official packages.
We can always improve
Looking back where we started 10 years ago, we’re in a very comfortable situation. This also includes making the initial setup a breeze, adding lots of documentation and helping on the community channels when things are on fire. With partners providing support, trainings and consultancy, the enterprise requirements are fulfilled even better.
In terms of new projects and ideas, it always turns out that we want to create 4 things, then 10 different customer issues arise, the proof of concepts don’t work out well and obviously, there’s only resources for one or two things being released later. Right on, many have asked about notification managers or mobile apps – that’s on hold for now, finishing the ongoing tasks with Core and Web major versions, Icinga DB and Icinga Reporting, and additional new module version for certificates (x509), Vsphere, Director and Business Process. Again, focus is required – you cannot manage 10 things in parallel. That being said, the monthly snap blog will be moved into a combined effort with the enhanced newsletter telling you about the latest and greatest developments. We also continue to have bi-yearly strategy workshops to refine future roadmaps and discuss changes in our ecosystem.
Those things are great to read, but even more, they are great to talk about. Our idea of meeting at Icinga Camps worked out pretty well, Berlin is by far the biggest one thus far. OSMC with the “State of Icinga” talk by Bernd is great and funny way to share the latest stories too. Our first Icinga team meeting was at OSMC 2009 with many more to come at Icinga Camps all over the world, where you’ll meet, chat and laugh. Our first Icinga Camp San Francisco 2014 happened at GitHub HQ. This is an incredible memory and kicked off the many community events after. In the past months, our community in Germany and Austria have been going strong with regular meetups, that’s truly inspiring. Not been there, or want to create your own meetup group? It is not too late.
We are also aiming for something bigger for everyone planning for next year: IcingaConf in Amsterdam in 2020.
Our way of not re-inventing the wheel but building our integrations with proven tools will continue. Today we are proud to enhance Elastic Stack with adding Icingabeat, Logstash outputs/filters and metric writers even. Adding monitoring as first class citizen in your DevOps lifecycle has become reality with official modules for Puppet, Ansible and Chef and a deeper integration into Foreman. If this isn’t enough, there’s more cloud provider integrations such as Terraform, AWS, Azure and whatnot. Enhancing metrics with Graphite and InfluxDB, provided with example Grafana dashboards and web module integrations. Dashing is still a thing, and reporting has come in its first early version for more SLA and visualization magic.
This is something we couldn’t imagine 10 years ago. We are also proud to work with many professionals and open source lovers all over the world, spreading the #monitoringlove.
The people behind, around and with Icinga
That’s not only me, I hear that often when people find my name on Google. This is a group of passionate people, each with their own character, ideas and responsibilities. Over the years, fellow friends went with us on our journey and left again for their new projects and private life.
Karo, who created the initial designs, blog posts and social media for Icinga. Amanda, who tackled technical problems and spread the love for Icinga on social media and press releases. Gunnar, who is the main architect of Icinga 2, and a passionate coder, the brain behind the DSL and many more. Hendrik who forked Icinga 1 as core and taught us how to use Git and debug the IDO backend.
Michael who worked many hours on Icinga Reports with Jasper, creating fancy web views. Thomas who took over the Oracle backend improvements for Core 1.x. Christian who tackled the Oracle backend in Icinga Web 1, followed by Jannis who enhanced version 1, created mobile 1 and helped create Icinga Web 2 back then. Scott who joined the Skype sessions from Australia, changing our German project into an English one.
Lara and Wolfgang who tackled the version 1 documentation in docbook xml, making our later decision for Markdown very easy. Christoph who created the first Icinga Core RPM packages, Shawn who tried to bring Icinga packages into EPEL, Sam helping here. Carl who’s been working on Solaris packages for 1.x. Nick and Tom paving the way for the first Puppet module. David who created the first VM appliances. Franz and Mike took care of test scripts, Carlos created the vim/nano syntax highlighting for the Icinga DSL, Jordan who’s been working on Docker containers.
Matthew who created the header status bar for Classic UI, Ricardo who took over improving the old CGIs with many enhancements. Rune who did everything testing and created the term “release fucker” back in the 1.x days – when developers didn’t test enough. Massimo helped with 1.x core issues back then. Alex created and maintained the Debian/Ubuntu packages for many years, Tim who did the same for SuSE. Gerd who worked on Icinga 2 reload and threading techniques. Simon who created the InfluxDB feature for Icinga 2.
Thank you deeply.
Many thanks also to Thomas Krenn who sponsored hardware for the initial build server, JetBrains, Navicat, Atlassian & Activestate for granting us open source project licenses back then.
Bernd, Marius and Julian are the remaining co-founders of Icinga. I was a little late to the party in May 2009, with the rest of us enjoying Icinga with our friends, partners and users every day. You’ll recognize Eric for our technical strategy and everything web and core, Blerim improving and managing all the products, helping partners, maintaining integrations, Tom pushing forward with designs and architecture and the shiny Director, VSphere, etc. modules, Markus taking care of QA, tests and the heavy build infrastructure, followed by Director development lately.
Jean moving along from Icinga 2 core into IcingaDB, followed by Noah with passion for Golang. Our UX designer Florian shaping the web in all its beauty, Johannes taking the lead for Icinga Web and many new modules, followed by Feu with her passion for CSS and better frontend widgets. Alex being the Graphite module developer who recently joined the core devs with the major network stack rewrite, helping my duties on leading the core development whilst sharing all the knowledge with our trainee Henrik. Thomas taking care of Icinga support and improving everything around it. Julia joined for spreading the love on social media and events.
Michael (mcktr) with his passion for the unforeseen Windows issue, Assaf taking care of automation with the official Ansible module, Lennart spreading the love for Puppet while Dirk is our SELinux and RedHat professional. Christian whose day job is in Sales, codes Windows PS at night, sharing his deep knowledge. Lars does a magnificent job on the FreeBSD ports, Matthew dives deep into Gentoo. Bodo is well known for his love for Ruby and Docker containers, appreciated by the community.
Carsten builds awesome Icinga Web modules, Nicolai does so too and jumps right into Director import source development as well. Kevin created a Python API library and knows Ansible&Director, Tobias did so too whilst testing quite a lot. Marianne spreads the Icinga love in her blog, OSMC talks and all over the Internet, even iX articles. Elias analysed, debugged and fixed critical bugs for Icinga 2.11. Virender creating and shaping the Chef cookbook. Our many friends at RedHat helping with the Stackguard Kernel problems.
Timo and Matthias believing in us in trouble times, and letting us test in production, Emanuel who did so too with a large cluster at Contintental. Brad who brought passion and fun from the US, What the Heck from the Netherlands as well. Dave always coming from Australia, and spreading the love (and TimTams). Marcel joining fresh and creating howtos. Jörg who believes in Open Source and still maintains PNP. Michael who develops NSClient++ in his spare time, running on hundreds of thousands Windows clients. Philipp talking things Elastic with Icinga, Jens sharing the love from Müller to the world. Sven & Michael for caring about Livestatus. Moritz spreading the love in Austria at meetups, Bernd taking care of Graylog Vagrant boxes.
Bas maintaining the Icinga packages in Debian. Rico convincing his boss to early adopt Icinga 2. Marco and Max finding all the nasty bugs in larger environments. Claudio creates plugins for ESXI monitoring and shares many howtos. Marc tackles core notifications and Director integrations. Morgan maintains the fancy Grafana notification scripts. Kris thought of the DevOps culture and believes in Icinga. The many of you improving the Icinga Template Library (ITL) commands, sending Graphite templates, documentation updates, issues, pull requests, etc.
Our many friends at customers and users who convince their managers to use and support Icinga. And to share the love – publicly talk about it. Whenever a new Audi or VW is built, Icinga watches. Whenever you buy things online, do bank transactions, go shopping, watch TV, use your mobile phone or enjoy life – Icinga takes care that it works. And when you look up the sky, maybe you can spot the ISS with Icinga exploring space.
There are many more great Icinga users, you know who you are. Thank you from our hearts for an incredible journey.
You know, we are party people celebrating after hard work. Say cheers with your favourite drink, I’ll have – obviously – a G&T.