Farewell Icinga API

In the days leading up to the v1.5 release, we bid our Icinga API goodbye and usher in a new API and Web concept.
You may ask yourself, what was this API anyway? Indeed, if you weren’t developing or adapting extensions for the new web interface, you wouldn’t have had much contact with this important project component. When Icinga was conceived, one of the main missions was to facilitate the development of addons and plugins. The API provided a set of commonly used request operations, removing the need to write sql-queries and generally a lot of excess code.
All was well till we decided to offer some extra database flexibility. When we added support for Oracle and PostgreSQL on top of MySQL, we also gave our Icinga API team some extra work. With each change, bug fix or new feature, Marius, Michael L and I had to edit the queries for each database back-end separately. This process was not only complicated and error-prone, but also a sign that we needed a more flexible architecture.
As of Icinga 1.5, the external Icinga API will be replaced by an internal database layer Doctrine, and merged into Icinga Web. Much like before, queries will run through this layer between the database (IDOUtils) and the web interface. However, with Doctrine we can use several database back-ends and querying the database is now much easier. In contrast to SQL, its object relational mapper (ORM) uses Doctrine Query Language, so we now have the flexibility minus the code duplication.

Icinga's new architecture - goodbye Icinga API, hello Doctrine

That being said, queries from the old API still exist, thanks to the ‘legacy layer’ which will transform old API queries into this new ORM type. In this way, we maintain compatibility with addons designed for older Icinga versions. The Rest API is also still there as part of Icinga Web, extending on our Doctrine layer with HTTP for addons that require only certain bits of monitoring info.
With the departure of a standalone API, the average Icinga user will barely notice a change, apart from the fact that the configuration has now been moved to the databases.xml. Best of all, every module developer can now easily access the Icinga database without much code overhead– so addon developers get hacking and let us know how you go!
For more information see our Wiki:
Development Guide for Icinga Web
Icinga Database Essentials
Icinga Web REST API

Icinga IDOUtils – More Improvements Part III

One last shot this time for upcoming Icinga 1.0.1 and IDOUtils:
After getting several core patches into the master and also fixing duplicated service/hoststatus updates being sent to the neb module (thanks to Matthieu Kermagoret) there will be more improvements for IDOUtils.
Since the threaded housekeeper is doing fine, it is possible to periodically clean more tables. By popular demand, the following options have been added to ido2db.cfg

They can be used for your likings, by default they are not set.
If you want to help us test for the upcoming release, you are very welcome to do so!
To help you with GIT, we now have a quite detailed tutorial how to use GIT based on Icinga in our Developer Wiki =)

Icinga IDOUtils – More Improvements Part II

As mentioned in the last post, there are other improvements for Icinga and IDOUtils.
This time, I want to give you a deeper look onto database performance and the housekeeping stuff.
As you might know, selecting, updating or even deleting a row from a table heavily depends on the row count. If table size grows bigger e.g. in the historical tables from IDOUtils, those queries will be slower and hold back the main process. Current approach of IDOUtils is one forked child of ido2db for one idomod connection – working sequentially on the gotten data.
So even one select taking longer will slow down the data processing and worst case the socket will get blocking and idomod complains about writing to data sink.
But how to resolve those issues?
First of all there were several approaches originally found in mysql-mods.sql – setting indexes on table columns which are being used within the WHERE clause. Regarding the fact that ido2db is not just an insert application, but also deletes historical data on demand (table trimming options), selects objects for caching and furthermore updates existing rows (service/hoststatus e.g.) we decided to apply most useful indexes on the table creation statements. It does slow down an insert a bit, but the overall benefit is much bigger than that 🙂
Also the upcoming Icinga Web benefits from that – e.g. the logentries tables select performs a lot faster when using the API and a RDBMS.
But that’s not all – indexes are only one approach of improvement. In the last few months, Hendrik, Christoph and myself discussed a lot about the periodic housekeeping. The basic approach was to remove housekeeping function from the main data processing. Simply because historical deletes on large tables will take even longer and prevent new data being written to the database.
There have been discussions about a cronjob and seperated forked processes for housekeeping, but we wanted something within ido2db and simple to use. So Hendrik came up with the idea to create an own thread within each ido2db child which runs completely seperated from the main data processing flow – the so-called threaded housekeeper.
The thread just waits for the appropriate instance getting connected and then performs the periodic housekeeping – independant from the main flow. And it does not interfere with the normal data processing. So to speak it resolves a big performance issue within IDOUtils.
Basically, this is the way it performs:

  • sleep a while after creation and intialization
  • idle wait for database connection and connected instance from main process
  • perform periodic maintenance not interferring with main process
  • will be terminated when ido2db shuts down

Best thing so far – it has been implemented and tested and improved quite a while. Mostly done in our own git branches, but the final solution is within current git master and will be one of the outstanding new features for Icinga IDOUtils in the upcoming Icinga 1.0.1 release.
Stay tuned for more updates!
… and prepare for Icinga 1.0.1! =)

Icinga IDOUtils – More improvements Part I

It’s been a while since I made several changes to the initial Oracle implementation in Icinga IDOUtils. Code has been split, first start of using prepared statements and binded params with ocilib and some other changes to the code.
In the last few weeks I have been investigating a lot on how to implement more improvements and optimize the critical path of data input from Icinga Core.
I want to start with IDOUtils Oracle, more information on other improvements for Icinga and IDOUtils will follow 🙂
Oracle implementation splits up into several parts taken care of:

  • Rewrite all queries to prepared statements and bind params at runtime
  • Add dynamic procedures for DELETE statements
  • Drop autoincrement emulation by one sequence and insert triggers
  • Add sequences for each table and use INSERT INTO … (id, …) VALUES (seq_name.nextval, …)
  • Add RETURNING id INTO :id for INSERT statements to save one round trip
  • MERGE does not support returning INTO, added SELECT seq_name.currval query instead for fetching last inserted id
  • Rewrite selecting cached objects from DB

The rewritten queries are divided as follows:

  • 1x SELECT latest data time as is (called only at startup)
  • dynamic procedure for DELETE on table by instance_id called at startup for cleaning config/status
  • dynamic procedure for DELETE on tably by instance_id, field compared to time called during periodic cleanup
  • all other queries are prepared with their own statement handler
    • 4x DELETE
    • 52x MERGE
    • 9x INSERT
    • 9x UPDATE
    • 5x SELECT

This summarizes into about 8000 lines (+) and 2000 lines (-) of code modifications 🙂
Furthermore I have been thinking on how to provide an upgrade path for all existing IDOUtils Oracle users. Importing data using the newly applied sequences might lead into errors regarding currval of each sequence. A basic upgrade procedure has been provided already – if you want to try, get the latest GIT master.
Stay tuned for more interesting stories to tell 🙂
… and watch out for Icinga 1.0.1 and fresh IDOUtils Oracle!