Monitoring as Code with Icinga: Approaches, Tools and Trade-offs

Monitoring as code means defining hosts, checks, thresholds and notification rules in version-controlled files, and deploying them through review and automated validation, the same way infrastructure as code handles servers and networks. With Icinga you can do this with plain configuration files in Git, the Ansible collection maintained by NETWAYS, the Puppet modules maintained by Vox Pupuli, the Icinga Director, or the REST API from tools such as Terraform.

Discussions about monitoring as code often center on Terraform providers for SaaS platforms. For many Icinga users, keeping monitoring configuration in Git is nothing new; it is how they have worked for years.

This guide covers what monitoring as code requires, which teams benefit from it, what belongs in code and what doesn’t, and how each approach works with Icinga, including where each one stops being the right fit.

What is monitoring as code?

Monitoring as code is the practice of defining what you monitor, how you check it and who gets notified in machine-readable files kept in version control. Changes are reviewed, validated automatically and rolled out by a tool, not typed into a web form on a production system.

The idea comes from infrastructure as code (IaC). Martin Fowler describes IaC as defining infrastructure through source code that is kept in source control, tested, and delivered with the same discipline as any other software (Fowler, 2016). The practice goes back to CFEngine in 1993; the term itself took hold around 2008 and 2009, when people close to Puppet and Chef started using it (O’Reilly, Infrastructure as Code).

Monitoring as code applies the same thinking one layer up. If a server is built from code, the checks for that server should be built from code too, ideally in the same commit. You will also come across related terms such as observability as code (dashboards, alert rules and SLOs, often in SaaS platforms) and configuration as code. They describe the same principle applied to different tools.

For Icinga users the concept is not new. Since Icinga 2.0, the configuration has been a text-based language with templates, apply rules and functions. That is already code. The open questions are where that code lives, who changes it, and how it reaches your monitoring servers.

What monitoring as code requires

Infrastructure as code only works when a few conditions are met, and monitoring as code inherits all of them. Tools help, but most of these requirements are about the system you monitor with and how your team works.

1. A declarative, text-based configuration. The monitoring system must accept its configuration as files or through an API, not only through clicks in a GUI. Icinga 2 reads its own configuration language from files, and the Icinga 2 REST API accepts objects and whole configuration packages.
2. Composable building blocks. A point that is easy to miss: “as code” only works if the platform’s objects are modular and can be combined. Otherwise you end up pasting exported JSON into a code file. In Icinga, templates can import other templates, apply rules attach services based on host attributes, and apply for creates one service per entry in a list or dictionary.
3. Version control as the single source of truth. Every change has an author, a timestamp and a reason. If someone changes production by hand, the next deployment overwrites it, and that is intentional.
4. Repeatable, idempotent deployments. Applying the same code twice must produce the same result. Configuration management tools like Ansible and Puppet are built around this.
5. Automated validation before rollout. icinga2 daemon -C checks the complete configuration and returns an exit code your pipeline can act on (Icinga 2 technical concepts).
6. Safe rollout and rollback. On reload, Icinga 2 validates the new configuration in a child process; if validation fails, the running process continues with the old configuration (Icinga 2 technical concepts). Rolling back in Git is a git revert.
7. A source of truth for your inventory. Monitoring code needs to know what exists. That can be an Ansible inventory, PuppetDB, a CMDB, a cloud API or a hand-maintained list, but there should be exactly one.
8. A plan for secrets. API passwords, SNMP communities and notification tokens do not belong in plain text in Git. Use Ansible Vault, encrypted Hiera, or your CI system’s secret store.
9. Team habits. Merge requests, reviews and clear ownership of directories or templates. These are usually harder to introduce than any of the tooling.

Is monitoring as code right for your team?

Monitoring as code fits teams that already work with Git, reviews and automation in other parts of their infrastructure. It is not automatically the better option for everyone: it adds process, and that process has to pay for itself. Whether it pays off depends less on company size than on existing workflows, skills and preferences. Four questions help:

• Who makes changes to monitoring? One platform team, or many application teams, or a NOC that adds hosts all day?
• How often does the monitored environment change? Weekly, or every few minutes through autoscaling and CI deployments?
• Does the team already review code? If merge requests are part of daily work, monitoring changes slot in easily. If not, you are introducing two things at once.
• Where does the inventory live? If servers are already described in Ansible, Puppet or Terraform, the monitoring definition can come from the same place.

Mixed setups are common and perfectly valid. A typical split: a platform team keeps templates, check commands and apply rules in Git, while hosts arrive automatically from an inventory source or are added by other teams through the Director.

Two rules of thumb help. Use the infrastructure-as-code tooling you already run rather than adding a new one. And invest where code lets teams onboard themselves, with an audit trail instead of tickets; for a traditional operations team or a NOC, a full monitoring-as-code workflow may be overkill. Writing code for something you configure once rarely pays off.

Two building blocks: configuration files and open APIs

Monitoring as code rests on two things: configuration that lives in files, and APIs that let other tools act on your monitoring. Files hold the desired state; APIs connect that state to the rest of your automation.

Configuration files

Plain-text configuration files give you history, review and reproducibility without any extra product. Everything else in a monitoring-as-code workflow builds on these properties.

• History and audit trail. git log and git blame show who changed a threshold, when and why. That answers audit questions and shortens incident reviews.
• Four-eyes principle. A merge request lets a colleague see a change before it reaches production, including the diff of every affected object.
• Reproducibility. If a monitoring server is lost, the configuration can be rebuilt from the repository. The same repository can feed a staging instance.
• Reuse through templates and apply rules. In the Icinga 2 configuration language you describe rules once, and new hosts pick up the right checks based on their attributes.
• Bulk changes. Renaming a variable on 2,000 hosts is a search-and-replace and a review, not an afternoon of clicking.
• Testing. Files can be linted and validated in CI, and a throwaway Icinga instance in a container can load them before production does.
• No lock-in. Text files work with any editor, any Git server and any CI system.

A short example shows the idea. The host only declares what it is; the apply rule decides which checks it gets:

template Host "linux-server" {
  import "generic-host"
  vars.os = "Linux"
}

object Host "web01.example.com" {
  import "linux-server"
  address = "192.0.2.10"
}

apply Service "ssh" {
  import "generic-service"
  check_command = "ssh"
  assign where host.vars.os == "Linux"
}

Add a hundred more hosts with import “linux-server” and each one gets its SSH check without touching the service definition.

Open APIs for custom workflows

Files describe the desired state; APIs handle everything that happens in between. Some monitoring APIs exist mainly to serve the vendor’s own dashboard and its import and export functions. An API that is useful for automation has to cover configuration, actions and events, and has to be documented well enough to build on.

The Icinga 2 REST API lets you query, create, modify and delete configuration objects, run actions such as rescheduling checks, subscribe to event streams, and manage configuration packages. Permissions are set per API user and per object type, so a CI job can be allowed to schedule downtimes without being able to delete hosts. The Icinga Director REST API works with the same JSON objects you see in its web interface.

Typical workflows built on these APIs:

• Provisioning and decommissioning. The pipeline that creates a VM registers it in monitoring; the one that removes it deletes the host, so nobody gets paged for a server that no longer exists.
• Deployment-aware monitoring. A release pipeline schedules a downtime before restarting services and removes it afterwards.
• Self-service for application teams. Teams add their own services through an internal portal or a merge request, while templates and notification rules stay with the platform team.
• ChatOps and ticketing. Acknowledgements, comments and status queries from chat or ticket systems, using the event streams to react to state changes.
• Automation in the other direction. Icinga can act as a dynamic inventory for Ansible, so playbooks target exactly the hosts that are currently monitored.

Scheduling a downtime from a deployment job takes one request:

curl --cacert ca.crt -s -u deploy-bot:$ICINGA_API_PASSWORD \
  -H 'Accept: application/json' \
  -X POST 'https://icinga-master.example.com:5665/v1/actions/schedule-downtime' \
  -d '{ "type": "Host", "filter": "host.name==\"web01.example.com\"",
        "start_time": 1760432400, "end_time": 1760434200,
        "author": "ci-pipeline", "comment": "Release 4.2", "all_services": true }'

Because the interfaces are plain HTTP and JSON, any tool can use them: Ansible, Puppet, Terraform, Python scripts or workflow tools like n8n. You are not tied to one integration path, and you can change tools later without changing your monitoring backend.

What belongs in code, and what doesn’t

Configuration belongs in code; operational state does not. Drawing this line early prevents most of the friction teams run into later.

The grey zone is configuration changed at runtime. Objects created through the REST API are stored by Icinga 2 in an internal package under /var/lib/icinga2/api/packages/_api, and attributes changed at runtime are kept in /var/lib/icinga2/modified-attributes.conf. Neither shows up in your repository. That is fine for a temporary change during an incident, but it turns into drift when a threshold is quietly raised and never reverted.

Three habits keep this under control:

• Write down which tool owns which object type, and make it part of your repository’s README.
• Give API users only the permissions they need. A provisioning pipeline may create hosts; a chat bot only needs actions such as acknowledging problems or scheduling downtimes.
• Look for drift now and then. Every object exposes its package and original_attributes through the API, which makes runtime-created objects and modified attributes easy to find.

Approaches to monitoring as code with Icinga

Every approach ends up in one of three places: configuration files on the Icinga 2 config master, objects created through the Icinga 2 REST API, or the Icinga Director, which itself deploys through that API. From the config master, Icinga’s config sync distributes the configuration to satellites and agents.

• Configuration files in Git with a CI pipeline. Icinga 2 configuration files live in Git and are copied to the config master or uploaded as a configuration package; config sync distributes them. Fits teams that already use CI and want full control.
• Ansible collection (NETWAYS). Objects are defined as YAML in the inventory and host_vars, and the role renders configuration files on the config master. Fits teams that manage servers with Ansible.
• Puppet modules (Vox Pupuli). Objects are defined in Puppet manifests and Hiera; defined types write configuration files, and exported resources collect them on the master. Fits teams that manage servers with Puppet.
• Icinga Director. Objects live in the Director database, with templates, import sources and sync rules exportable as JSON baskets. The Director deploys configuration packages through the Icinga 2 API. Fits teams that import large inventories or prefer a GUI.
• Director API via Ansible. Objects are defined as Ansible YAML and sent to the Director REST API, followed by a Director deployment. Fits teams that use Ansible but want the Director’s interface.
• Terraform provider. Objects are defined in Terraform code and created through the Icinga 2 REST API. Fits teams that provision cloud resources with Terraform.
• Custom scripts. Your own code calls the Icinga 2 or Director REST API. Fits teams that build self-service portals or special workflows.

These paths can coexist. The Director does not write to /etc/icinga2 and can run alongside manual configuration or configuration shipped by other tools (Icinga Director docs). What you should avoid is two tools owning the same object: decide per object type which path is authoritative.

Configuration files in Git with a CI pipeline

The most direct form of monitoring as code with Icinga needs no extra tool: keep the Icinga 2 configuration in Git, validate it in CI, and deploy it to the config master, a GitOps-style workflow. It is also the natural first step before adding Ansible, Puppet or Terraform.

A typical workflow:

1. Structure the repository like the configuration directory. Mirror zones.d with one directory per zone, plus global-templates for templates and commands that every zone needs. One file per host or per team keeps merge requests small and avoids conflicts.
2. Validate every merge request. A CI job loads the repository into a container or validation host with the same Icinga 2 version, zones.conf and constants as production, then runs icinga2 daemon -C.
3. Deploy from the main branch only. Either copy the files to the config master and reload, or upload them as a configuration package through the REST API.
4. Let config sync do the rest. The config master distributes the zone directories to satellites and agents.
5. Roll back with Git. git revert and a new pipeline run restore the previous state.

A simplified GitLab CI pipeline:

stages: [validate, deploy]

validate:
  stage: validate
  # internal image with the same Icinga 2 version as production
  image: registry.example.com/icinga2-validate:latest
  script:
    - rsync -a etc/icinga2/ /etc/icinga2/
    - icinga2 daemon -C

deploy:
  stage: deploy
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
  script:
    - rsync -a --delete etc/icinga2/zones.d/ icinga@master1.example.com:/etc/icinga2/zones.d/
    - ssh icinga@master1.example.com 'icinga2 daemon -C && sudo systemctl reload icinga2'

The API-based alternative deploys without SSH access to the master. You create your own configuration package and push each version as a new stage via the config management endpoints. Icinga 2 then tries to start with the new stage and keeps running with the previous configuration if that fails (Icinga Director docs). The Director uses the same mechanism. Package names starting with an underscore are reserved for internal use.

This setup also makes it possible to deploy monitoring together with the application. Application teams can keep service definitions in their own repository and contribute them through the same pipeline, so a new service and its checks ship in the same release.

Ansible: the netways.icinga collection

If your servers are already managed with Ansible, the netways.icinga collection installs the Icinga stack and generates monitoring objects from the same inventory. It was previously published as icinga.icinga; the project moved to the NETWAYS namespace, and versions 0.4.0 of both are identical apart from the name. New releases appear only under netways.icinga, so update your requirements and role references.

The collection currently contains eight roles (NETWAYS):

• repos for the Icinga package repositories
• icinga2 for masters, satellites and agents, including certificates and features
• icingadb and icingadb_redis for the Icinga DB backend
• icingaweb2 for Icinga Web and its modules, including the Director
• monitoring_plugins for the check plugins
• ifw for Icinga for Windows
• icinga_kubernetes for Icinga for Kubernetes

It also ships an inventory plugin that uses Icinga as a dynamic inventory source for Ansible.

Monitoring objects as YAML. The icinga2 role turns the variable icinga2_objects into Icinga 2 configuration files. A parser decides how each value is written, detects function calls, and lets you bypass parsing where you need raw configuration language (collection docs). The useful trick: you can define objects in an agent’s host variables and have them deployed on the master. Each host then describes its own monitoring, next to the rest of its configuration:

# host_vars/web01.example.com.yml
icinga2_objects:
  icinga-master1.example.com:   # deployed on the master, not on web01
    - name: "{{ inventory_hostname }}"
      type: Host
      file: "zones.d/main/hosts/{{ inventory_hostname }}.conf"
      imports:
        - linux-host
      address: "{{ ansible_host }}"
      vars:
        os: Linux

From the master, config sync distributes the files to satellites and agents like any other configuration.

What to keep in mind. NETWAYS points out two limits in its own write-up: a configuration error is rolled out to every affected machine just as quickly as a correct change, and a successful Ansible run does not mean the resulting Icinga configuration does what you intended. The roles take the manual work off your hands, but not the need to understand Icinga. Run icinga2 daemon -C as part of your play or pipeline, and review the rendered files at least once.

The collection includes a full-stack demo playbook for a single test host. For production layouts with HA masters, satellites and agents, the recorded webinar How to Manage Icinga with Ansible walks through an example inventory.

Puppet: the Vox Pupuli modules

For Puppet environments, the modules puppet-icinga2, puppet-icingaweb2, puppet-icingadb and puppet-icinga cover the Icinga stack. They were originally written by Icinga; the maintainers handed them to Vox Pupuli, a community group that maintains Puppet modules and tooling, and transferred existing issues and pull requests along with them (puppet-icingadb README). On the Puppet Forge they are published under the puppet namespace.

The Icinga module provides helper classes for repositories, agents, servers, Icinga DB, Icinga Web and the Director. The icinga2 module installs and configures Icinga 2 on Linux and Windows hosts and offers a defined type for each object type, such as icinga2::object::host, icinga2::object::service and icinga2::object::hostgroup, each written to a target file you choose (puppet-icinga2).

Exported resources. The classic pattern lets every node describe itself and the config master collect those descriptions from PuppetDB:

# on every monitored node
@@icinga2::object::host { $facts['networking']['fqdn']:
  address => $facts['networking']['ip'],
  import  => ['linux-host'],
  vars    => { 'os' => 'Linux' },
  target  => "/etc/icinga2/zones.d/main/${facts['networking']['fqdn']}.conf",
}

# on the config master
Icinga2::Object::Host <<| |>>

What to keep in mind. Collecting exported resources gets slow in large environments, a limitation users have discussed in the module’s issue tracker. The module therefore offers an export parameter on its object types together with the class icinga2::query_objects, which creates the objects on the targets you name without collecting exported resources. Exported resource titles must also be unique across your whole fleet; two nodes exporting the same service name break the catalog. If your Puppet data model is the source of truth but you prefer to manage templates in a GUI, the Director’s PuppetDB import is an alternative (see below).

Icinga Director: a GUI with code underneath

Many Icinga users manage their configuration in the Icinga Director, and it combines well with monitoring as code. The Icinga Director keeps objects in its own database, renders them into Icinga 2 configuration and deploys them as configuration packages through the Icinga 2 API, with the same fallback to the previous configuration if a deployment fails (Director docs).

Three parts of the Director can be managed as code:

• Configuration baskets. Templates, commands, service sets, notifications, import sources, sync rules and more can be exported as JSON snapshots and restored on any Director instance (Configuration Baskets). Keep the snapshots in Git and restore them from a pipeline with icingacli director basket restore < basket.json. Baskets deliberately do not contain single hosts or services (Director CLI).
• Import sources and sync rules. Hosts come from the systems that already know about them: databases, LDAP, REST APIs, and modules for PuppetDB, files, VMware vSphere and AWS. Sync rules map those records onto your templates, and Director jobs run import, sync and deployment on a schedule.
• REST API and CLI. Every object can be created or changed as JSON, for example icingacli director host create web01 --json < web01.json. For Ansible users, the telekom_mms.icinga_director collection wraps the Director API in modules for hosts, services, templates, commands and deployments.

A common pattern: templates and import definitions live in Git as baskets, hosts flow in from a CMDB or PuppetDB, and the GUI is used to review results and handle the occasional exception. Icinga for Windows fits in as well: its Get-IcingaCheckCommandConfig cmdlet generates a basket with the Windows check commands, ready to import (Icinga for Windows docs).

The trade-off. For the objects it manages, the Director’s database is the source of truth, not your repository. GUI changes are recorded in the Director’s activity log rather than in Git. If every change must go through a merge request, limit GUI write access to the object types you deliberately leave to the Director.

Terraform and direct API access

If your infrastructure is provisioned with Terraform, the Icinga 2 provider manages hosts, host groups, services, check commands, users and notifications through the Icinga 2 REST API. The virtual machine and its monitoring object are created in the same terraform apply, and terraform destroy removes both.

resource "icinga2_host" "web01" {
  hostname      = "web01.example.com"
  address       = azurerm_linux_virtual_machine.web01.private_ip_address
  check_command = "hostalive"
  templates     = ["linux-server"]
  vars = {
    os = "Linux"
  }
}

This works best with a clear division of labour. Terraform creates the objects that follow the infrastructure lifecycle, mainly hosts. Templates and apply rules stay in configuration files or the Director, where Icinga’s inheritance and apply rules do the composing. Keep in mind that objects created this way are runtime objects stored in Icinga’s internal API package, not files in your repository (see What belongs in code, and what doesn’t).

For a full walkthrough, the webinar Infrastructure as Code for Icinga with Terraform covers setting up and authenticating the provider, managing hosts, services, check commands and notifications as Terraform resources, and integrating Icinga into existing Terraform workflows.

Tools without a dedicated integration, such as Pulumi programs, Python scripts or workflow engines, can call the REST API directly; every request is HTTPS with JSON. Teams using Salt or Chef can follow the same patterns as with Ansible and Puppet: render configuration files on the config master, or call the API.

Common pitfalls

Most problems with monitoring as code are not tooling problems. These come up most often:

• Two tools owning the same object. If Ansible and the Director both define web01, the next deployment fails validation because the object is defined twice. Decide ownership per object type and write it down.
• A large blast radius. A change to a widely used template reaches every host that imports it, at once. Review template changes with extra care and load them on a staging instance first.
• Own files in the Director’s zone. The global zone director-global is reserved for the Director. Shared templates of your own belong in global-templates.
• Secrets in Git. Once committed, a password stays in the history. Use a vault or your CI system’s secret store from the first commit.
• Silent drift. Runtime changes made through Icinga Web or the API that never make it back into the repository.
• Version mismatch between CI and production. Validate with the same Icinga 2 version you run in production, and validate again before upgrading.
• Codifying everything at once. Code for objects that are configured once and never change costs more than it saves. Start with what changes often or exists many times.
• Exported resources at scale. In large Puppet environments, collecting exported resources slows down runs on the config master. Use icinga2::query_objects or the Director’s PuppetDB import instead.

Getting started: a step-by-step path

You do not need to pick a final architecture on day one. Each of these steps is useful on its own, and you can stop at any of them.

1. Put your current configuration into Git as it is. Copy zones.d and your own files from /etc/icinga2 into a repository, without restructuring anything yet. From this moment you have history.
2. Add validation. A CI job that runs icinga2 daemon -C on every change catches syntax errors and broken references before they reach production.
3. Decide the source of truth per object type. For example: templates, commands and apply rules in Git; hosts from your inventory tool or a Director import; downtimes and acknowledgements stay operational.
4. Automate the deployment. Copy and reload from the main branch, or push configuration packages through the API.
5. Connect the tool you already use. If servers are managed with Ansible, Puppet or Terraform, let that tool create the host objects, so a new server is monitored as soon as it exists.
6. Open it up for other teams. Merge requests against the monitoring repository, or API users with narrow permissions, let application teams add their own services without opening tickets.

Monitoring as code is less about a specific tool than about where your configuration lives and who can change it. Icinga’s configuration language, its REST API and the Director offer several ways to get there, and they can be combined. The Icinga 2 documentation and the Director documentation cover the details, the monitoring automation overview shows the related features, and the Icinga community forum is the place for questions and experience reports.