Rule based monitoring with Icinga apply rules

Apply rules simplifies creation of objects like Service, Notification, Dependency, ScheduleDowntime which require object relation. In this blog post we will understand rule based monitoring with Icinga apply rules with examples.

Requirement: Icinga 2 and Icinga Web 2 installed. Knowledge of monitoring basics. To see the path to the configuration files check this link.

Two examples have been illustrated here to demonstrate rule based monitoring with Icinga apply rules.

Example-1: A service object “ping” need to be created for each host objects with host address. Without using Icinga apply rules, this is done by creating “ping” service objects one by one for each host objects with address attribute or by matching host_name attribute with some pattern. This can become chaotic when there is an huge number of hosts. By using Icinga apply rules we need to create one service apply rule “ping” for all hosts containing address attribute. Hence, for creating objects that require object relations it is better to use rule based monitoring. Apply rules bring order to creation of objects with relations, which could otherwise be chaotic.

Create the “generic-host” and “generic-service” templates in templates.conf.

template Host "generic-host" {
  max_check_attempts = 3
  check_interval = 1m
  retry_interval = 30s

  check_command = "hostalive"
}

template Service "generic-service" {
  max_check_attempts = 5
  check_interval = 1m
  retry_interval = 30s
}

Create “dummy1”, “dummy2” and “dummy3” host objects.

object Host "dummy1" {
  import "generic-host"
  address = "127.0.0.1"
  vars.os = "Linux"
  vars.dummy_text = "I am dummy 1."
}

object Host "dummy2" {
  import "generic-host"
  address = "127.0.0.1"
  address6 = "::1"
  vars.dummy_text = "I am dummy 2."
}

object Host "dummy3" {
  import "generic-host"
  address6 = "::1"
  vars.dummy_text = "I am dummy 3."
}

Add a “ping” service apply rule which is assigned to hosts containing address attribute.

apply Service "ping" {
  import "generic-service"

  check_command = "ping4"

  assign where host.address
}

Restart Icinga 2.

systemctl restart icinga2

This “ping” service applies only to “dummy1” and “dummy2” host objects which contains address attribute and ignores “dummy3” host object.

Example-2: In this example, we create “mail-testuser” notification apply rule, which is applied to all the services of the hosts for which contains notification custom variable with nested key mail.

Add templates “generic-user” and “mail-service-notification” to templates.conf.

template User "generic-user" {

}

/**
 * Provides default settings for service notifications.
 * By convention all service notifications should import
 * this template.
 */
template Notification "mail-service-notification" {
  command = "mail-service-notification"

  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    notification_logtosyslog = false
  }

  period = "24x7"
}

Modify the “dummy3” host object as below in hosts.conf.

object Host "dummy3" {
  import "generic-host"
  address6 = "::1"
  vars.notification["mail"] = {
    /* The UserGroup `testusers` is defined in `users.conf`. */
    groups = [ "testusers" ]
  }
  vars.dummy_text = "I am dummy 3."
}

Create “ping6” service apply rule in services.conf for host objects containing address6 attribute. In this case dummy3.

apply Service "ping6" {
  import "generic-service"

  check_command = "ping6"

  assign where host.address6
}

Add user objects “testuser1” and “testuser2”; and user group object “testusers” to users.conf.

object User "testuser1" {
  import "generic-user"

  display_name = "Test User 1"
  groups = [ "testusers" ]

  email = "test1@localhost"
}

object User "testuser2" {
  import "generic-user"

  display_name = "Test User 2"
  groups = [ "testusers" ]

  email = "test2@localhost"
}

object UserGroup "testusers" {
  display_name = "Test User Group"
}

Create “mail-testuser” notification apply rule to service in notifications.conf.

apply Notification "mail-testuser" to Service {
  import "mail-service-notification"
  user_groups = ["testusers"]
  users = ["testuser1", "testuser2"]

  interval = 2h

  assign where host.vars.notification.mail
}

Restart Icinga 2.

systemctl restart icinga2

This notification applies to all services whose host objects contain notification custom variable with nested key mail. Here, this rule will only be applied to “ping6” service object of host “dummy3”.

In the same way Icinga apply rules can be used for other objects which require object relation. There is more to Icinga apply rules and you can learn about it here.