Debugging Filters and Apply Rules using the Script Debugger

Have you ever been in a situation where something in your Icinga configuration did not work as expected and you ended up doing small changes and reloading Icinga over and over again? This can be especially tricky with apply rules and filters if they don’t match the objects you hope for. This post will show you how you can use the Icinga Script Debugger in this situation to get an interactive console in the context where the apply rule or filter is evaluated.

Let’s consider the following config snippet as our base. It just creates a few hosts which are members of the host groups webserver, dbserver, or both:

object HostGroup "webserver" {}
object HostGroup "dbserver" {}

object Host "bart" {
  check_command = "dummy"
  groups = ["webserver"]
}

object Host "lisa" {
  check_command = "dummy"
  groups = ["webserver", "dbserver"]
}

object Host "maggie" {
  check_command = "dummy"
  groups = ["dbserver"]
}

API Permission Filters

Now suppose we want to create an API user that is restricted to querying only web-related hosts. The first attempt could look like this:

object ApiUser "top" {
  password = "secret"
  permissions = [{
    permission = "objects/query/Host"
    filter = {{ match("web*", host.groups) }}
  }]
}

However, when trying to use the new API user, we notice that it only works partially. It only returns the host bart, even though lisa is also a member of the webserver group.

$ curl -skSu top:secret -H 'Accept: application/json' \
       'https://localhost:5665/v1/objects/hosts?pretty=1&attrs=name'
{
    "results": [
        {
            "attrs": {
                "name": "bart"
            },
            "joins": {},
            "meta": {},
            "name": "bart",
            "type": "Host"
        }
    ]
}

To debug this, we can now add a debugger breakpoint to the filter. It can also be guarded by an if condition, for example we are only interested in what happens when the filter is evaluated for the host lisa:

    filter = {{
      if (host.name == "lisa") {
        debugger
      }
      match("web*", host.groups)
    }}

The script debugger is disabled by default. In order to use it, we have to run Icinga 2 in the foreground using the additional -X flag (icinga2 daemon -X). When repeating the API request, it won’t complete immediately but instead, we are greeted by the script debugger in the terminal running Icinga:

Breakpoint encountered.
[...]
You can inspect expressions (such as variables) by entering them at the prompt.
To leave the debugger and continue the program use "$continue".
For further commands see "$help".
<1> =>

This interactive console can now be used to evaluate different expressions in the same context as the filter rule is evaluated in, for example:

<1> => host.name
"lisa"
<2> => host.groups
[ "webserver", "dbserver" ]
<3> => match("web*", host.groups)
false

Hopefully, at some point we discover our problem. In this example, the match function, when applied to an array, checks that each element matches the given pattern, but want to check if any element matches. Luckily, this behavior can be changed by passing the additional MatchAny flag.

<4> => match("web*", ["webserver", "dbserver"])
false
<5> => match("web*", ["webserver"])
true
<6> => match("web*", ["webserver", "webapp"])
true
<7> => match("web*", host.groups, MatchAny)
true

Now that we know the problem, we can update the filter, remove the debugger breakpoint, and verify that all the expected hosts are returned.

Apply Rules

The script debugger can also be used for apply rules. Imagine the same mistake in one of these, here we can also add a breakpoint like this:

apply Service "http" {
  check_command = "http"
  assign where {
    if (host.name == "lisa") {
      debugger
    }
    match("web*", host.groups)
  }
}

Now, validating the configuration with the additional -X switch (icinga2 daemon -C -X) will invoke the debugger.

API Request Filters

Last but not least, when running Icinga with the icinga2 daemon -X command, you can also add breakpoints to filters passed in API requests like this:

curl -skSu root:icinga -H 'Accept: application/json' \
     --get 'https://localhost:5665/v1/objects/hosts' \
     --data-urlencode pretty=1 \
     --data-urlencode filter='if (host.name == "lisa") { debugger }; match("web*", host.groups)'