Playing GIFs in the Icinga DB Web Service Grid: What It Taught Me About Icinga 2, Icinga DB and Redis

by Jan Schuppik | Oct 21, 2026

What happens when you treat the service grid like a screen.

Every now and then a side project starts as one thing and quietly turns into something much sillier. Mine started as a way to get more comfortable with the Icinga 2 configuration. It ended with a question nobody asked: can the service grid in Icinga Web play GIFs?

Spoiler: kind of. And on the way, I learned a lot about how Icinga 2, Icinga DB and Icinga Web work together.

Where It Started

I was playing around with devdocker, the Docker-based environment we use for development. I wanted more hosts and services in it. Partly to have more to look at, partly as an excuse to finally learn the Icinga 2 configuration language. So I added dummy hosts with services that switch their state randomly.

Once I had a wall of services changing color on their own, the resemblance was hard to unsee. The service grid of Icinga DB Web shows services as small colored tiles. Each host is a row, each service a column. A tile is a pixel. And a grid of pixels that changes over time is a GIF.

I briefly tried the Cube module as well, but it renders way slower. The service grid was the clear winner. To keep my changes separate from Icinga DB Web, I copied the service grid into a small module. From there, I adjusted it step by step. Everything else, from the models to the filters and the pivot table, is still Icinga DB Web. The module is a playground, not a product, so I haven’t published it. But the snippets below show the interesting parts.

Building a Service Grid Display in Icinga 2

I started small, with a display of 32×32 services: 32 hosts as rows and 32 services as columns. That’s 1,024 pixels. A white pixel is OK, a black pixel is UNKNOWN. In the default colors of the service grid, that’s green and purple, but more on that later. Thanks to an apply for rule, the configuration is surprisingly small. Growing the display later only meant changing two constants:

const PixelRows = 32
const PixelCols = 32

for (row in range(PixelRows)) {
  var pixelRow = pixelPad(row)

  object Host pixelHostName(row) use (pixelRow) {
    check_command = "dummy"
    vars.pixel_row = pixelRow
  }
}

apply Service for (name => pixelCol in pixelCols) {
  check_command = "dummy"
  check_interval = 1d     // Never due on its own, the clock triggers it
  max_check_attempts = 1  // Every result is a hard state right away

  vars.pixel_index = number(host.vars.pixel_row) * PixelCols + number(pixelCol)

  vars.dummy_state = {{
    if (frames[pixelClock.frame].substr(service.vars.pixel_index, 1) == "1") {
      return 0 // OK, white
    }
    return 3 // UNKNOWN, black
  }}

  assign where host.vars.pixel_row
}

The object needs use (pixelRow), because Icinga 2 evaluates its body later, outside of the loop. Without it, the body can’t see the loop’s variables.

I zero-padded the rows and columns (001r, 002r, …), because the service grid sorts them as strings. Without the padding, row 10 would end up between row 1 and row 2. (The names changed along the way, so the first videos below still show pixel-00 and c00.)

The dummy_state is a lambda, so Icinga 2 evaluates it on every check of the pixel. It looks up the current frame and picks the pixel’s character from it.

Getting the GIF into the Configuration

The frames come from the animated Icinga logo. A short Python script with Pillow does the conversion. It scales every frame down to the size of the display and converts it to grayscale. A brightness threshold then turns every pixel into 1 or 0. Finally, the script joins all rows of a frame into one long string. The whole animation becomes a single Icinga 2 array with one string per frame:

globals.frames = ["0000000…", "0000000…", …]

75 frames, 1,024 characters each at 32×32. Not elegant, but Icinga 2 reads it without complaint.

Step 1: One Service to Rule Them All

The pixels never check on their own (check_interval = 1d). Instead, a single service acts as the clock of the display. It runs every 20ms. Whenever it advances to the next frame, it triggers an immediate check for every pixel:

object Service "clock" {
  host_name = "pixel-clock"
  check_command = "dummy"
  check_interval = 20ms

  vars.dummy_text = {{
    var now = get_time()
    var index = (pixelClock.frame + 1) % len(frames)
    pixelClock.frame = index

    for (pixel in pixelClock.pixels) {
      pixel.next_check = now // check this pixel right away
    }

    return "frame " + string(index)
  }}
}

I simplified the snippet: the real clock only moves on once all pixels have their new state. In the runs in between, it just checks whether they’re done yet.

Setting next_check from within a lambda is all it takes to trigger a check. You wouldn’t do that in production. But it shows nicely how much you can do with the configuration language alone.

Step 2: Polling the Service Grid Faster Than Icinga Web Wants To

With the clock in place, Icinga 2 was playing the animation. Now the service grid had to show it. But the autorefresh of Icinga Web exists for monitoring, not for animation. It refreshes at most once per second. That’s perfectly reasonable for watching infrastructure and completely useless for playing a GIF.

My first hack was to ignore it: a small JavaScript function that fetches the states on its own timer. That worked, but it hit its ceiling almost immediately. Depending on the browser, I got about 3 to 4 requests per second before things started falling apart. Also, the requests had nothing to do with the clock’s timing. They often caught a frame halfway through, and the service grid showed a mix of two frames.

That left two problems. The browser had to get the states faster, and it had to know which states belong together.

Step 3: Only Touch What Changed

Before solving those, the clock itself had to get faster, because growing the display made it slow down quickly. Triggering every pixel for every frame doesn’t scale. The number of pixels grows with the square of the display size. 32×32 is 1,024 services, 64×64 already 4,096, and 128×128 is 16,384.

Most pixels don’t change from one frame to the next, though. So another script compares every frame with the previous one. It lists only the pixels that flip from 1 to 0 or back. Instead of looping over all pixels, the clock now only triggers those.

That cut the workload down a lot, but it introduced a new quirk: the time per frame is no longer constant. A frame where almost nothing changes is cheap, a frame where half the logo flips is expensive. Same resolution, very different cost, depending entirely on what’s happening in the animation at that moment.

Step 4: Pushing Changes to the Service Grid

To get rid of the request limit, I turned the direction around. Instead of the browser asking for the service grid, the server now pushes changes via server-sent events. The browser opens one long-lived connection, and the module streams only the services which changed since the last event. The module reads the states straight from Icinga DB’s Redis. It queries the SQL database only once, to find out which services the grid shows.

Much faster, but the service grid still mixed frames. The stream had no idea which states belong together.

Teaching the Stream What a Frame Is

The solution: let the clock and the pixels tell the stream what they are doing. They use the one thing every check result has anyway, its output.

The clock waits until all triggered pixels have their new state. Then it announces frame <id> seq <seq> complete for exactly one run. <seq> counts all frames ever played and never repeats.
Every pixel ends its output with valid <from>-<until>, the range of frames its current state is correct for. The clock knows when a pixel changes next, so it can tell.

The stream polls only the clock’s state, every 5ms. Once the clock announces a complete frame, the stream reads all pixel states. It checks that every single one is valid for that frame. If one isn’t, it reads them again.

That last check turned out to be necessary. Icinga 2 knows a pixel’s new state immediately, but Icinga DB writes it to Redis shortly afterwards. So the clock waits one more run before it announces the frame as complete. This “settling” gives Redis enough time to catch up in almost all cases. If it doesn’t, the stream drops the frame rather than showing a broken one.

All of this comes at a price: triggering, waiting, settling and announcing take at least one clock run each. So a frame takes at least 80ms, which caps the display at about 12.5 frames per second. And that’s the best case, where Icinga 2 checks every triggered pixel within a single run.

Step 5: Buffering Service Grid Frames in the Browser

Whole frames at last, but they still arrive irregularly, as every frame takes a different amount of time. If the browser shows them as they arrive, the animation stutters.

So the browser buffers frames before playing them. It collects 50 frames, then plays one frame per interval, based on the average time between arriving frames. If the buffer gets too full or too empty, the browser stretches or shortens the interval by up to 10%. That way, the buffer neither runs dry nor keeps growing.

That works as long as Icinga 2 produces frames about as fast as the browser plays them. At 128×128, it doesn’t. If the source is slower than the playback, no buffer is large enough. It always runs dry eventually.

The final step embraces that: the browser keeps the complete state of every frame it receives. Once it has seen a full loop, it plays the stored frames at a fixed rate, forever. Frames that keep arriving replace the stored ones, so changes still show up, just not in real time. While the browser collects the first loop, it dims the service grid.

For the final touch, a few lines of CSS show the pixels in black and white instead of green and purple. That’s the same version you saw at the very top of this post, but in black in white.

Is this still Icinga playing a GIF, or just a browser replaying what Icinga showed it once? I’ll leave that question to you. Every frame did go through Icinga 2, Icinga DB and Redis at least once.

Things I Learned on the Way

Beyond the GIF itself, I stumbled upon a few things worth knowing when you work with Icinga 2 and Icinga Web:

The globals namespace is read-only at runtime. So you have to define everything the clock changes later as a dictionary at config load. That includes its current frame and the pixels it waits for. You can change the dictionary’s contents, but not replace the variable itself.
get_service() is expensive. Looking up thousands of services for every frame is not an option. So the clock looks them up once and keeps them in an array.
Icinga 2 doesn’t splay short check intervals. Usually, Icinga 2 spreads checks out randomly to avoid load peaks. It skips that for intervals of one second or less. So the clock really runs every 20ms from the start.
Icinga 2 and Redis are not always in sync. Icinga 2 has a new state immediately, Icinga DB writes it to Redis shortly afterwards. Anything that reads Redis needs to account for that.
Autorefresh and streams don’t mix. An autorefresh re-renders the service grid, which restarts the stream. So the module turns the autorefresh off while a stream is active.

Was It Worth It?

Is this useful? Not in the slightest. Nobody should monitor their infrastructure with a GIF, and nobody should play GIFs with their monitoring.

But as a way to learn, it was great. To make the logo move, I had to understand a lot. How the Icinga 2 configuration language handles arrays, strings and lambdas. How check results travel from Icinga 2 through Icinga DB into Icinga Web. How the service grid renders its tiles. And where each part of that chain hits its limits. Pushing a system far beyond its purpose is a surprisingly effective way to learn how it actually works.

Want to try the service grid yourself? Get started with Icinga and see what your own services look like as pixels. And if you have an idea for an equally pointless project with Icinga, let us know. We might need a new side quest.