Not all monitoring work happens remotely. Some of it means packing a bag, taking the train across three countries, and sitting down with the customer’s team. This is a report from a week-long Icinga consulting engagement for a customer near Basel, covering the journey there and the work we plan to deliver on site.
Day 1: Travel to the Site
The engagement began with a long ICE ride: first to Munich, then down through a corner of Austria and across Switzerland to Basel. Being on site is important for this kind of work. Server migrations and monitoring changes are far easier to coordinate when you can talk to the customer’s team in person instead of online.
Our base for the week was the Dreiländerbrücke area, right where Germany, France, and Switzerland meet, not far from the customer’s location. With the essentials sorted out, the focus shifted to the actual scope of the engagement.
Day 2: Server Migration and the Move to Icinga DB
After picking up a visitor’s badge, we started on our work started on the customer’s main objective: consolidating their monitoring onto newer infrastructure.
The first task was to transfer everything one server was responsible for over to another server. The key word here is responsible. Before moving anything, we verified that the second server could actually take on the full workload, so the migration would not leave gaps in coverage. Checking first, migrating second.
The second objective was moving the customer off their older monitoring backend and onto Icinga DB. Icinga DB is the modern data layer for Icinga, so this brings the customer onto a current, well-supported foundation and sets them up for the features and performance that come with it. We went through everything that had to be done to make sure the switch went smoothly, including every employee’s dashboards, so they would work the way people were used to
Day 3: Rolling Out Ansible Plugins
Day three centered on automation: copying the Ansible plugins across to the second server so both machines would be provisioned consistently.
Most of it went as planned, but a few checks refused to cooperate after the copy. The cause was a dependency issue. Some checks relied on a component that existed only locally on one of the servers, so simply moving the plugin across was not enough. A check would pass on one machine and fail on the other, purely because the resource it depended on was missing on the target machine.
This is a common problem when standardizing across hosts: a plugin is only as portable as the things it assumes are already there. We identified the root cause during the day and carried the fix over into the following day’s work.
Day 4: Verification and Maintenance
With the migration and the plugin rollout completed over the first three days, day four was dedicated to validation. This is a deliberate and important part of any migration, not an afterthought: confirming that a new setup runs reliably under real operating conditions is what separates something that technically works from something a customer can trust.
The day went into reviewing the migrated services and re-checking that everything moved to the second server was reporting correctly, along with confirming that IcingaDB was collecting and displaying data as expected. We revisited the Ansible plugin rollout to make sure the earlier dependency fix held on both servers, and worked back through the checks to catch anything that might only surface once the system had been running for a while.
Handing over a monitoring setup that has been verified end to end, rather than one that simply started up, is a large part of the value in an engagement like this.
Day 5: Checkout and the Journey Back
The final day was a short one. With the work done and verified, it was time to check out of the hotel and start the long trip back the other way, through Frankfurt. Wrapping up on a stable, finished setup made the ride home an easy one.
Conclusion
Looking back, the engagement delivered what it set out to do. More than the individual tasks, the week showed why being on site matters for this kind of work.
Coordinating a migration, checking assumptions, and catching the small things is far quicker when you can talk to the customer directly and see the systems in front of you. It made for a solid first consulting experience.
