Customer Story
University Hospital Ulm
The University Hospital Ulm (UKU) in Baden-Württemberg, Germany, is known for its highly specialized healthcare, groundbreaking research, and excellent teaching. As critical infrastructure, the hospital is always aware of its special responsibility towards its patients and trusts in the high robustness and reliability of Icinga for monitoring.
The University Hospital Ulm offers diagnostic and therapeutic services at a university level and employs over 7,000 staff in its 29 clinics and 14 institutes. The various clinics are the basic units of the overall hospital, where patients are treated directly on an inpatient or outpatient basis according to medical disciplines. The institutes, on the other hand, are particularly involved in specialized diagnostics, as well as in research and teaching.
The Challenge
With Microsoft SCOM, setting up a new system was very labor-intensive manual work.
At the University Hospital Ulm, the entire hardware environment is monitored by Karsten Lanzinger’s team. It is virtualized to 90-95% using VMware and consists almost exclusively of Windows machines and a few isolated Linux environments with SUSE and Debian servers.
The requirements for the monitoring solution arise from the fact that the university hospital is critical infrastructure and must be functional 24/7, 365 days a year. Monitoring plays a key role for the entire institution. The team needs to be informed about all devices in the hospital to intervene in time before a problem arises, for example, if a server is inadvertently shut down somewhere. This can happen even though access rights are quite restrictively limited because 90% of the data in the hospital is personal data.
Too Much Manual Work
Karsten still remembers the times before Icinga. Initially, the tools used were Wotan, and later Microsoft SCOM. He recalls, “There was simply a lot of monitoring going on, and information was being spit out that we didn’t need.” What is more, setting up SCOM on a new system was very labor-intensive manual work and simply consumed a lot of time.
In 2019, when searching for a better monitoring system, the main focus, besides absolute reliability, was to reduce as much work for the individual administrator as possible. Karsten Lanzinger, who came to the University Hospital at the time, advocated for Icinga. He had been using it for over 10 years at his previous company and was highly satisfied with it.
The Solution
A Fresh Start on Green Field
From the outset, there has been close collaboration with Icinga partner NETWAYS and their consultant, Lennart Betz. Recognizing the vital role of monitoring, they also decided to secure a support agreement right from the start.
Additionally, as a KRITIS operator, meaning an institution with significant importance for the state community, the university hospital is legally required not to run any software without a support contract. The university hospital started with Icinga essentially from scratch, as they completely replaced the old solution. Initially, a basic installation was set up for testing purposes, and then various approaches were discussed with NETWAYS, leading to the development of an appropriate concept.
Initially, the database server was running on the same machine with Icinga Web. However, when the vSphere DB Module was added and the server load reached a critical point, it was decided to separate them again. Thus, the Icinga environment was gradually built up and grew slowly. In the system currently overseeing 1300 servers, not so many applications from medical technology are monitored, but rather whether the necessary services for each software are running.
The monitoring system has a certain key functionality for the entire institution, so a maintenance or service contract is mandatory.
Monitoring Infrastructure
The team at UKU has created an Icinga environment with multiple servers to allow easy management of the components and to make automation processes possible. Linux machines are managed with Foreman and Katello, which holds the repositories for the Linux servers. The Microsoft track is managed through Microsoft Software Distribution. For Icinga itself, there are 2 master servers that cover each other in case of failure, as well as the database server which hosts a PostgreSQL for the regular Icinga data and an additional MySQL database for the Icinga vSphere DB module. The Icinga Web interface runs on a separate machine.
The hospital’s monitoring team uses the Icinga Director and they have a Puppet setup with which they provision new machines. The Puppet server is the sixth server that also belongs to this environment. They have currently deliberately left out integration with a ticket system; notifications are sent via emails.
Efficient Work Routines
Today, monitoring works more efficiently with a basic framework of 15 parameters. When a new Windows system is introduced, only this basic package is initially monitored. Any adjustments and addition of further parts or services to be monitored are subsequently added manually. In addition, there is a second package with rather rudimentary checks for the MySQL server.
If there are very rare problems, we have a second package called SQL Advanced, where an additional 20 more in-depth checks are temporarily activated.
Success
Additionally, I can easily define my own groups, windows, etc., and create a dashboard that makes it easier for people to extract information.
Easier work Routines and Perfect Overview
In daily operation, the significantly improved user-friendliness becomes apparent. As an admin, one no longer needs to constantly keep watch over the systems but instead receives notifications via email. Karsten Lanzinger particularly appreciates that he can “set various warning and critical parameters for each service, allowing people to intervene at exactly the right time.”
He also finds it extremely helpful that Icinga automatically provides information and feedback, for example, when required updates for Linux servers are visible at a glance.
Another major success of the Icinga migration, according to Karsten Lanzinger, is that the monitoring is clear and runs reliably without them having to do much: “With this structure using YAML files, no one needs to keep a close eye on it anymore. I also have a configuration point outside the Director where I can handle 90% of the configuration.”
Open Structure Facilitates Mutual Representation
At the university hospital, all admins from different clinic areas have the same overview and can easily cover for each other. Karsten Lanzinger explains: “We have an open structure where every Icinga user, without major restrictions, can see all house groups and select systems relatively easily.” The Icinga services grid is also quite popular, where each admin can see what is currently happening in their own host group.
It’s a clear advantage that the support team is close to the developers, so there are sometimes directly helpful feedbacks from there.
Community Help plus Support Contract
Karsten Lanzinger is also extremely satisfied with the community, where he finds many tips for problem-solving. And if he doesn’t make progress there, there is still the proven assistance of the NETWAYS support team: “The support team clearly meets my expectations; we are just renewing our support contract. From response times to tips for finding solutions, to remotely connecting and troubleshooting the problem live.”
Future Projects
The next major topic on the hospital’s IT agenda is to better monitor the SAP environment. SAP itself intends to gradually withdraw from monitoring. Additionally, Karsten Lanzinger, as project manager, is currently setting up the first new datacenter with many new devices. Here, hardware monitoring also comes into play, as there is a requirement for all devices to be SNMP monitorable, including air conditioning units and uninterruptible power supplies (UPS).
And Icinga remains a proven key system for monitoring hospital operations in the future as well.
Tackle Your Monitoring Challenge
Learn about the basics and essentials of Icinga, and start your own Icinga by following our installation course.
