Clusters get retired. A test cluster is torn down, a workload moves to a new environment, a provider contract ends. Icinga for Kubernetes does not know about any of that.
Since v0.3.0, several clusters can write into one shared database, and Icinga for Kubernetes Web shows them side by side. But stopping or uninstalling the daemon for a cluster leaves all of its data in place. The cluster keeps showing up in the web interface, with its last known pods, nodes, and other resources, long after it stopped existing. Until now, the only way to get rid of it was to delete rows by hand.
The main branch now has a proper way to do this: --remove-cluster.
Removing a cluster in three steps
--remove-cluster deletes a cluster and every piece of data that belongs to it from the database, then exits. It does not start the daemon.
- Stop the instance that monitors the cluster. A running instance keeps writing data. For a package installation, stop and remove the service instance. For Helm, uninstall the release.
- Find the cluster UUID. Query the database:
SELECT HEX(uuid) AS uuid, name FROM cluster;
If the cluster still exists, its UUID is the UID of its kube-system namespace:
kubectl get namespace kube-system -o jsonpath='{.metadata.uid}' -
Run the removal with the same database configuration the instance used:
icinga-kubernetes --config /etc/icinga-kubernetes/config.yml --remove-cluster <uuid>
To remove several clusters in one run, repeat the flag or pass a comma-separated list. If your instance sets database options through ICINGA_FOR_KUBERNETES_DATABASE_* environment variables, set the same ones for this command.
The command logs “Removed cluster” with the UUID for each cluster and exits with status 0 when it is done. The removal cannot be undone, so double-check the UUID.
How it works
Three design decisions shaped the feature.
It works without the cluster. The most common reason to remove a cluster is that it is gone. So the removal must not depend on reaching it. We changed the startup order: the database is now configured before the Kubernetes client. The removal path only opens the database connection and never touches the Kubernetes API, so it needs no kubeconfig. Invalid UUIDs are rejected while the flags are parsed, before any connection is made.
Deletes follow the schema, not hand-written SQL. Every table in the Icinga for Kubernetes schema that belongs to a cluster carries a cluster_uuid foreign key. The cluster type now declares these relations explicitly: configs, instances, events, Prometheus metrics, and every resource type from config maps to stateful sets. The removal is a cascading delete of the cluster entry that walks those relations. Nested data such as a pod’s conditions is removed through the same mechanism that already keeps the database in sync during normal operation.
Reruns pick up where a failed run stopped. If no cluster entry exists for a UUID, the command logs a warning instead of failing, and still deletes any remaining data for that UUID. If a removal is interrupted, for example by a lost database connection, you can run the same command again. The rerun deletes the cluster’s remaining resources and their related data.
There is one edge to this. If the interrupted run had already deleted a resource such as a pod, the rerun cannot find that pod anymore, and data that hung only off it, such as its conditions, stays behind.
What it does not remove yet
Labels, annotations and selectors are currently not deleted with the cluster. They live in shared tables that carry no cluster UUID and are linked to resources through join tables. Fixing this is tracked separately in #261 so it can get its own review.
Until that lands, expect some unreferenced label and annotation rows to remain after a removal.
Try it and tell us
--remove-cluster is merged into the main branch (#244) and will be part of the next release. The full procedure is in the new Commands chapter of the documentation.
This is part of the work toward a stable v1.0, and multi-cluster setups are exactly where we need feedback. If you run several clusters against one database, we would like to hear:
- whether the removal finished cleanly on your setup
- whether anything was left behind that you did not expect
- whether the workflow of stopping the instance, finding the UUID and running the command fits how you operate
Open an issue on GitHub or post in the community forum.
