Introduction
We all know that the history data is important in monitoring. But this history data becomes obsolete over time and those records become garbage which would only fill up space. So it is important to remove obsolete history records to free up space. We call this process housekeeping. This needs to be performed periodically to cleanup the history records whenever they exceed their maximum age and become obsolete. We are introducing such housekeeping to Icinga DB to remove the history records from the database exceeding their maximum age.
Housekeeping in Icinga DB
We need to cleanup the history records in such a way that it does not affect the performance of the Icinga DB daemon. To do that we make use of go routines to run the delete statements independently without blocking other functionality of the daemon. We will also control the flow of go routines using Contexts in go programming language. In some cases, the number of records exceeding the maximum age might be huge. In these cases the time taken to delete would be significant. This in turn would degrade the performance of the Icinga DB daemon by slowing it down. To avoid this, we divide the number of records to be deleted into chunks and these chunks are deleted in multiple runs.
We need to also check the time taken to execute a single delete statement. Occasionally the time taken to delete becomes significant. In which case we will halve the size of the chunk for the next delete execution.
Note: Retention period is the number of days, the history records could be kept in the database starting from the time it was created.
