Today, we are going to look at a few things related to handling secrets. While Icinga 2 has no dedicated mechanisms for secret handling, there are a few tricks you can do with standard features. This is not meant as a step-by-step tutorial, but rather as an inspiration where you can adopt the ideas that make sense in your setup.
Hiding Secret Values in Lambdas
If you add vars.example_password = "N0t4n4cTuAlP4S5w0rd" to a service definition, the value is synced to the database and may be shown in the custom variables section in Icinga Web, unless they are limited by the icingadb/denylist/variables or icingadb/protect/variables restrictions.
However, there is a neat little trick you can do: Just write vars.example_password = {{ "N0t4n4cTuAlP4S5w0rd" }} instead. {{ ... }} is the shorthand syntax for an anonymous lambda function that takes no parameters (the long form would be () => { ... }). Usually, lambdas are used if you want to have a short piece of code generating a dynamic value. However, nothing is stopping you from just returning a constant string. If you use "$example_password$" in a macro string, functions will be evaluated transparently, so it will work the same in your CheckCommands, but stop the actual value from being synced to Icinga DB.
Avoid Passing Secrets on the Command Line
Even if you do that, with many CheckCommands, the password will still appear in another place in Icinga Web: the executed check command line. This can also be restricted using the icingadb/object/show-source permission in Icinga Web, but passing sensitive information on the command line is something that one should avoid in general, given that it may be visible to all local system users.
There are two common ways to avoid this. However the check plugins need to explicitly support one of the following methods, so you have to verify if that is possible for the specific plugins you are using.
The first option is using environment variables instead as these are not visible to other users. To do this, the CheckCommand type has the attribute env that is a dictionary that will be used to populate the environment variables for the check plugin.
The other option is to use files to pass the secrets, thereby relying on standard file ACLs to protect them. However, note that these files have to exist locally on the machine executing the check, so you have to use your own configuration management to create them there.
Examples of check plugins supporting this mode of operation are those from the Monitoring Plugins project with their --extra-opts argument that allows reading additional options from a file. As an example, this can be used to specify the basic auth credentials for check_curl:
object Service "secrets-from-file" {
host_name = "demo.example.com"
check_command = "curl"
vars.curl_url = "/basic-auth-demo/hello.txt"
vars.curl_expect_content_string = "Hello"
vars.curl_extra_opts = "localhost_basic_auth@/etc/icinga2/secret_opts.ini"
}
The contents of /etc/icinga2/secret_opts.ini look as follows. localhost_basic_auth is a reference to a section within the INI file, allowing the same file to be shared across multiple checks.
[localhost_basic_auth] authorization=icinga-check:St1LLn0r3aLpa$$
Store Secrets Locally
Having to create the secret files on the node executing the check comes with a trade-off between convenience and security. If your Icinga master does not need a secret for anything, you can also take this as an opportunity to not store it there at all. So for example if a backup of your master was exposed somehow, that would leak fewer secrets.
You can also do something similar with secrets that are passed to check plugins in environment variables or command line arguments by defining them in a local config file that is not part of the synced configuration. In order to do this, create a file like /etc/icinga2/secrets.conf with contents like the following:
const LocalSecrets = {
"example_api_token" = {{ "A5iFT1sW45aR3aIT0k3nTh1sT1m3" }}
}
After making that constant available in your configuration by adding include "secrets.conf" to /etc/icinga2/icinga2.conf, this can then be used in a service definition like this:
object Service "local-secret" {
host_name = "demo.example.com"
check_command = "example-api"
vars.example_api_token = LocalSecrets.example_api_token
}
Encrypting with systemd Credentials
It is also possible to take locally stored secrets a step further and make use of a neat feature systemd provides: service credentials. If you have never heard of that: it allows you to encrypt your secrets using an on-disk secret key and/or (at your choice) a TPM-backed key. When configured properly in the service unit, when systemd starts the service, it will decrypt the credentials and provide them to the service.
To encrypt the file secrets.conf (from the current directory, the contents are the same as in the one above) and place the output in /etc/icinga2/secrets.conf.cred, the following command can be used. By default, both an on-disk and TPM-backed (if available) key will be used, but the exact behavior can be changed by giving the additional --with-key flag, see the corresponding manual page for details. As only symmetric cryptography is used, the secrets have to be encrypted on the machine that will use them later.
systemd-creds encrypt --name=secrets.conf secrets.conf /etc/icinga2/secrets.conf.cred
Afterwards, the plain text file is no longer needed and can be erased.
Next, the icinga2.service unit needs to be extended to load the encrypted credentials. This can be achieved easily by placing a drop-in file in /etc/systemd/system/icinga2.service.d/credentials.conf, followed by executing systemctl daemon-reload to instruct systemd to read the updated configuration:
[Service] # The secrets are only readable to the user the service is started as, so if it # is started as root, Icinga 2 cannot read the secrets anymore after dropping # the permissions itself. Thus, systemd must be configured to start Icinga 2 as # the service user. # # Note: on Debian/Ubuntu, you have to use "nagios" here as that is used for # historic reasons. User=icinga LoadCredentialEncrypted=secrets.conf:/etc/icinga2/secrets.conf.cred
From now on, when Icinga 2 is started, systemd will prepare a credentials directory for the service, decrypt the file, and place the decrypted contents in a file secrets.conf in that directory. The path of that directory will be provided to the service in the CREDENTIALS_DIRECTORY environment variable. It can then be loaded from /etc/icinga2/icinga2.conf like this:
include getenv("CREDENTIALS_DIRECTORY") + "/secrets.conf"
However, given that whenever you run the icinga2 command outside of the systemd service, no credentials are loaded and the configuration will fail to load and validate when doing a simpleicinga2 daemon --validate. With systemd-run, it is possible to run commands in a transient unit with the appropriate additional properties that instruct systemd to load the credentials there as well. The command is a bit clunky, so you might want to create your own shell alias for it.
systemd-run --pipe --wait \ -p User=icinga \ -p LoadCredentialEncrypted=secrets.conf:/etc/icinga2/secrets.conf.cred \ icinga2 daemon -C
Alternatively, you can only load the credentials file conditionally if started as a systemd service by performing the include inside an if block. In that case, you need to be aware that the evaluated configuration will differ slightly based on whether it is loaded with or without the credentials, and you could have config errors that only appear in one mode or the other.
if (getenv("CREDENTIALS_DIRECTORY")) {
include getenv("CREDENTIALS_DIRECTORY") + "/secrets.conf"
} else {
log(LogWarning, "config", "systemd CREDENTIALS_DIRECTORY not set, not loading secrets")
const LocalSecrets = null
}
Bonus: Check if You Even Need Secrets At All
As a final remark, I would like to point out an example where passwords are used frequently, even though they are not really necessary: accessing services on the same machine, like databases.
Take PostgreSQL as an example: it supports peer authentication for local connections made using a Unix domain socket. This means that it will simply ask the operating system which system user initiated the connection and uses this to authenticate the connection without any password.
MariaDB and MySQL also support the same mechanism, for them, the corresponding authentication plugins are called unix_socket and socket_auth, respectively.
Therefore, it might also be worth a look to see if instead of handling a secret better, you can just get rid of it instead without having to sacrifice security.
Proudly written without AI.
