Using writable bind mounts with Icinga 2 in rootless Podman

Running Icinga 2 in a rootless Podman container is pretty straightforward, it works just the same as on Docker, so all the examples on our Docker Hub page work as expected. For example this one to generate certificates and initialize the master configuration:

podman run --rm -it \
       --network icinga \
       --hostname icinga-master \
       --volume ./data/var:/data/var \
       docker.io/icinga/icinga2:latest icinga2 node wizard

Same as with mounting some existing configuration into the container:

podman run --detach \
      --network icinga \
      --name icinga-master \
      --hostname icinga-master \
      --publish 5665:5665 \
      --volume icinga-master:/data \
      --mount=type=bind,source=/absolute/path/to/your/api-users.conf,target=/data/etc/icinga2/conf.d/api-users.conf \
      --env ICINGA_MASTER=1 \
      docker.io/icinga/icinga2:latest

But once you want to combine the two, for example to store the certificates on the host and mount them into the container, generating or renewing certificates will fail:

podman run --rm -it \
       --network icinga \
       --hostname icinga-master \
       --volume ./data/var:/data/var \
       docker.io/icinga/icinga2:latest icinga2 node wizard

 

critical/Application: Error: Function call 'mkdir' for file '/data/var/lib/icinga2/ca/' failed with error code 13, 'Permission denied'

Mounting into a rootless Podman container

The reason the previous command failed is that the icinga2 process inside the container is running with a user id of 5665, which due to Podman’s user namespace neither maps to user 5665 outside the container, nor the host user running the rootless container. The host user is actually mapped to user id 0 (i.e. root) inside the container, so when Icinga 2 tries to write to the mounted directory, it fails, because to it the mount directory seems to be owned by root.

Changing ownership of the mounted directory

On Docker, the solution would be to change ownership of the mounted directory to UID 5665 on the host. On rootless Podman, this gets a little bit more complicated because 5665 inside the container maps to a different UID/GID on the outside (for example, it translates to 105664 on my system), so you’d have to do something like this:

podman unshare chown -R 5665:5665 ./data/var

This works, but on the outside you’d still see the high UID/GID that isn’t owned by any existing user. Anything that touches them would still need root permissions.

Running Icinga as root

One option to fix that is to just have Icinga 2 run as root inside the container which means it gets full access to the mounted directories. This sounds scary, but unlike when doing the same on Docker, it’s safe on rootless Podman, because remember, root inside the container /is/ the unprivileged user on the host.

It’s not entirely that simple as just adding --user root, because Icinga 2 will still try to drop privileges down from what it thinks is actual root down to the icinga user. So if you just add that to the Podman command, it will still fail unless you change theICINGA2_GROUP environment variables and tell it explicitly to run as root:

podman run --rm -it \
       --env ICINGA2_USER=root \
       --env ICINGA2_GROUP=root \
       --user root \
       --network icinga \
       --hostname icinga-master \
       --volume ./data/var:/data/var \
       docker.io/icinga/icinga2:latest icinga2 node wizard

Mapping the host user to 5665 inside the container

However, the best alternative is to just tell Podman to translate your host user to 5665 inside the container with the --userns option:

podman run --rm -it \
       --userns=keep-id:uid=5665,gid=5665 \
       --network icinga \
       --hostname icinga-master \
       --volume ./data/var:/data/var \
       docker.io/icinga/icinga2:latest icinga2 node wizard

This maps the host user’s id to 5665 inside the container. Unlike before, this now gives the icinga2 container’s default user access to the mounted files. No need to change the user Icinga 2 runs as no environment variables need to be set. And the process stays unprivileged even inside the container’s namespace.

That way, when starting the container as your normal user, you can edit the configuration quickly without copying back and forth and without sudoedit using your favorite editor, which is great for development setups.

Starting an existing configuration in rootless Podman

With this in mind, migrating your package-based setup on a host into a rootless Podman container could be as easy as mounting the existing /etc/icinga2 and /var/lib/icinga2 directories into the container and starting the container as the right user.

Note that this is not a recommendation to migrate your Icinga setup in this way, and there may be additional complications, for example when using SELinux.

First the host user will need entries in /etc/subuid and /etc/subgid set up, which isn’t always the case for system users.

Run as root:

usermod --add-subuids 200000-265535 --add-subgids 200000-265535 icinga

Note that on Debian the icinga user will likely instead be called nagios.

Podman compose

One option to automatically start such a container is with Podman compose. Put this (for example) in /opt/icinga2/docker-compose.yml:

x-podman:
  in_pod: false

networks:
  default:
    name: icinga

services:
  icinga2:
    image: docker.io/icinga/icinga2:latest
    userns_mode: "keep-id:uid=5665,gid=5665"
    restart: always
    logging:
      driver: "json-file"
    networks:
      - default
    ports:
      - 5665:5665
    volumes:
      - /etc/icinga2:/data/etc/icinga2
      - /var/lib/icinga2:/data/var/lib/icinga2
      - /var/log/icinga2:/data/var/log/icinga2

Note the following:

• The userns_mode requires disabling the in_pod setting. Otherwise the container will not start.
• The restart: always line allows your container to be restarted by the podman-restart service.
• The in_pod: false setting is only required when using podman-compose, but not docker-compose as a backend.

Enable the user services for that user to run even if no user is logged in:

loginctl enable-linger icinga

Then switch to the Icinga user and enable the service and start the container:

sudo -u icinga env \
     XDG_RUNTIME_DIR=/run/user/$(id -u icinga) \
     DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u icinga)/bus \
     bash
systemctl --user enable --now podman-restart.service
cd /opt/icinga2 && podman compose up -d

Now the container will restart on every boot.

Systemd generator

Modern Podman has its own systemd generator which doesn’t rely on compose, that can be used to start containers by placing unit files in /etc/containers/systemd/users/${UID}/ or $XDG_CONFIG_HOME/containers/systemd/:

For example:
/var/lib/icinga2/.config/containers/systemd/icinga.network:

[Network]
NetworkName=icinga

 

/var/lib/icinga2/.config/containers/systemd/icinga2.container:

[Unit]
Description=Icinga 2
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/icinga/icinga2:latest
ContainerName=icinga2
HostName=icinga-master

UserNS=keep-id:uid=5665,gid=5665

Network=icinga.network
PublishPort=5665:5665

Volume=/etc/icinga2:/data/etc/icinga2
Volume=/var/lib/icinga2:/data/var/lib/icinga2
Volume=/var/log/icinga2:/data/var/log/icinga2

[Service]
Restart=always
TimeoutStopSec=30

[Install]
WantedBy=default.target

• With that file the resulting service will also be called icinga2. And if you still have the packages installed this will conflict. In that case you’ll need to choose a different name, but at least ensure that the existing icinga2.service is stopped and disabled so they won’t run at the same time.
• This doesn’t need the logging: driver: json-file bit from the compose example, because with this method, logs can be conveniently accessed with journalctl.

Then to enable the services again switch to the Icinga user and enable the generated units:

sudo -u icinga env \
     XDG_RUNTIME_DIR=/run/user/$(id -u icinga) \
     DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u icinga)/bus \
     bash
systemctl --user daemon-reload
systemctl --user start icinga2

• Unlike before, we’re not manually enabling the service, since it is WantedBy=default.target and the daemon-reload will enable it to be started on the next restart. Running systemctl --user enable icinga2 would fail in this case.
• The initial start will take a while to initialize the container because it will likely need to pull the image first.
• Unlike the previous option, this is now a regular systemd service that can be started or stopped via systemctl.

Final thoughts

The error in the beginning of this blog post was due to the icinga/icinga2 container running Icinga 2 with a user id of 5665, while rootless Podman maps the host user to 0 (root). I’ve shown you several solutions (some better than others) to this problem and how to effectively run this container with rootless Podman.

Please don’t take running the Podman container with the whole host configuration as a general recommendation over installing the packages, but it might be a helpful option in some cases, for example in a development or testing environment where you want to quickly switch between versions.