Advisory for latest security updates on RHEL 7

by | Jun 20, 2017

Community members told us today that Icinga 2 stopped working with the most recent RedHat Enterprise Linux 7 Kernel update 3.10.0-514.21.2. This update includes a security patch for the stack guard vulnerability.
Update 2017-06-20 19:55 Europe/Berlin: CentOS 7 is currently rolling the kernel update and affected too. Upstream bug report has been created.
Update 2017-06-21 11:30 Europe/Berlin: RedHat’s kernel team is investigating on this possible regression. We are in touch with them. Meanwhile apply the quickfix below.
Update 2017-06-21 19:20 Europe/Berlin: https://bugzilla.redhat.com/show_bug.cgi?id=1463241 is now public.
Update 2017-06-22 14:45 Europe/Berlin: We are investigating on the Icinga 2 side how to better handle rlimits. v2.7 is postponed until the Kernel issue is fully resolved. Other distributions might be still affected, there’s ongoing investigation on the oss-sec mailing lists.
Update 2017-06-23 10:45 Europe/Berlin: RedHat provided us with a fixed test package and our tests went fine. Please open a support case at RedHat to receive “accelerated fixes” or to get a test binary package. You also raise awareness by doing so, this should help getting an official release sooner.
Update 2017-06-29 12:15 Europe/Berlin: RedHat created knowledge base articles #3092281 and #3098341 with a solution being in progress. Fixed kernels seem to be deployed, CentOS should catch up soon. We’ve also seen that Debian updated their Kernel patches fixing a regression. Probably more things were affected by the stack guard patches. Since some of you asked: Icinga 2 v2.7 won’t help here (you still need the workaround below!), so we’ll wait until users deployed fixed Kernel updates and then start a 2.7 release cycle again.
Update 2017-07-24 14:00 Europe/Berlin: Updated Kernel versions (RHEL, CentOS) have been released. Thanks to everyone involved!
 

Analysis

We’ve analysed the issue and reproduced the issue on RHEL7. Debian Jessie and Stretch also released a security update for CVE-2017-1000364 but Icinga 2 does not crash.
Icinga 2 reduces the default stack size from 8 MB to 256 KB for spawned threads. This is to avoid huge memory reservation and troubles with swap overcommit being disabled.
We consider this behaviour a bug inside the RHEL Kernel and have therefore created an upstream issue (hidden by default).
 

Quickfix

If you are planning to update your RHEL/CentOS system, you can apply this workaround: Add “–no-stack-rlimit” to ExecStart in your systemd configuration file. In order to change this permanently copy the existing systemd service file and then apply the changes.

cp /lib/systemd/system/icinga2.service /etc/systemd/system/icinga2.service
vim /etc/systemd/system/icinga2.service
ExecStart=/usr/sbin/icinga2 daemon -d -e ${ICINGA2_ERROR_LOG} --no-stack-rlimit
systemctl daemon-reload

Edit 2017-06-20 17:55 Europe/Berlin: You’ll also need to patch the /usr/lib/icinga2/prepare-dirs script to use –no-stack-rlimit parameter.

-ICINGA2_USER=`$DAEMON variable get --current RunAsUser`
+ICINGA2_USER=`$DAEMON variable get --current RunAsUser --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsUser variable. Error '$ICINGA2_USER'. Exiting."
         exit 6
 fi
-ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup`
+ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsGroup variable. Error '$ICINGA2_GROUP'. Exiting."
         exit 6

 
A simpler workaround for the icinga2 shell wrapper in /usr/sbin/icinga2 is mentioned in #5367:

vim /usr/sbin/icinga2
#exec $ICINGA2_BIN "$@"
exec $ICINGA2_BIN --no-stack-rlimit "$@"

 
The long term solution will be applied in Icinga 2 v2.7 by making this setting a Systemd/SysVInit configuration variable.
RHEL 6 is not affected in our ongoing tests.
If you are running RHEL 6, edit the /etc/init.d/icinga2 init script:

cp /etc/init.d/icinga2 /etc/init.d/icinga2.orig
vim /etc/init.d/icinga2
if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -d -e $ICINGA2_ERROR_LOG --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then
...
if ! $DAEMON daemon -c $ICINGA2_CONFIG_FILE -C --no-stack-rlimit > $ICINGA2_STARTUP_LOG 2>&1; then

 

You May Also Like…

Icinga in the year 2021

Icinga in the year 2021

The year is coming to an end, this will be our last blogpost for this year too, abd a lot of people are already in the...

Subscribe to our Newsletter

A monthly digest of the latest Icinga news, releases, articles and community topics.