All things sysadmin

Today I had a colleague approach me about a oneliner I sent him many months ago, saying that it kept rebooting a server he was running it on.

It was little more than running sar in a loop, extract some values and run another command if certain thresholds were exceeded. Hardly anything that you’d think would result in a reboot.

After whittling down the oneliner to the offending command, it turned out that sar was the culprit. Some further debugging revealed that sar merely spawns a process called sadc, which does the actual heavy lifting.

In certain circumstances, if you send SIGINT (ctrl+c, for example) to sar, it can exit before sadc has done its thing.
When that happens, sadc becomes an orphan, and /sbin/init being a good little init system, takes it under its wing and becomes its parent process.

When sadc receives the SIGINT signal, it’s signal handler will pass it up to its parent process… You see where this is going, right?
Yep, /sbin/init gets the signal, and does what it should do. Initiates a reboot.

If you want to reboot an Ubuntu 14.x server, simply run this in a terminal (as root, this is NOT a DoS/vulnerability, merely a bug):

1
2
3
4
5
6
7
8
9

root@elcrashtest:~# echo $(sar -b 1 5)
^C
root@elcrashtest:~# ^C
root@elcrashtest:~#
Broadcast message from root@elcrashtest
    (unknown) at 18:06 ...

    The system is going down for reboot NOW!
    Control-Alt-Delete pressed

Rapidly hitting ctrl+c twice does the trick.
Obviously this command doesn’t make sense to run in isolation, but the bug was hit in the context of a more involved oneliner, and being in a subprocess seem to trigger it more often. You may need to run it a couple of times as a few things need to line up for it to happen. The above command reboots the server like 8-9/10 times.

If executed in another subshell, you only need to hit ctrl+c once to trigger it.

A more unrealistic, but sure-fire way to trigger it looks like this:

1
2
3
4
5

root@elcrashtest:~# sar -b 1 100 > /dev/null &
[1] 3777
root@elcrashtest:~# kill -SIGKILL $! ; kill -SIGINT $(pidof sadc);
Broadcast message from root@elcrashtest
...

Basically killing sar forcefully (thus orphaning sadc), and then send SIGINT to sadc. This has a 100% success rate.

This was fixed in 2014, but Canonical has neglected to backport it.
A colleague of mine, who is a much better OSS citizen than I am, has raised this with Canonical

I only tested this on Ubuntu 14.04 and 14.10. Debian and RedHat/CentOS does not appear to suffer from this. It’s surprising that it’s still present in Ubuntu Trusty, since this is backported in Debian Jessie.

Only on a Friday afternoon…