Categories
SysOps

How to protect system against accidental shutdown

Protect the operating system against accidental shutdown or reboot.

Installation

Install molly-guard package.

$ sudo apt install molly-guard

It will basically override halt, shutdown, poweroff and reboot utilities.

It is possible to also override pm-hibernate, pm-suspend and pm-suspend-hybrid utilities, but it will require additional operations due to Debian Bug report logs – #812545 pm-utils: Relative pm-* symlinks in /usr/sbin/ cannot be diverted by molly-guard.

The configuration is stored in /etc/molly-guard/ directory.

Testing

The default configuration will protect against accidental operations.

ansible@example:~$ sudo reboot --molly-guard-do-nothing
I: demo mode; molly-guard will not do anything due to --molly-guard-do-nothing.
W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot: example
molly-guard: would run: /lib/molly-guard/reboot
ansible@example:~$ sudo reboot --molly-guard-do-nothing
I: demo mode; molly-guard will not do anything due to --molly-guard-do-nothing.
W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot: x
Good thing I asked; I won't reboot example ...
W: aborting reboot due to 30-query-hostname exiting with code 1.

Beware, you can use a more direct approach to reboot the server.

$ ssh ansible@example sudo reboot --molly-guard-do-nothing
I: demo mode; molly-guard will not do anything due to --molly-guard-do-nothing.
molly-guard: would run: /lib/molly-guard/reboot

Creating additional messages

You can create messages for specific operations and store these in /etc/molly-guard/messages.d directory.

Define the message for reboot operation.

$ echo -e "\nI hope you know what you're doing...\n" | sudo tee /etc/molly-guard/messages.d/reboot

Every other operation without a defined message will use default file if it exists.

$ echo -e "\nPlease inform DevOps team before executing this operation.\n" | sudo tee /etc/molly-guard/messages.d/default

The following examples will illustrate this behavior.

ansible@example:~$ sudo reboot --molly-guard-do-nothing
I: demo mode; molly-guard will not do anything due to --molly-guard-do-nothing.
I hope you know what you're doing...

W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot: example
molly-guard: would run: /lib/molly-guard/reboot
ansible@example:~$ sudo shutdown --molly-guard-do-nothing
I: demo mode; molly-guard will not do anything due to --molly-guard-do-nothing.

Please inform DevOps team before executing this operation.

W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot: example
molly-guard: would run: /lib/molly-guard/shutdown

Creating additional checks

Require stdin opened on a terminal to not execute any of these operations when using SSH non-interactively.

Create 20-require-stdin check-in /etc/molly-guard/run.d/ directory.

#!/bin/sh
#
# 20-require-stdin - require stdin opened on a terminal
#
set -eu
# require an interactive terminal connected to stdin
if [ ! -t 0 ]; then
  echo "STDIN is not opened on a terminal; I won't $MOLLYGUARD_CMD ..." >&2
  exit 1
fi

Ensure that proper permissions are applied to this file.

The following example will use this check to protect the system against accidental reboot.

$ ssh ansible@example sudo reboot --molly-guard-do-nothing
I: demo mode; molly-guard will not do anything due to --molly-guard-do-nothing.

I hope you know what you're doing...

STDIN is not opened on a terminal; I won't reboot ...
W: aborting reboot due to 20-require-stdin exiting with code 1.
ansible@example:~$ sudo reboot --molly-guard-do-nothing
I: demo mode; molly-guard will not do anything due to --molly-guard-do-nothing.

I hope you know what you're doing...

W: molly-guard: SSH session detected!
Please type in hostname of the machine to reboot: example
molly-guard: would run: /lib/molly-guard/reboot

Additional notes

I have used --molly-guard-do-nothing parameter in every example described here, remove it to execute the specific (shutdown, reboot) operation.