A systems administrator with four years of experience has released Prod is Down, a free browser-based trainer that lets users receive root access to deliberately broken Linux servers and practice incident response without risking production systems.
The platform currently contains 13 scenarios. After a short briefing that includes recent changes and symptoms, the server begins to fail while metrics are collected. Players then open a live terminal, investigate with standard tools such as top, df, journalctl, and ss, and finally press a verification button that runs an external health-check script.
Security relies on gVisor (runsc) instead of runc, per-container Docker networks marked internal, removal of most Linux capabilities, strict memory and process limits, and a watchdog that deletes containers exceeding 1 GB of writes. The author also added --allow-suid and --overlay2=none flags after discovering that default gVisor settings broke sudo and disk accounting.
Because players have root inside the container, verification never trusts files or binaries present in the environment. The check script is supplied at runtime from the host, and any modification to /usr, /bin, /etc/sudoers or scenario-specific files causes immediate rejection. Scenario parameters are stored as a SHA-256 hash so players cannot alter the failure conditions.
An emulated systemd written in Python provides realistic behavior for systemctl, journalctl, Restart=, EnvironmentFile=, and LimitNOFILE= so that experience transfers directly to real servers that run full systemd.
Early user feedback prompted interface changes: the “Take incident” button was moved to a fixed position, and a hint prompt now appears after three failed verification attempts. The first scenario remains open without registration.


