Linux administrators often encounter situations where a server suddenly consumes more resources, unknown network connections appear, or a previously stopped process restarts after reboot. This does not always indicate a compromise and may result from faulty applications, incorrect service configurations, backup operations, or routine maintenance tasks.
Starting with Processes
Each running program instance receives a unique PID. To list processes sorted by CPU usage, run ps aux --sort=-%cpu | head -20. For memory consumption, use ps aux --sort=-%mem | head -20. Interactive monitoring is available via top or htop when installed.
High resource usage by an unexpected binary such as /opt/unknown/worker warrants further investigation. Useful details include the parent process ID, start time, and command line obtained with ps -p 1842 -o pid,ppid,user,lstart,etime,cmd.
Examining the Executable File
The actual binary path is revealed by readlink -f /proc/1842/exe. Command-line arguments are shown with ps -p 1842 -o args=. File metadata including owner and modification time can be checked using stat. On Debian-based systems, dpkg -S identifies the package owning a file when applicable.
Reviewing Network Connections
Listening TCP and UDP sockets are displayed with sudo ss -lntup, while established connections use sudo ss -tnp. The lsof -i command provides additional process-to-port mapping when available. LISTEN and ESTAB states indicate listening services and active connections respectively.
Checking systemd Services
Running services are listed via systemctl list-units --type=service --state=running. Unit files and failed services are examined with systemctl list-unit-files --type=service and systemctl --failed. Detailed status and configuration are retrieved using systemctl status and systemctl cat, paying special attention to the ExecStart directive.
Investigating Scheduled Tasks
User cron jobs are viewed with crontab -l. System-wide schedules reside in /etc/cron.d/, /etc/cron.hourly/, and similar directories. systemctl list-timers --all shows systemd timers. Each discovered job should be verified against system documentation and file ownership.
Auditing User Accounts and SSH Keys
Local users are enumerated with getent passwd. Accounts possessing UID 0 are identified by awk -F: '$3 == 0 {print $1 ":" $3}' /etc/passwd. Authorized SSH keys are typically stored in ~/.ssh/authorized_keys and should be compared against approved lists.
Correlating Events and Preserving Evidence
System logs are queried with sudo journalctl --since "today" or filtered by service name. When multiple indicators align, such as an unusual SSH login followed by new service creation and unexpected outbound connections, further investigation is warranted. If compromise is suspected, first record timestamps and preserve logs before remediation.
