Server logs can reveal significant details about potential attackers by capturing sequences of events on Linux systems. This practical guide for junior Blue Team specialists shows where to locate traces of suspicious activity, how to interpret SSH and web server records, and why a single unusual request does not necessarily indicate a compromise.
Essential Log Sources
Three primary sources support initial investigations. System journals contain service events, errors, and operating system messages; on systemd-based distributions these are accessed with journalctl. SSH and authentication logs record successful and failed login attempts, unknown usernames, and access-related events. Web server and application logs document HTTP requests, requested resources, response codes, and application errors.
On Debian and Ubuntu systems authentication events typically reside in /var/log/auth.log; on RHEL-based systems they appear in /var/log/secure. Recent SSH events can be viewed with:
sudo journalctl -u ssh --since "1 hour ago"
or, on distributions where the service is named sshd:
sudo journalctl -u sshd --since "1 hour ago"
The last 100 lines of an Nginx access log are retrieved with:
sudo tail -n 100 /var/log/nginx/access.log
Failed SSH Login Attempts
Example entries show repeated failures from the same source IP using different usernames at short intervals. Such patterns may indicate automated credential testing, yet they can also result from misconfigured automation or forgotten passwords. Quick searches use:
sudo grep -E "Failed password|Invalid user" /var/log/auth.log
Key Indicators in Logs
- Event frequency: hundreds of attempts in minutes versus occasional failures.
- Username diversity: sequential testing of many accounts suggests automation.
- Successful logins following failures: these require verification against expected user behavior.
- Timing: nighttime activity must be compared with normal operational schedules.
Web Server Request Analysis
Nginx access logs may contain sequences such as requests for /admin, /backup.zip, and /.env. Consecutive checks of common sensitive paths often signal automated reconnaissance, although individual requests can originate from legitimate users or crawlers. Response codes alone do not confirm compromise because applications may return 200 OK for missing resources.
Identifying Active IP Addresses
The following command counts the most frequent client addresses in an Nginx access log:
sudo awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -20
High counts require context; they may reflect API usage, load balancers, or reverse proxies rather than malicious traffic. Headers such as X-Forwarded-For are reliable only when trusted proxies are correctly configured.
Application-Level Logging
System and web-server logs often lack details about authorization decisions. Structured JSON events should record timestamp, event type, user identifier, resource, result, and reason. The OWASP Logging Cheat Sheet recommends capturing successful and failed authentication, access-control denials, input-validation errors, and administrative actions.
Information That Must Not Appear in Logs
- Passwords and session tokens
- Private keys
- Database connection strings containing credentials
- Unnecessary personal data
Logs must be protected against tampering, rotated regularly, and shielded from log-injection attacks through proper encoding.
Limitations of Log Data
Even well-configured logging cannot capture every event. Clocks may be unsynchronized, IP addresses may represent NAT or VPN pools, and privileged attackers can alter or delete local logs. Mature environments forward events to centralized logging or SIEM platforms.
Minimal Investigation Plan
- Record the exact time window of suspicious events.
- Preserve original log files and system metadata without modification.
- Correlate SSH, web-server, application, and system logs for the same period.
- Check for successful actions after sequences of failures.
- Compare observed activity against known baselines such as backups and scheduled tasks.
- Assess impact on accounts, data, and configurations.
- Execute response procedures only after evidence supports the hypothesis.
Server logs remain a foundational Blue Team tool when events contain sufficient context, timestamps are consistent, and analysis combines multiple data sources.
