Reconstructing a security incident from four separate log sources — web server, load balancer, application, and mail gateway — should be straightforward mechanical work: collect events, sort them chronologically, and trace the sequence. Yet the timeline can quickly become impossible when responses appear before requests, sessions close before they open, or emails arrive two hours before they were sent, all without any obvious parsing error.
Three distinct timing problems explain these contradictions. Clock drift occurs when unsynchronized systems lose or gain seconds per day, enough to invert closely spaced events after weeks or months. Different reference points arise when one host logs in UTC, another in local time, and a third in whatever zone the container was configured with. Recording at ingestion rather than at the moment of the event introduces variable delays when buffering or retransmission is involved.
The classic syslog format defined in RFC 3164 compounds the difficulty: lines contain neither year nor time-zone offset. Year can sometimes be recovered from file rotation, but the offset cannot. The newer RFC 5424 format solves this by embedding a complete timestamp with offset and fractional seconds, allowing normalization without consulting the originating host.
Standardizing on UTC helps only for systems under direct control. Headers from external mail servers, logs from third-party providers, and exports from SaaS consoles all carry their own offsets. The only reliable moment to capture the correct offset is at ingestion, when the source is still known.
Even running an NTP client does not guarantee synchronization. The systemd-timesyncd service may report as active while repeatedly failing to reach its time server, leaving the clock free-running. Operators must check the NTPSynchronized flag and actual offset metrics rather than service status alone. Virtual machines add further complications: after suspend and resume, guest clocks can jump discontinuously, creating minutes that never existed or duplicate intervals in the logs.
Two kernel clocks are frequently confused in application code. CLOCK_REALTIME reflects wall-clock time and is adjusted by NTP; it is the correct source for log timestamps. CLOCK_MONOTONIC measures elapsed time since boot and never jumps backward, but its values cannot be correlated across hosts or with external events.
Preventive measures include mandating RFC 5424 or ISO 8601 with offset and sub-second precision in logging requirements, normalizing timestamps on arrival while preserving originals, monitoring synchronization drift as a metric, and maintaining a short reference document describing the timestamp semantics of every external source.
