Distributed Crawler Poses as Human Visitor to Evade Analytics and Ad Filters

A detailed investigation revealed a sophisticated distributed crawler that successfully mimicked legitimate human browsing behavior across multiple unrelated websites. The crawler generated realistic events in Google Analytics while avoiding ad script loading and resource chains that would confirm genuine user sessions. Server logs showed consistent patterns of fake search engine referrers, internal navigation with fabricated Referer headers, and selective requests limited mostly to images returning 404 errors. Analysts built a graph-based detection system that correlated events across sites to expose the coordinated activity despite individual requests appearing benign. The campaign rotated browser signatures and IP addresses frequently, making single-site rate limiting ineffective. The case demonstrates how modern crawlers can exploit Measurement Protocol and incomplete bot detection to consume server resources without contributing to revenue.

HabrTranslated from Russian

Read at source

A site owner watching traffic graphs climb while advertising revenue stayed flat would normally celebrate algorithmic success. In this case the numbers told a different story: Google Analytics recorded new sessions, the server performed full TLS handshakes and dynamic page generation, yet no corresponding ad impressions were billed.

The discrepancy arises because GA4 accepts events via the Measurement Protocol, a server-to-server channel originally intended for offline interactions. A crawler that crafts the correct payload can register a session without ever rendering advertisements or executing the full browser resource chain of CSS, JavaScript, fonts and favicons.

Operators of a network of unrelated sites noticed the same subtle footprint on each property. Individual requests carried ordinary modern browser strings, arrived from search-engine referrers, and followed internal links at a human-like pace. Only when events were aggregated into a cross-site graph did the coordinated pattern emerge: dozens of IP addresses replaying nearly identical sequences while rotating User-Agent strings and never loading supporting page assets.

One particularly convincing candidate, masked as 203.0.113.0/24, produced roughly one hundred events over eight days. Its profile included search transitions, internal navigation, and a modest set of static resources. Raw logs, however, showed that almost half the requests openly declared themselves with the string “crawler” and that the supposed resources were exclusively images returning 404 responses without Referer headers.

Further examination of referrer data exposed additional contradictions. The same addresses claimed simultaneous arrivals from Google, Yandex, Bing and Baidu within minutes, an impossible sequence for any single human reader. Internal-link Referers were fabricated to simulate article-to-article reading, yet no stylesheets, scripts or actual page images were ever requested.

The investigation relied on a continuously updated graph whose vertices represented addresses, network blocks, browser masks and requested resources. Edges captured temporal and causal relationships, allowing analysts to test whether short, seemingly natural sessions on separate sites actually belonged to the same automated actor. GPU-accelerated processing made parallel comparison of hundreds of candidate biographies feasible.

The episode illustrates that server logs remain the only immutable record when analytics platforms and advertising systems can be fed synthetic events. By focusing on missing resource chains and contradictory origin stories rather than any single IP or country, defenders can surface distributed crawlers that have studied human browsing patterns in detail.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Malware & Botnets

Yoba Campaign Uses Signed ChatGPT and Claude Binaries for DLL Sideloading and XMRig Deployment
uCreator Spyware Infiltrates Modified Telegram, WeChat and GCash Builds on Android
Researchers Deploy Cowrie Honeypot Disguised as NVIDIA H100 Server to Study SSH Botnet Tactics