Website Protection Against Bots: Six Years of Traffic Filtering Evolution from Behavioral Bots to Multi-Layer Analytics

The article traces the author's journey starting in 2020 when anomalous traffic from social networks began flooding websites, initially appearing as visits from Twitter, Instagram, YouTube, and VKontakte. Early attempts using .htaccess rules to block referrers failed as bots dynamically switched to direct or external domain transitions. Server-side JavaScript and cookie checks altered bot behavior but did not stop the flow, revealing that successful browser checks only confirm technical capability rather than human presence. IP blocking of ranges like 31.173.80.0/21 and 178.176.64.0/19 provided temporary relief until mobile proxies rotated addresses, exposing the limits of static blacklists. IPv6 adoption further demonstrated how address-family-specific rules quickly become obsolete. Fingerprint spoofing and constant rotation of digital prints made single-signal identification unreliable. The core lesson emphasizes real-time filtering at the edge over post-hoc analytics exclusion to prevent bots from reaching the web server at all.

HabrTranslated from Russian

Read at source

The author first encountered the issue of web traffic quality in 2020 while working on web development and SEO. Normal traffic patterns in analytics were replaced by storms of anomalous visits that showed identical behavior impossible for real users. Detailed inspection of Yandex Metrika Webvisor reports confirmed these were not genuine visitors but automated scripts aimed at manipulating behavioral factors.

Initial experiments relied on Cloudflare custom rules because no other tool offered comparable technical flexibility. The first published article on the topic documented cases of social-network bot traffic and its negative SEO impact. Research across multiple client projects consistently linked the appearance of this traffic with organic position drops and pessimization.

Between 2020 and 2021 the first wave manifested as 30–60 daily visits from social platforms where the site had no real presence. Blocking referrers via .htaccess simply caused bots to switch to direct entries or endless external domains. Server-side JavaScript and cookie validation forced bots to change their reported source but did not halt the traffic.

Two mobile IP ranges belonging to MegaFon were temporarily blocked with short-term success, yet the bots immediately migrated to new pools. The same pattern repeated with IPv6 traffic, proving that static IP blacklists become a losing race against rotating proxies.

Digital fingerprints proved equally unreliable. Each bot visit presented a fresh browser profile, rendering fingerprint-based detection ineffective. The author coined the term “behavioral bots” to distinguish this class of automation from classic crawlers because the scripts actively imitated human navigation and actions.

Excluding suspicious IPs from Yandex Metrika reports created an illusion of clean statistics while the actual requests continued to hit the origin server. The author repeatedly warned that visual cleanup of analytics does not equal protection and that genuine filtering must occur before requests reach the application.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in Malware & Botnets

Yoba Campaign Uses Signed ChatGPT and Claude Binaries for DLL Sideloading and XMRig Deployment
uCreator Spyware Infiltrates Modified Telegram, WeChat and GCash Builds on Android
Researchers Deploy Cowrie Honeypot Disguised as NVIDIA H100 Server to Study SSH Botnet Tactics