The operators of the decentralized Cat Tunnels service arrived at work expecting a routine day until the director burst in with reports that users across Russia could no longer connect reliably.
Although the service is designed to be decentralized, it still depends on a set of tracker nodes that help new clients locate entry points. At the time of the incident, several dozen such nodes were running in different countries, providers, and data centers. Every single one of them was blocked inside Russia.
The team began manually spinning up replacement servers, only to see each new instance blocked within minutes. This reactive approach proved unsustainable, leading to the development of Kotator-Rotator.
Kotator-Rotator continuously monitors relay availability using statistics collected directly from clients. Nodes that consistently disappear from client views are automatically decommissioned and replaced. The more severe the blocking, the more aggressively the system rotates infrastructure.
The rotation threshold is governed by Grohotator, an aggregated metric calculated from client technical logs. When the value exceeds a defined limit, a loud siren sounds across the office floor. The metric proved essential because server-side checks showed all relays as healthy and reachable from the public internet; only client perspectives revealed the actual connectivity failures.
Review of the logs also uncovered operational patterns on the blocking side. Activity drops sharply on Friday evenings and resumes around 8 a.m. on Monday, suggesting the responsible teams follow standard working hours.
