Citadel

PKI Storm: Managing 100,000 Simultaneous Certificate Requests in Kubernetes Recovery Scenarios

A large organization's PKI infrastructure faced a critical bottleneck when a data center outage triggered simultaneous startup of tens of thousands of Kubernetes pods, each requiring mTLS certificates. The existing setup using ESAUS and Citadel routed all requests through external certificate authorities that could only sustain 50-70 RPS against an incoming burst of 100,000 requests. Average daily load of 10-11 RPS had masked the thundering herd risk during mass recovery. Scaling the CA 15x was rejected due to cost and the fundamental dependency on real-time signing. The team introduced pre-issuance of certificates stored in a dedicated Unified Secret Storage (ЕХС) layer that supports 14,000 RPS reads while the CA continues normal operation. This architectural separation of issuance and consumption reduced recovery time from nearly 24 minutes to seconds while shifting focus to secure secret lifecycle management including KRA key protection.