Local LLM Deployment for SOC: How Many Incidents Can One NVIDIA RTX PRO 6000 Handle?
In the second part of the experiment, R-Vision analysts tested the Qwen3.5-122B-A10B-GPTQ model running locally with vLLM on an NVIDIA RTX PRO 6000 Blackwell Max-Q GPU with 96 GB VRAM. The evaluation moved from synthetic stress tests to realistic SOC workloads using anonymized real incidents from their internal operations center. Tasks included incident ranking, summarization, similarity search, retrospective analysis, and preliminary verdict generation within the R-Vision SOAR orchestration pipeline. Two load scenarios were modeled: a calm shift with 10-15 incidents per hour handled by 3-5 L1/L2 analysts, and a peak scenario with 50-100 incidents arriving in a short period involving 5-7 analysts. Resources were split between a high-priority interactive chat pool and a lower-priority background SOAR pool using an AI Gateway layer to manage queues, context length, and KV-cache usage. Results showed that one GPU can process up to 5 incidents simultaneously in the background pipeline, delivering a theoretical maximum throughput of 300 incidents per hour under the tested conditions and prompt profiles.
HabrJul 27, 2026Translated from Russian