vLLM

Local LLM Deployment for SOC: How Many Incidents Can One NVIDIA RTX PRO 6000 Handle?

In the second part of the experiment, R-Vision analysts tested the Qwen3.5-122B-A10B-GPTQ model running locally with vLLM on an NVIDIA RTX PRO 6000 Blackwell Max-Q GPU with 96 GB VRAM. The evaluation moved from synthetic stress tests to realistic SOC workloads using anonymized real incidents from their internal operations center. Tasks included incident ranking, summarization, similarity search, retrospective analysis, and preliminary verdict generation within the R-Vision SOAR orchestration pipeline. Two load scenarios were modeled: a calm shift with 10-15 incidents per hour handled by 3-5 L1/L2 analysts, and a peak scenario with 50-100 incidents arriving in a short period involving 5-7 analysts. Resources were split between a high-priority interactive chat pool and a lower-priority background SOAR pool using an AI Gateway layer to manage queues, context length, and KV-cache usage. Results showed that one GPU can process up to 5 incidents simultaneously in the background pipeline, delivering a theoretical maximum throughput of 300 incidents per hour under the tested conditions and prompt profiles.

Former OpenAI CTO Mira Murati Launches Thinking Machines' Inkling: Open-Weights Multimodal AI Model with 975 Billion Parameters and Self-Training Demo

Thinking Machines, founded by former OpenAI technical director Mira Murati, has released Inkling, its first open-weights multimodal AI model that supports text, images, and audio in a unified architecture. The model uses a mixture-of-experts design with 975 billion total parameters but activates only 41 billion at once, supports a 1-million-token context window, and was trained on 45 trillion tokens spanning text, images, audio, and video. A smaller Inkling Small variant with 12 billion active parameters was also introduced for faster and cheaper inference. Key innovations include adjustable reasoning depth that lets developers control compute usage per query and a self-training experiment where the model autonomously fine-tuned itself via the Tinker platform to avoid using one letter of the English alphabet. Weights are now available on Hugging Face with support for Transformers, vLLM, SGLang, and llama.cpp, positioning Inkling as a flexible foundation for further customization rather than a direct competitor to closed frontier models.