Garak

Local LLMs in AppSec: What Open Tools and Models Can Handle in Application Security

The article examines how local large language models can be integrated into application security workflows without relying on cloud subscriptions. It traces the evolution from llama.cpp in 2023 through agentic frameworks like PentAGI and Strix in 2025-2026. Teams can now run models on local hardware, connect them to code analyzers such as Semgrep, and delegate specific tasks like vulnerability path reconstruction or multi-step application testing. The piece highlights practical scenarios including static code review, dynamic testing with browser and terminal agents, and security evaluation of embedded AI assistants using tools like Garak and PyRIT. It stresses the importance of task decomposition, context management, and verification of model outputs to avoid misleading results. Hardware performance, model selection, and custom agent configurations are presented as key factors for successful adoption.

Hermes Emerges as Modular Harness for Practical AI Security Testing

The article details the development of Hermes, a lightweight modular harness designed to turn raw language models into reliable AI security agents. Unlike heavy frameworks such as OpenClaw, Hermes uses dynamic skill loading, strict runbooks stored in Git, and a file-system source of truth to eliminate hallucinations and context bloat. It continuously scans dozens of sources including NVD, CISA KEV, arXiv and research blogs, then applies the custom TIPS scoring metric to rank threats before attempting reproduction. A dedicated Paper-to-PoC skill converts academic attack descriptions into executable code that is validated inside an isolated Ollama and Docker sandbox. Eight mandatory validation gates must pass before any finding is accepted, ensuring reproducibility and freshness of data. The system already supports tools such as Garak, PyRIT, promptfoo, fickling, modelscan and presidio, and evolves autonomously by committing successful attack scenarios back to its repository.