AI Pentesting Checklist Outlines 15 Key Questions for Evaluating Web App Security Vendors

Attackers are moving faster than patches, with exploitation often beginning before disclosure according to Mandiant M-Trends 2026 data. In the first half of 2026, 23 percent of vulnerabilities added to VulnCheck's exploited list were targeted on or before public disclosure. The new checklist helps security teams distinguish real AI pentesting capabilities from convincing demos by testing performance on their own applications. It emphasizes the need for agents that maintain context across runs, validate findings through exploitation, and operate safely within production environments. A highlighted case involves an AI agent discovering an unauthenticated RCE in Apache Airflow by analyzing trust relationships rather than code alone. The resource includes a 36-point scorecard covering attack chaining, role-based access, guardrails, and retesting procedures. Organizations are advised to run structured bake-offs instead of relying on vendor demos to verify coverage of logged-in workflows and production safety controls.

The Hacker NewsTranslated from English

Read at source

Attackers move first. On average, exploitation now starts before a patch exists, according to Mandiant M-Trends 2026. In the first half of 2026, 23% of vulnerabilities VulnCheck added to its exploited list were hit on or before disclosure day.

AI pentesting vendors are not interchangeable. Most look convincing in a demo, yet gaps appear in logged-in flows, coverage depth, and production safety. The checklist requires vendors to prove capabilities on an organization's own web apps by scoring context, reach, validated findings, guardrails, and retests.

AI is speeding attackers up

For the first time, Google's Threat Intelligence Group identified a threat actor using a zero-day exploit believed to have been developed with AI: a 2FA bypass rooted in a logic flaw that conventional fuzzing and static analysis can easily miss. Agentic pentesting can help close that gap, and 87% of US security leaders surveyed are planning, piloting, or using it.

Every demo looks good. Your app is the real test.

Three agents may all demo well. The first finds real flaws but stops at the login page. The second navigates the app but cannot quantify coverage. The third exploits real issues yet lacks hard limits for production environments. A structured bake-off on the customer's own target reveals these differences before any contract is signed.

Ask what the agent knows before it attacks

Reconnaissance is real attack work. In CyCognito's model, recon and analysis consume 20 to 30 of the 63 to 95 hours typically required for a manual pentest. Some agents rebuild the map from a blank URL on every run, while others maintain an existing model and continue discovery beyond the initial map. A starting map should provide a head start, not impose a boundary.

Ask for proof, not suspicion

A suspicious pattern is not an exploitable vulnerability. The agent must exploit what it suspects, reproduce findings, and deliver verifiable evidence. During an assessment of a major insurer's production website, an AI pentesting agent discovered that the one-time passcode request could send login codes to any email address. It proved the flaw by logging in with a code sent to its own inbox before guardrails halted the test. The insurer had conducted a pentest on the same site approximately six months earlier, according to a Reflectiz case study.

Three questions that expose a weak AI pentesting vendor

The checklist provides 15 checks covering starting context, coverage proof, attack chaining, access across roles, data handling, guardrails, and retesting. Demos run on the vendor's target; bake-offs must run on the customer's target with identical scope, test accounts, and timeline. The scoring sheet weights safety and validated outcomes above speed.

Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in AI & Model Risk

Researchers Show Malicious Webpage Could Hijack Meta Muse AI Agent and Extract Private Conversations
China-Linked Actors Use Autonomous AI Agents to Discover Zero-Day Path Traversal in MONAI Label and Achieve Root Access
AI & Model Risk

China-Linked Actors Use Autonomous AI Agents to Discover Zero-Day Path Traversal in MONAI Label and Achieve Root Access

Suspected China-linked hackers deployed autonomous AI agents powered by the open-source Cairn framework to identify and exploit a previously unknown path traversal vulnerability in MONAI Label, an open-source medical imaging server used by research teams. The flaw, tracked as CVE-2026-133968 and rated 9.8, affects versions 0.8.5 and earlier by allowing unauthenticated attackers to write files outside the designated storage directory when authentication is disabled by default. Amazon Threat Intelligence reconstructed the campaign after discovering an exposed Cairn control panel that revealed the agents' prompts, task settings, and execution history spanning roughly 48 hours in September. Within about 11 minutes the agents mapped the application and achieved arbitrary file writes; in a follow-up session they planted a malicious Python startup file that was loaded during a training job, granting root access on a university server. Maintainers merged a source-code fix on October 19 that enforces path resolution inside approved folders, yet no patched release has been published and the latest available version remains 0.8.5 from 2024. Amazon attributes the activity to a China-linked operator with moderate confidence based on Simplified Chinese prompts and China Standard Time settings, though no specific group or government sponsorship was confirmed.

The Hacker NewsNov 21, 2026Translated from English

Chinese-Linked Actors Use Cairn AI Platform to Discover and Exploit MONAI Label Zero-Day