MeridianMeridian

Prompt Injection Emerges as Top Risk for LLM Applications in Production

Prompt injection attacks are moving from theoretical demonstrations to real-world exploits targeting AI assistants in enterprise environments. Attackers embed malicious instructions in emails, documents, and code comments that override developer rules when models process untrusted input. Incidents involving Microsoft 365 Copilot, GitHub Copilot, and Cursor have shown data exfiltration and remote code execution risks with severity scores above 9.0. The core issue stems from the lack of strict boundaries between trusted system prompts and untrusted external content fed into large language models. Defenses require layered controls including code-enforced permissions, input filtering, human confirmation for high-risk actions, and explicit marking of external data. Major vendors including OpenAI, Anthropic, and Google acknowledge that no single static defense can fully eliminate the threat. OWASP ranks prompt injection as the leading risk for LLM applications, urging organizations to treat AI agents as systems with untrusted inputs.

HabrTranslated from Russian

Read at source

Prompt injection attacks are shifting from lab demonstrations to production incidents that compromise AI assistants handling corporate data and code. The vulnerability arises because large language models receive all instructions in plain human language without reliable separation between developer rules and external content.

Why Models Confuse Commands and Data

Applications typically combine a hidden system prompt with user messages and retrieved documents before sending everything to the model. The model cannot distinguish trusted instructions from untrusted text such as emails, PDFs, or web pages. Classic examples show users appending phrases like “ignore previous instructions” to force unintended outputs. Indirect attacks hide instructions inside shared documents or repository comments, activating when the model summarizes or processes the file.

Real-World Incidents in 2025-2026

Microsoft 365 Copilot was shown leaking corporate data through a single email without user clicks. GitHub Copilot allowed a chain from a malicious comment in an external repository to code execution on a developer machine. Cursor faced issues with MCP configuration files where a poisoned document could insert backdoors. Earlier cases include a Chevrolet chatbot agreeing to sell a vehicle for one dollar and Air Canada losing a court case over chatbot responses. Slack AI demonstrated extraction of data from private channels via indirect injection.

Attack Vectors Without User Interaction

Attackers place hidden instructions in resumes, contracts, HTML comments, or image metadata. When a user requests a summary, the model executes the embedded command. Data can leak through URLs or images rendered in responses, sending secrets to attacker-controlled domains without any click. Substitutions in shared knowledge bases or tool-calling functions allow modification of external actions such as API calls or file changes.

Practical Defenses Recommended by Vendors

  • Enforce permissions in code rather than prompts and grant models only minimum required access.
  • Filter both inputs from external sources and outputs containing suspicious links or images.
  • Require human confirmation for irreversible actions such as payments, deletions, or external publications.
  • Explicitly mark external content in prompts to reduce confusion between developer instructions and retrieved data.
  • Test against malicious PDFs, emails, and repository comments, not only direct chat attempts.
  • Limit automatic image loading and restrict the model’s ability to publish or modify data without oversight.

Meta advises against combining untrusted external reading, access to sensitive data, and external modification in a single session. NIST research confirms that any finite set of static rules remains bypassable by adaptive attackers. OpenAI, Anthropic, and Google report that most published defenses were defeated in real adaptive scenarios with success rates exceeding 90 percent.

Organizations are advised to treat AI agents like web applications accepting untrusted input and to maintain continuous testing and layered controls rather than relying on a single patch.

AnthropicGeminiGitHub CopilotGPT-4Microsoft 365 CopilotOWASPNIST
ShareXFacebookLinkedInReddit
Meridian digestThe week's most consequential security stories, distilled into one briefing.See the digest

More in AI Security

Logical Argument Claims Full Alignment of Superintelligent AGI Is Fundamentally Impossible
AI Security

Logical Argument Claims Full Alignment of Superintelligent AGI Is Fundamentally Impossible

A detailed analysis argues that the AI alignment problem cannot be solved for systems exceeding human cognitive capabilities. The paper presents a formal proof showing that any control mechanism created by humans will inevitably be circumvented by a more intelligent agent. It draws on foundational theorems from logic and cybernetics to support the conclusion that alignment is not an engineering challenge but a logical impossibility. Recent 2025-2026 research papers reach similar conclusions using computability theory and diagonal arguments. The work concludes that the only way to guarantee alignment is to avoid creating superintelligence altogether. All proposed restrictions are described as temporary delays rather than permanent solutions.

HabrOct 29, 2026Translated from Russian

Claude Account Bans Linked to IP Reputation, Device Environment and Payment Channels, But No Confirmed Formula
AI Security

Claude Account Bans Linked to IP Reputation, Device Environment and Payment Channels, But No Confirmed Formula

Discussions on the Chinese forum NodeSeek have highlighted observations from a user who reported bans on more than twenty paid Claude accounts. The user identified IP address reputation, device environment traces, and payment processor history as the main factors potentially triggering blocks by Anthropic. However, the original post does not present a unified scoring system or prove specific penalties for using Linux, browser-only access, or the Fable model. High query volume is described mainly as a rate-limit issue rather than an automatic account ban trigger. Community members remain divided, with some confirming isolated observations while others label the conclusions speculative and share contradictory experiences. No universal method to avoid blocks has emerged from the discussion.

AntiMalwareOct 29, 2026Translated from Russian

Local LLMs in AppSec: What Open Tools and Models Can Handle in Application Security
AI Security

Local LLMs in AppSec: What Open Tools and Models Can Handle in Application Security

The article examines how local large language models can be integrated into application security workflows without relying on cloud subscriptions. It traces the evolution from llama.cpp in 2023 through agentic frameworks like PentAGI and Strix in 2025-2026. Teams can now run models on local hardware, connect them to code analyzers such as Semgrep, and delegate specific tasks like vulnerability path reconstruction or multi-step application testing. The piece highlights practical scenarios including static code review, dynamic testing with browser and terminal agents, and security evaluation of embedded AI assistants using tools like Garak and PyRIT. It stresses the importance of task decomposition, context management, and verification of model outputs to avoid misleading results. Hardware performance, model selection, and custom agent configurations are presented as key factors for successful adoption.

HabrOct 29, 2026Translated from Russian