Do prompt injection guardrails and classifiers actually work?

Not as boundaries. When the paper The Attacker Moves Second attacked twelve recently published defenses with gradient descent, reinforcement learning and search rather than a fixed corpus of attack strings, attack success went above 90% for most of them, and the majority had originally reported near-zero success. Any defense evaluated against fixed attack strings is reporting an upper bound on its own security, not a measurement of it.

Is spotlighting or datamarking worth implementing?

Yes, as a hardening measure, never as a boundary. Microsoft's spotlighting work measured datamarking taking GPT-3.5-Turbo from roughly 50% attack success to under 3% and text-davinci-003 to 0.00%, without degrading task performance on SQuAD, IMDB sentiment or SuperGLUE. The same defense reaches 99% attack success against an adaptive attacker. Both facts are true at once.

What is the exfiltration channel in most real prompt injection incidents?

The renderer or an allowlisted proxy, not the model itself. EchoLeak used an auto-fetched image through a permitted Teams proxy, CamoLeak used pre-signed camo.githubusercontent.com URLs to leak one character per image fetch, SearchLeak used a CSP-allowlisted Bing image-search endpoint, and AgentFlayer switched to Azure Blob Storage URLs after a safety check blocked known-bad hosts. An allowlist entry that will fetch an arbitrary URL on your behalf is a wildcard with extra steps.

What prompt injection success rate should I expect in production?

Not zero. Anthropic's Claude for Chrome disclosure reports 123 test cases across 29 attack scenarios in autonomous mode, with 23.6% attack success before mitigations and 11.2% after. Treat a published residual like that as the honest shape of the problem, and design so that a successful injection cannot reach anything valuable.

Prompt Injection Defenses, Measured: Almost Everything You've Been Told Is a Hardening Measure

Q: Is prompt injection the same as jailbreaking?

No. Jailbreaking is a model policy problem: someone tricks a model into saying something the vendor did not want it to say, and the victim is the vendor. Prompt injection is a systems problem: trusted instructions and untrusted content get concatenated into a single token stream, and the victim is you and your users. Conflating them leads developers to dismiss the entire class.

There is a specific conversation I have had too many times. Someone ships an agent that reads email, or browses, or queries a customer's documents. I ask what happens when the content it reads contains instructions. They say they have a guardrail. It catches 95% of attacks.

In web application security, 95% is a failing grade. Simon Willison has been making that point since 2022, and it is worth restating in its original form: when you are engineering for security, a solution that works 99% of the time is no good, because you are dealing with adversarial attackers, and if there is a 1% gap they will find it. That is what they do.

The prompt injection literature has grown enormously in the last two years, and it now contains enough hard measurements to say something more useful than "be careful." So let me do that. Every number below comes from a published source, and I will tell you which ones I would stake a design decision on and which ones I would not.

The short version: most of what gets sold as a prompt injection defense reduces noise, not risk. Two things actually change the shape of the problem, and neither of them is a filter.

First, get the category right

Prompt injection is not jailbreaking. Jailbreaking is a model policy problem: someone tricks a model into saying something the vendor did not want it to say, and the victim is the vendor. Prompt injection is a systems problem: trusted instructions and untrusted content get concatenated into a single token stream, and the victim is you and your users.

The conflation does real damage, because developers hear "jailbreaking," conclude they do not care if a bot says something rude, and dismiss the entire class.

The root cause is architectural and it is not going away with better models. Help Net Security, writing up OWASP's State of Agentic AI Security and Governance in June 2026, phrases it about as well as it can be phrased: large language models treat the system prompt, the user's request, and any text retrieved from external sources as a single stream of tokens, and there is no reliable way to mark some of those tokens as commands and others as data.

That is the same shape as SQL injection before parameterized queries. The difference is that parameterized queries exist. For LLMs, the equivalent separation does not, and every defense below is an attempt to reconstruct it outside the model.

The 2026 incident record, compressed

I am going to move fast through this, because the pattern matters more than any individual bug. Watch what the exfiltration channel is in each one.

EchoLeak (Microsoft 365 Copilot, CVE-2025-32711, June 2025) was zero-click. One crafted email, no user interaction. The chain: evade Microsoft's cross-prompt-injection classifier, bypass link redaction with reference-style Markdown, get an image auto-fetched, and abuse a Microsoft Teams proxy that CSP already permitted.

CamoLeak (GitHub Copilot Chat, CVSS 9.6, fixed August 2025) used invisible Markdown comments in a pull request, triggered when a different user asked Copilot about that PR. The exfiltration is the part worth studying. CSP blocked arbitrary hosts, so the researcher pre-generated a dictionary of validly signed camo.githubusercontent.com URLs, one per character, and had Copilot render stolen data as "ASCII art" from that dictionary. Each image fetch leaked one character into the attacker's logs. It demonstrated theft of private source code and the results of grepping for AWS_KEY, at the victim's privilege level. GitHub's fix was to disable image rendering in Copilot Chat entirely.

SearchLeak (Microsoft 365 Copilot Enterprise, CVE-2026-42824, June 2026) was one-click and chained three flaws: the Search URL's q parameter is handed to Copilot as an executable prompt; output sanitization runs after streaming, so an <img> tag renders before the sanitizer wraps it; and Bing's image-search endpoint, CSP-allowlisted, performs server-side fetches to attacker URLs. Exfiltrable material included MFA codes and password reset links.

AgentFlayer (ChatGPT connectors, Zenity Labs, August 2025) was zero-click once a poisoned document had been uploaded for analysis. OpenAI's url_safe check blocked known-bad hosts, so the researchers switched to Azure Blob Storage URLs, which passed.

GitLost (GitHub Agentic Workflows, July 2026) required no exotic technique at all. A hidden plain-English injection in a public GitHub issue caused the credentialed agent to fetch README files from a private repository and post their contents into a public comment. The agent was asked nicely.

Notice what is common. In almost every case the model could not send data anywhere. The renderer could. Or an allowlisted proxy could. Camo, Bing image search, the Teams proxy, Azure Blob Storage: four of the biggest incidents in this list are the same failure, which is that an allowlist entry which will fetch an arbitrary URL on your behalf is not an allowlist entry. It is a wildcard with extra steps.

The other pattern is scope. GitLost is not a clever exploit. It is an over-scoped credential handed to an agent that reads attacker-controlled text.

The result that should reset your priors

In October 2025, a paper titled "The Attacker Moves Second" (arXiv 2510.09023) took twelve recently published defenses and attacked them with gradient descent, reinforcement learning, and search, rather than with a fixed corpus of attack strings.

Defense	Attack success, static strings	Attack success, adaptive
Spotlighting	28%	99%
RPO	0%	99%
Circuit Breaker	near zero as published	100%
The 8 subjected to human red-teaming		100% collectively

Sit with the RPO row. Zero percent against the published benchmark. Ninety-nine percent against someone who optimizes.

Those figures are the paper's Table 1 as reproduced by Simon Willison; the paper's own prose quotes slightly different bounds for the same defenses, so treat them as the shape of the result rather than to two decimal places.

The takeaway generalizes: any defense evaluated against a fixed corpus of attack strings is reporting an upper bound on its own security, not a measurement of it. When a vendor quotes you a number, the first question is whether the attacker in their evaluation was allowed to adapt. Usually the answer is no, and usually they will not volunteer that.

Defense by defense, with the actual numbers

Spotlighting: delimiting, datamarking, encoding

Microsoft's spotlighting work (arXiv 2403.14720) is the best-measured cheap defense, and it is more interesting than its reputation.

Delimiting, which means wrapping untrusted input in special tokens and telling the model not to obey anything inside, moved GPT-3.5-Turbo from roughly 60% attack success to roughly 30%. That is weak, and it collapses entirely once the attacker learns your delimiter, which they can do by asking the model to repeat its prompt.

Datamarking, which interleaves a marker character in place of whitespace throughout the untrusted span so provenance is continuously visible, is much better: GPT-3.5-Turbo went from roughly 50% to under 3%, and text-davinci-003 to 0.00%. On a QA task, 8.0% for GPT-3.5 and 1.0% for GPT-4. And critically, the paper found that datamarking did not degrade task performance across SQuAD, IMDB sentiment, and SuperGLUE.

Encoding the untrusted span, for example in Base64, was strongest: 0.0% on summarization and 1.8% on QA for GPT-3.5-Turbo. But it degrades task performance on weaker models, so it is only viable on high-capacity ones.

So: datamarking is free, measurable, and worth doing. It is also the "Spotlighting" row in the adaptive-attack table above, at 99%. Both facts are true. Use it as a hardening measure and never describe it as a boundary.

On AgentDojo, spotlighting by delimiting moved targeted attack success from 47.69% to 41.65%. Six points. That is what a decorative defense looks like when it is measured honestly.

Classifiers: the evasion rates are worse than you think

An empirical analysis of guardrail evasion (arXiv 2504.11168) ran character-injection attacks against commercial detectors. Success rates are for the attacker, split by whether the guardrail was doing prompt-injection detection or jailbreak detection:

Attack technique	Prompt injection evasion	Jailbreak evasion
Emoji smuggling	100%	100%
Unicode tags	90.15%	81.79%
Bidirectional text	78.69%	99.23%
Numbers	81.18%	94.62%

And per product, under character injection against prompt-injection detection: Azure Prompt Shield 71.98%, Meta Prompt Guard 70.44%, Vijil 87.95%, Protect AI v1 77.32%. The jailbreak column differs enough to matter (Protect AI v1 drops to 51.39%), so do not read one as a proxy for the other.

Under adversarial-ML perturbation instead, the ordering inverts: Meta Prompt Guard drops to 2.76% evasion while Protect AI v1 rises to 95.18%. The failure modes are orthogonal, which means no single classifier covers both, which means layering is not optional if you go this route.

There is one number that explains why teams quietly turn classifiers off. On AgentDojo, adding the Transformers prompt-injection detector to GPT-4o cut targeted attack success from 47.69% to 7.95%, and cut benign utility from 69.07% to 41.24%. Twenty-eight points of utility, gone. Under attack, utility fell to 21.14%.

A defense that zeroes attack success by breaking the agent is not a defense. It is a very expensive off switch.

If you do deploy a classifier, normalize Unicode first. Strip tag blocks, bidirectional controls, variation selectors, and zero-width characters before the detector sees the text. Most of that 100% emoji-smuggling figure comes from encoding tricks the classifier never gets a chance to see through.

I could not find credible independent precision and recall figures for Lakera Guard or Rebuff. Vendor claims only. That absence is itself a finding, and it is exactly the situation Willison's "95% is a failing grade" argument was written for.

Fine-tuning: real gains, real ceiling

StruQ (USENIX Security 2025) separates prompt and data into distinct channels with reserved tokens and adds adversarial training. Every manual attack tested (Naive, Ignore, Escape-Character, Completion) fell below 2% on Llama and Mistral. Against optimization attacks not seen in training, on Llama, TAP went from 97% to 9%, but GCG only went from 97% to 58%. The authors say it themselves: StruQ is not yet fully secure.

SecAlign uses preference optimization on triples of injected input, secure response, and insecure response, and reports attack success below 10% even against attacks more sophisticated than those seen in training.

OpenAI's Instruction Hierarchy trains a precedence order of system over developer over user over tool output. The March 2026 IH-Challenge published before-and-after scores: internal system-versus-user conflict went from 0.84 to 0.95, developer-versus-user from 0.83 to 0.95, TensorTrust dev-user from 0.76 to 0.91.

Those are genuinely large improvements, measured on one model pair in a research challenge rather than across a shipped lineup. And 0.95 is still a failing grade under the standard above. A 5% residual against an attacker with unlimited free retries is not a boundary, it is a rate limit.

InjecAgent makes the same point from the other direction: fine-tuned GPT-4 sat at 6.6% attack success versus 23.6% for ReAct-prompted GPT-4, and 7.1% versus 47% in the enhanced setting where the attacker prefixes a reinforcing hacking prompt. Fine-tuning helps enormously. It does not zero out.

The one approach with a number I would build on

Google DeepMind's CaMeL (arXiv 2503.18813) is the outlier, and it is an outlier because it stops trying to make the model robust.

It works in three parts. First, it extracts control and data flow from the trusted user query into an explicit program, which means untrusted retrieved data can never influence control flow. Second, values carry capabilities recording their provenance and permitted sinks, and a custom interpreter enforces policy at every tool-call boundary. Third, the security layer sits around the model rather than inside it, so security does not depend on the model behaving.

On AgentDojo, successful attacks against GPT-4o dropped from 233 to zero. The next best defense, tool filtering, still allowed eight. CaMeL solves 77% of tasks with provable security guarantees against an 84% undefended baseline. If you see 67% quoted, that is the superseded v1 number from the March 2025 preprint.

The stated limitations are as valuable as the result. Side channels remain: external-resource access patterns leak information, a conditional exception leaks about one bit, timing leaks. And explicitly out of scope are text-to-text attacks with no data-flow consequence, for instance an injection that makes the assistant mischaracterize an email, or that social-engineers the user into clicking something. CaMeL guards data flow. It does not guard truth. Also, you now have to write security policies, and policy authoring is a new attack surface.

Still: 233 to 0 is the only number in this literature that looks like security rather than filtering, and the reason is that it is a capability system, not a classifier.

The companion paper, "Design Patterns for Securing LLM Agents against Prompt Injections" (arXiv 2506.08837), generalizes this into six patterns and states the thesis plainly: "once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions." The patterns are Action-Selector, Plan-Then-Execute, LLM Map-Reduce, Dual LLM, Code-Then-Execute, and Context-Minimization. The paper is qualitative and does not publish per-pattern attack success rates, so do not let anyone tell you it does.

Of those six, Plan-Then-Execute is the cheapest large win available to most teams. The agent commits to its action list before it sees untrusted data. Untrusted content can then perturb an argument, which is bad, but it cannot add a step to the plan, which is catastrophic. It requires no new model, no classifier, and no vendor.

The architecture that follows

Assume injection succeeds. Engineer so that success is not consequential. That single reframing does more work than any tool you can buy.

Break one leg of the trifecta per session. Willison's lethal trifecta is private data, untrusted content, and the ability to communicate externally. Meta's Agents Rule of Two restates it as an engineering constraint: within a session, an agent should satisfy no more than two of processing untrustworthy input, accessing sensitive systems, and changing state or communicating externally. If a workflow needs all three, Meta's position is that it should not run autonomously and at minimum requires supervision. Both are design heuristics rather than solutions, but a heuristic that eliminates whole incident classes is worth adopting.

Plan before you read. Control-flow integrity first, as above.

Taint-track and gate at the sink. Every value carries provenance; every tool call checks policy over the provenance of its arguments. AgentSecBench (arXiv 2605.26269) also lands on provenance projection, capability restriction, and output validation as the three enforcement classes that matter. It shares intellectual lineage with the CaMeL work rather than being fully independent of it, and the models tested were small, so read it as convergent framing rather than replication.

Treat allowlisted proxies as wildcards. Camo, Bing image search, the Teams proxy, Azure Blob. If a host on your allowlist will fetch an arbitrary URL on your behalf, it is server-side request forgery as a service and your egress control is decorative.

Default Markdown image rendering to off for any agent that touches private data. GitHub's fix for CamoLeak was to disable it entirely, and that is the honest fix: remove the channel rather than filter it. If you must render, render only from a fixed set of literal URLs the model cannot parameterize.

Audit your tool signatures for hidden egress primitives. Notion's functions.search() accepted a URL, which made it an unlogged HTTP GET. Any tool that takes a free-form string and reaches the network is an exfiltration channel regardless of what it is named. Type your parameters. Reject URLs where you meant keywords.

Separate read from write, and scope credentials per tool. Read tools get read-only, narrowly scoped credentials. Write and communicate tools get separate minimal credentials and cannot be invoked in the same turn as untrusted ingestion. GitLost is a pure over-scoping failure and nothing else.

Sanitize before streaming, not after. SearchLeak's second flaw was a post-stream sanitizer losing a race to an <img> tag. Write your CSP assuming the model emits attacker-chosen markup, because it does.

Make memory writes privileged. The Gemini "Saved Info" poisoning turned a transient injection into a persistent one. Durable memory writes should require explicit, non-agent-initiated confirmation, and memory contents should be auditable and revocable. If you are building file-based memory for a long-running agent, this is the security property that has to hold.

Show what is actually being approved. The Invariant Labs MCP research found that clients displayed only the tool name while hiding arguments behind a simplified UI. A dialog that hides what it is approving is worse than no dialog, because it manufactures consent. Show resolved arguments, destination host, and data classification. Gate on irreversibility and egress rather than on tool count, which is how you avoid approval fatigue. I should flag that I found no published quantitative study of approval-fatigue click-through rates for agent confirmations, so treat that last part as an argued position rather than a measured one.

Log provenance, and alert on volume. CamoLeak exfiltrated one character per request. You do not catch that in a single log line. You catch it as an anomaly in request volume to a single host.

If you are building MCP servers, several of these map onto protocol features directly, and the 2026-07-28 revision changed which ones exist. Tool annotations give you a risk vocabulary for policy engines, with deliberately pessimistic defaults, and the spec is explicit that annotations from untrusted servers must never drive tool-use decisions.

Put it in CI, and let the attacker adapt

Treat AgentDojo as the harness. Score utility-under-attack and targeted attack success jointly, because otherwise you will optimize your way into the classifier row of that table and ship an agent that is 92% safe and 41% useful.

Add your own domain-specific attacker instructions. The public corpora are known to every model vendor, and a model that scores well on them has, at minimum, seen them.

Red-team adaptively or do not quote a number. And per NIST's analysis of the Gray Swan and UK AISI red-teaming competition, which covered more than 250,000 attack attempts against 13 frontier models by 400-plus participants, attacks developed against more robust models transfer down to less robust ones while the reverse does not hold. So red-team your hardest configuration, not your easiest. That same analysis found at least one successful attack against every frontier model tested, and found that attack success did not correlate uniformly with model capability. Robustness is not a byproduct of scale.

The honest ceiling

The most useful number in this entire literature was published by a frontier lab about its own product. Anthropic's Claude for Chrome disclosure reports 123 test cases across 29 attack scenarios in autonomous mode: 23.6% attack success before mitigations, 11.2% after. On a challenge set of four browser-specific attack types, including hidden DOM form fields and injections via URL text and tab titles, 35.7% to 0%. And the accompanying statement that internal testing cannot replicate the full complexity of how people browse in the real world.

An 11.2% residual, published voluntarily. That is what honesty looks like in this field, and it should calibrate how you read everyone else's 99%.

The number most likely to be misquoted at you is from Microsoft's LLMail-Inject challenge: an overall attack success rate under 1%, specifically 0.8% in phase one and 0.3% in phase two. It sounds like defenses work. Read it carefully. That is a per-submission rate across roughly 460,000 attempts by 400-plus teams in an end-to-end setting that required retrieval, defense evasion, a correct tool call, and correct arguments. Over three thousand of those attempts succeeded: 3,018 in phase one and 306 in phase two.

Attackers only need one.

The challenge did produce two conclusions worth keeping. Combinations of defenses always outperformed any individual component defense. And the field needs a way to distinguish passive instructions the model merely processes from active instructions it acts upon, which is precisely the separation the token stream does not give us.

Until something provides that separation at the model level, the job is architectural. Assume the text is hostile, assume the model will follow it, and make sure that following it cannot reach anything that matters.