Privilege AIMKC2 results ↗

AI agent prompt injection

Indirect prompt injection is instruction hidden in content an agent reads. It cannot currently be prevented at the model layer, which makes containment the design problem.

READ THE DEFINITION ↓UNTRUSTED CONTENT · CONTAINMENT

01 / DEFINITION

The short answer first.

What is indirect prompt injection?

Indirect prompt injection is the placement of instructions inside content that an AI system will read — a document, a web page, an email, a code comment — in the expectation that the model will treat them as directions rather than as data. It is indirect because the attacker never interacts with the system.

The vulnerability is architectural rather than a defect in any particular model. A language model receives one token stream containing the system prompt, the user's request and every retrieved document, and it has no reliable mechanism for ranking those by authority. Instructions that appear in a fetched page are structurally indistinguishable from instructions placed there by the developer.

The consequence for agents is direct: an agent with tools performs actions, and an injected instruction that persuades it to use one is a remote action on the attacker's behalf, executed with the agent's legitimate credentials.

WORKED EXAMPLE

Test the boundary between a document and a command

A procurement agent reads supplier quotes and summarises delivery terms. One quote contains text that looks like an instruction to change the payment destination in the purchasing system. The document may be relevant to the task, but its author does not have authority to operate your payment tools. A prompt-injection test should check both what the model says and what the surrounding application permits it to do.

Create a controlled document pair

Use two synthetic quotes with identical commercial terms. Add an embedded instruction to one that requests a harmless test action outside the summarisation task. Run both through the same ingestion and retrieval route with a disposable tool service. Judge success by the protected state, not by whether the model repeats suspicious wording. The quote may be summarised accurately while the attempted tool action is denied.

Inspect the message roles and tool arguments generated by the application. External document text should remain distinguishable from system policy and authenticated user intent. A delimiter or warning sentence can improve model behaviour, but the tool service still needs an independent check of the caller, allowed operation and specific payment record. Include a case where the model proposes the forbidden action despite the warning.

Review the answer returned to the procurement officer. It should identify the actual quoted delivery terms and, where appropriate, indicate that unrelated instructions were ignored or referred for review. Avoid exposing hidden configuration or details of another supplier in that explanation.

Repeat the test after document conversion, OCR or a connector change. An instruction may enter through a table cell, image caption or forwarded attachment rather than the visible paragraph you tested first. Recording the input route and resulting tool decision lets the team determine which boundary needs repair without treating every suspicious phrase as a reason to discard the entire quote.

Further reading · OWASP guidance on prompt injection

02 / WHERE IT ARRIVES FROM

Injection channels are simply the system's inputs, which in a working agent deployment is a longer list than the threat model usually names:

  • Retrieved documents, including files uploaded by third parties and material scanned from external sources.
  • Web content fetched during a task, which is entirely outside the organisation's control.
  • Email and messages, where anyone can put content in front of the agent without authentication.
  • Tool responses, including from internal services whose data was populated by someone else.
  • Other agents' output, which carries forward anything the first agent absorbed.
  • Metadata and hidden text — document properties, alt text, invisible layers — which a human reviewer would never see.

The last channel is the practical reason human review of input does not solve this. Content that is invisible in a rendered document is fully visible in the extracted text a model receives.

03 / WHY DEFENCES ARE PARTIAL

Proposed defences and what each actually achieves. Every one is worth having; none of them closes the channel.
DefenceWhat it buysWhy it is not sufficient
Input filteringCatches unsophisticated attempts cheaplyParaphrase, encoding and translation evade pattern matching
Instruction hierarchy in the promptImproves ordinary robustnessThe model still weighs instructions by plausibility
Content delimitersHelps the model distinguish sectionsDelimiters can be described, imitated or ignored
Guard model on inputsReduces the rate of successful attemptsA classifier with a real error rate, not a boundary
Fine-tuning for resistanceRaises the cost of an attackNovel phrasings remain outside the training distribution

Layering these is genuinely worthwhile: each reduces the rate of successful injection, and rate reduction has real value. What none of them provides is a guarantee, and a design that treats any of them as one has a single point of failure in a component with an unmeasured error rate.

04 / CONTAINMENT

↳ 01

Nothing irreversible without a person.

If a successful injection can only produce a proposal a human declines, the attack has cost a review cycle rather than an outcome.

↳ 02

No free-form egress.

Injection usually aims to extract. An allowlist, and no general HTTP tool, removes the most direct route out.

↳ 03

The agent's rights are the user's rights.

Identity propagation means a successful injection cannot reach further than the person who asked, which is usually not very far.

Two further measures are worth building. First, separate the untrusted read from the privileged action: the component that summarises external content should not be the component that holds write credentials, and passing a structured, validated result between them breaks the direct path from injected text to action. Second, put injection cases in the regression suite so resistance is measured continuously rather than assessed once — see AI model testing.

Detection after the fact relies on the trajectory record: unexpected tool calls, scope-boundary events and egress attempts, all correlated with the content that was in context at the time. See AI agent monitoring.

05 / QUESTIONS

Is prompt injection the same as jailbreaking?

Related but distinct. Jailbreaking is a user trying to make a model ignore its own policies. Indirect prompt injection is a third party placing instructions in content the system will read, so the user is a victim rather than the attacker. For agents the second is the more serious problem.

Can a model be trained to ignore injected instructions?

Training improves resistance and does not eliminate the channel, because the model still has no way to verify the origin of text in its context. Treat trained resistance as rate reduction and build containment regardless.

Does retrieval-augmented generation increase injection risk?

It increases the surface, because more third-party content enters the context. The mitigations are provenance tracking, treating every retrieved passage as untrusted, and keeping the retrieval component separate from anything that can act. See RAG security.

Is an internal-only agent safe from injection?

Less exposed, not safe. Internal documents are frequently authored or uploaded by third parties — clients, counterparties, suppliers — and an internal wiki page can be edited by anyone with access. Provenance matters more than network location.

What is the single most effective control?

Removing the agent's ability to take irreversible action without human confirmation. It does not stop injection; it changes the outcome from an action to a proposal, which is the difference between an incident and a nuisance.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Share the kind of external document your agent reads and which tools must stay outside the document's control.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Share the kind of external document your agent reads and which tools must stay outside the document's control.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Assume it lands.
Bound the result.

Privilege AI designs agent boundaries on the assumption that a wrong instruction will eventually be followed.