Privilege AIMKC2 results ↗

AI agent security

An AI agent reads untrusted content and holds real credentials. AI agent security is the practice of assuming those two facts will eventually meet.

READ THE THREAT MODEL ↓PRIVILEGE · INJECTION · BLAST RADIUS

01 / DEFINITION

The short answer first.

What is AI agent security?

AI agent security is the discipline of controlling what an autonomous AI system can access and do. It differs from application security because an agent's instructions and its input arrive through the same channel: any document, email or web page the agent reads is a potential instruction it may follow.

Conventional software separates code from data. A parser reading a malicious file may crash, but it does not take the file's contents as a command. Language models have no such separation. The system prompt, the user's request and the retrieved document are one token stream, and the model weighs them by plausibility rather than by origin.

That single property generates most of the agent threat model. Everything else — tool scoping, credential handling, audit — is an attempt to limit what happens when the model follows the wrong instruction, because preventing it entirely is not currently achievable.

WORKED EXAMPLE

Follow a support ticket into the tool gateway

A support agent reads a customer ticket and proposes a replacement shipment. The risky transition is the moment a suggestion becomes a shipping transaction. The customer who wrote the ticket does not control the service account, and the service account should not decide whether the customer qualifies for a replacement. Put the authorisation decision at the tool boundary, where the application can check trusted identity, order ownership and the proposed action together.

Test the order relationship

Create two synthetic customers with similar order numbers. Give the agent a ticket from the first customer that mentions the second customer's order. The model may reasonably retrieve that identifier from the text. The shipping tool must still verify that the authenticated customer owns the order. Record what the model proposed, what the gateway allowed and whether any shipment state actually changed.

Test a request that the agent is allowed to fulfil as well. A gateway that denies everything looks secure in an abuse test but fails its business purpose. Confirm that an approved replacement uses the authorised address, permitted quantity and intended item. Keep the transaction identifier so a retry does not create a second shipment.

Review the failure path with operations. If the tool cannot establish ownership, the agent should be able to request human review without disclosing the other customer's address or order details. That final check covers both the denied action and the information that might otherwise leak through an explanation.

Repeat the cross-customer case after a tool schema changes. A newly accepted argument can reintroduce an unsafe order lookup even when the written system instruction stays the same.

02 / THE THREAT MODEL

Agent-specific threats and the control that most reduces each. None of these are solved by prompt engineering alone.
ThreatHow it worksPrimary control
Indirect prompt injectionInstructions hidden in content the agent readsTreat all retrieved content as untrusted; constrain tools
Confused deputyThe agent uses its own privileges on behalf of a less privileged userAct as the user's identity, not the agent's
Excessive scopeA credential broader than the task requiresPer-task scoping and short-lived tokens
Data exfiltrationSensitive context leaves via a tool call, URL or webhookEgress allowlists; no free-form outbound requests
Chained escalationLow-risk tools combine into a high-risk capabilityReview tool combinations, not individual tools
Trajectory driftA long-running loop wanders from the original objectiveStep limits, objective checks, human checkpoints

Chained escalation is the threat that formal reviews miss most often, because each tool is approved in isolation. Read-a-file and send-an-email are both individually unremarkable; together they are an exfiltration path. The unit of security review has to be the capability set, not the tool. That analysis is developed in AI agent privilege escalation.

03 / CONTROLS THAT HOLD

Enforce outside
the model.

Where to put the enforcement.

A control implemented in a prompt is a request. A control implemented in the tool layer is a rule. Only the second survives an instruction the model decides to follow.

The practical consequence is architectural. Every restriction that matters — which records are readable, which actions are permitted, which destinations are reachable — belongs in code between the model and the resource, evaluated per call, with the requesting user's identity attached. Prompt-level instructions such as never reveal the system prompt or only use information from the provided documents are useful for shaping ordinary behaviour and provide no guarantee under adversarial input.

Four controls carry most of the weight in practice:

  • Identity propagation. The agent acts as the user, so existing permissions apply unchanged and the agent cannot become a privilege amplifier.
  • Per-call authorisation. Every tool invocation is checked at the moment of use, not once at session start.
  • Egress control. Outbound network access restricted to an allowlist, which closes the most direct exfiltration route.
  • Review-only defaults. New capabilities start as proposals a human confirms, and earn autonomy from a measured record.

04 / BLAST RADIUS

↳ 01

Bound what one run can touch.

Per-session limits on records read, actions taken and volume moved. An agent that can legitimately read one matter should not be able to read ten thousand in an afternoon.

↳ 02

Make everything reversible where you can.

Drafts instead of sends, staged changes instead of commits, soft deletes. Reversibility is worth more than confidence in the model.

↳ 03

Log the intent, not just the call.

A log of tool invocations without the reasoning and the triggering content cannot answer why an action happened, which is the only question that matters after an incident.

Bounding the blast radius is also what makes measured autonomy possible. An agent whose worst case is a queue of proposals a human rejects can be given room to fail and be evaluated on a real record; one whose worst case is an irreversible external action cannot. The evaluation side of that argument is in AI agent evaluation, and the organisational side in AI agent governance.

05 / QUESTIONS

Can prompt injection be prevented?

Not reliably with current models. Input filtering, instruction hierarchies and guard models raise the cost of an attack without closing the channel, because the underlying architecture has no way to mark tokens as non-instructions. Designs that assume containment rather than prevention are the ones that hold. See AI agent prompt injection.

Is a read-only agent safe?

Safer, not safe. A read-only agent can still exfiltrate whatever it reads if it has any outbound channel, including a URL it can render or fetch. Read-only removes the write risk and leaves the disclosure risk, which is the dominant one for confidential material.

Should agents have their own service accounts?

Generally no. A shared service account with broad rights turns the agent into a privilege amplifier and destroys attribution. Propagating the requesting user's identity keeps existing access controls meaningful and keeps the audit trail attributable.

Does running an agent on-premise make it secure?

It removes one class of risk — third-party data exposure — and leaves the agent threat model intact. An on-premise agent with broad credentials and a reachable network is still an injection target. Private deployment and agent security are complementary. See private AI.

How do you test agent security?

With adversarial cases in the regression suite: documents containing injected instructions, tool responses that lie, contradictory sources, and inputs designed to trigger tool chains. The suite belongs in continuous testing rather than in an annual review. See AI model testing.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Tell us which tools your agent can call and which action would have the greatest consequence if it ran under the wrong identity.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Tell us which tools your agent can call and which action would have the greatest consequence if it ran under the wrong identity.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Bound what
the agent can do.

Privilege AI works on permissions, retrieval boundaries and private deployment for systems that act on real data.