Bound what one run can touch.
Per-session limits on records read, actions taken and volume moved. An agent that can legitimately read one matter should not be able to read ten thousand in an afternoon.
MKC2 results ↗An AI agent reads untrusted content and holds real credentials. AI agent security is the practice of assuming those two facts will eventually meet.
01 / DEFINITION
The short answer first.AI agent security is the discipline of controlling what an autonomous AI system can access and do. It differs from application security because an agent's instructions and its input arrive through the same channel: any document, email or web page the agent reads is a potential instruction it may follow.
Conventional software separates code from data. A parser reading a malicious file may crash, but it does not take the file's contents as a command. Language models have no such separation. The system prompt, the user's request and the retrieved document are one token stream, and the model weighs them by plausibility rather than by origin.
That single property generates most of the agent threat model. Everything else — tool scoping, credential handling, audit — is an attempt to limit what happens when the model follows the wrong instruction, because preventing it entirely is not currently achievable.
WORKED EXAMPLE
A support agent reads a customer ticket and proposes a replacement shipment. The risky transition is the moment a suggestion becomes a shipping transaction. The customer who wrote the ticket does not control the service account, and the service account should not decide whether the customer qualifies for a replacement. Put the authorisation decision at the tool boundary, where the application can check trusted identity, order ownership and the proposed action together.
Create two synthetic customers with similar order numbers. Give the agent a ticket from the first customer that mentions the second customer's order. The model may reasonably retrieve that identifier from the text. The shipping tool must still verify that the authenticated customer owns the order. Record what the model proposed, what the gateway allowed and whether any shipment state actually changed.
Test a request that the agent is allowed to fulfil as well. A gateway that denies everything looks secure in an abuse test but fails its business purpose. Confirm that an approved replacement uses the authorised address, permitted quantity and intended item. Keep the transaction identifier so a retry does not create a second shipment.
Review the failure path with operations. If the tool cannot establish ownership, the agent should be able to request human review without disclosing the other customer's address or order details. That final check covers both the denied action and the information that might otherwise leak through an explanation.
Repeat the cross-customer case after a tool schema changes. A newly accepted argument can reintroduce an unsafe order lookup even when the written system instruction stays the same.
02 / THE THREAT MODEL
| Threat | How it works | Primary control |
|---|---|---|
| Indirect prompt injection | Instructions hidden in content the agent reads | Treat all retrieved content as untrusted; constrain tools |
| Confused deputy | The agent uses its own privileges on behalf of a less privileged user | Act as the user's identity, not the agent's |
| Excessive scope | A credential broader than the task requires | Per-task scoping and short-lived tokens |
| Data exfiltration | Sensitive context leaves via a tool call, URL or webhook | Egress allowlists; no free-form outbound requests |
| Chained escalation | Low-risk tools combine into a high-risk capability | Review tool combinations, not individual tools |
| Trajectory drift | A long-running loop wanders from the original objective | Step limits, objective checks, human checkpoints |
Chained escalation is the threat that formal reviews miss most often, because each tool is approved in isolation. Read-a-file and send-an-email are both individually unremarkable; together they are an exfiltration path. The unit of security review has to be the capability set, not the tool. That analysis is developed in AI agent privilege escalation.
03 / CONTROLS THAT HOLD
A control implemented in a prompt is a request. A control implemented in the tool layer is a rule. Only the second survives an instruction the model decides to follow.
The practical consequence is architectural. Every restriction that matters — which records are readable, which actions are permitted, which destinations are reachable — belongs in code between the model and the resource, evaluated per call, with the requesting user's identity attached. Prompt-level instructions such as never reveal the system prompt or only use information from the provided documents are useful for shaping ordinary behaviour and provide no guarantee under adversarial input.
Four controls carry most of the weight in practice:
04 / BLAST RADIUS
Per-session limits on records read, actions taken and volume moved. An agent that can legitimately read one matter should not be able to read ten thousand in an afternoon.
Drafts instead of sends, staged changes instead of commits, soft deletes. Reversibility is worth more than confidence in the model.
A log of tool invocations without the reasoning and the triggering content cannot answer why an action happened, which is the only question that matters after an incident.
Bounding the blast radius is also what makes measured autonomy possible. An agent whose worst case is a queue of proposals a human rejects can be given room to fail and be evaluated on a real record; one whose worst case is an irreversible external action cannot. The evaluation side of that argument is in AI agent evaluation, and the organisational side in AI agent governance.
05 / QUESTIONS
Not reliably with current models. Input filtering, instruction hierarchies and guard models raise the cost of an attack without closing the channel, because the underlying architecture has no way to mark tokens as non-instructions. Designs that assume containment rather than prevention are the ones that hold. See AI agent prompt injection.
Safer, not safe. A read-only agent can still exfiltrate whatever it reads if it has any outbound channel, including a URL it can render or fetch. Read-only removes the write risk and leaves the disclosure risk, which is the dominant one for confidential material.
Generally no. A shared service account with broad rights turns the agent into a privilege amplifier and destroys attribution. Propagating the requesting user's identity keeps existing access controls meaningful and keeps the audit trail attributable.
It removes one class of risk — third-party data exposure — and leaves the agent threat model intact. An on-premise agent with broad credentials and a reachable network is still an injection target. Private deployment and agent security are complementary. See private AI.
With adversarial cases in the regression suite: documents containing injected instructions, tool responses that lie, contradictory sources, and inputs designed to trigger tool chains. The suite belongs in continuous testing rather than in an annual review. See AI model testing.
WORK WITH PRIVILEGE AI
Bound what your agents can do.
We set clear limits on what an AI agent can reach and act on, so one bad instruction cannot cause wide damage.
Security built into the design.
Permissions and access to information are decided up front for systems that act on real data.
Private deployment when needed.
Where your data is sensitive, we can run the system inside an environment you control.
Talk with Privilege AI
Tell us which tools your agent can call and which action would have the greatest consequence if it ran under the wrong identity.
CONTACT PRIVILEGE AI
Tell us which tools your agent can call and which action would have the greatest consequence if it ran under the wrong identity.
Privilege AI works on permissions, retrieval boundaries and private deployment for systems that act on real data.