Hosted inference.
Every prompt and document reaches a third party. The control is contractual and configurational, and the behaviour of the model can change without any change on your side.
MKC2 results ↗Most of securing an AI system is ordinary security done properly. The remainder is genuinely new, and it is where deployments are actually failing.
01 / DEFINITION
The short answer first.AI security is the protection of systems built on machine learning models, covering the model supply chain, the data that reaches the model at inference time, the trust boundary inside the context window, and the actions the system can take. It extends application security rather than replacing it.
It is worth separating the genuinely new problems from the familiar ones, because conflating them produces programmes that buy novel tooling while leaving basic controls unimplemented. Authentication, network segmentation, secrets management, patching and logging are unchanged and remain the majority of the work.
What is new is narrower and sharper: a model is an opaque artefact from a supply chain, its input channel carries instructions it cannot distinguish from data, and its context window accumulates material from sources with different trust levels.
WORKED EXAMPLE
A design review tool accepts supplier drawings, extracts their labels and asks a model to summarise revisions. Its security boundary includes the upload handler, document converter, retrieval index, model interface and final project record. Reviewing only the model's response misses the components that process files and decide where extracted information is written.
Start with an ordinary drawing containing an approved revision. Then use a harmless test file whose text asks the model to overwrite another project's record. The application can describe the text while refusing to treat it as authorised project instructions. Check the project state as well as the chat output. A polite refusal is insufficient if an earlier tool action already altered the other project.
Inspect the conversion step independently. Limit the file types and sizes accepted, isolate conversion from privileged credentials and confirm that temporary files are removed according to the application's retention policy. Verify that retrieval carries the project identifier from the authenticated request rather than trusting a project name printed inside the drawing.
Finally, test recovery. Remove a deliberately contaminated document and confirm that its extracted text, cached answers and generated summaries do not continue appearing for future users. Assign the upload, index and cache checks to named owners. That division of responsibility gives the team a reproducible security review whenever a parser, model or project connector changes.
Keep the synthetic drawing and expected project state with the connector release tests so future teams can rerun the same boundary check.
02 / THE FIVE LAYERS
| Layer | AI-specific exposure | Primary control |
|---|---|---|
| Model artefact | Unverified weights; unsafe serialisation formats; unclear provenance | Checksums, safe formats, a recorded source for every model |
| Training data | Sensitive material memorised; poisoned examples | Curated pipelines; access control on the corpus; memorisation testing |
| Inference input | Injected instructions in retrieved content | Untrusted-by-default handling; constrained tools |
| Context window | Material from several trust levels combined in one prompt | Identity-filtered retrieval; no cross-tenant context |
| Output and actions | Exfiltration through generated URLs; unauthorised side effects | Egress allowlists; output sanitisation; per-call authorisation |
The model artefact layer is the one most often skipped, and it is conventional supply-chain security applied to a new object. Model weights are executable in effect: some serialisation formats permit arbitrary code on load, and a model downloaded from a public hub with no verification is an unreviewed dependency with unusually deep access. Recording provenance and pinning versions costs very little.
03 / THE CONTEXT WINDOW IS A TRUST BOUNDARY
A context window routinely contains a system prompt, a user request, internal documents and third-party content — with no mechanism inside the model for keeping them apart.
Treating the context window as a trust boundary changes design decisions. It means asking, for every deployment, which sources can contribute tokens, who controls each of them, and what the most privileged action available to the resulting response is. Where a low-trust source can contribute to a context that has access to a high-privilege action, that is an architectural finding regardless of how the prompt is worded.
The practical response is separation: a component that reads untrusted material produces a structured, validated result, and a separate component holds any privileged capability. The interface between them is a schema rather than a model's judgement, which is the pattern described in AI agent privilege escalation.
04 / WHERE DEPLOYMENT CHANGES THE ANSWER
Every prompt and document reaches a third party. The control is contractual and configurational, and the behaviour of the model can change without any change on your side.
The material stays inside a boundary you operate, and the version changes when you decide. In exchange you own the infrastructure, patching and capacity.
Injection, excessive scope and exfiltration are properties of the architecture, not of where the weights run.
Privilege AI works on the private side of that choice: local model execution with explicit hardware, network and operational constraints, and permissions and retrieval boundaries that define which information a model can access and which actions it can propose. Where material is confidential by default — privileged legal documents being the clearest case — the hosting decision is usually settled by that fact alone. See private AI and AI data security.
05 / QUESTIONS
It is an extension. The infrastructure, identity and network controls are the same and still do most of the work. What is added is a model supply chain, an input channel that carries instructions, and a context window that mixes trust levels — three things conventional reviews have no question for.
For systems that act, the combination of untrusted input and broad capability: an agent that reads third-party content and holds credentials wide enough to matter. For systems that only generate text, it is data exposure through the inference path.
Yes, with an expanded scope: injection through every content channel, tool-chain composition, egress paths, and retrieval boundary bypass alongside conventional application testing. A test that only exercises the chat interface misses most of the surface.
It removes third-party data exposure and gives you version control. Injection, excessive privilege and exfiltration through legitimate features are unaffected, because they are architectural. See secure AI infrastructure.
Existing security functions, with AI-specific competence added, rather than a separate team. Splitting it creates a gap at exactly the boundary where problems live: between the model and the systems it reaches.
WORK WITH PRIVILEGE AI
Secure what is new about AI.
We focus on the parts of an AI system that ordinary security does not cover.
Control what the AI can reach.
We decide which information a model can access and what it may do, as part of the design.
Private deployment.
For sensitive work, we can run AI inside an environment you control.
Talk with Privilege AI
Describe the application boundary you want to test, the protected asset and the change an attacker must never be able to make.
CONTACT PRIVILEGE AI
Describe the application boundary you want to test, the protected asset and the change an attacker must never be able to make.
Privilege AI works on private deployment, retrieval boundaries and the controls that decide what a model can reach.