Enumerate the destinations.
Every network destination the inference path can reach, from configuration and from observed traffic. An allowlist that nobody has tested is a statement of intent.
Nothing
leaves the room.
Private inference is a claim about a request path. Made precisely, it is one of the few statements about AI that a reviewer can verify.
READ THE DEFINITION ↓PATH · COPIES · EGRESS
01 / DEFINITION
The short answer first.What is private AI inference?
Private AI inference means the model runs on infrastructure the organisation controls, so prompts, retrieved documents and generated outputs are never transmitted to a third party. The claim is about the request path: nothing in it crosses an organisational boundary.
Stated that way it is checkable. A reviewer can ask which processes handle a request, which network destinations they reach, and what persists afterwards — and the answers are either verifiable or they are not. Most privacy claims about AI systems are not stated this precisely, which is why they are difficult to assess.
02 / WHAT A HOSTED REQUEST CARRIES
Naming what actually crosses the boundary.A single hosted inference call transmits more than the user's question. The payload contains the system prompt, the conversation so far, every retrieved passage the pipeline selected, any tool definitions and results, and the output that comes back. For a retrieval-augmented system on confidential material, the retrieved passages are the sensitive part and they are the bulk of the payload.
Provider-side handling then determines what persists: request logs, abuse-detection retention, caching, and whatever the terms permit regarding training. These are real and configurable protections, and they are assurances about behaviour rather than facts about location. For material that is confidential by default, the distinction between they undertake not to retain it and it never arrived is usually the one that decides the architecture.
| Stage | Hosted path | Private path |
|---|---|---|
| Prompt assembly | Local | Local |
| Retrieved content | Transmitted to the provider | Stays inside the boundary |
| Model execution | Provider infrastructure | Your hardware, your version |
| Output | Returned across the network | Never leaves |
| Provider logs | Exist, subject to configuration and terms | None, because there is no provider |
| Version changes | Possible without notice | Only when you decide |
03 / WHAT STILL APPLIES INSIDE
Private is not the same as safe.The boundary holds.
The controls still matter.
A local inference path removes third-party exposure and leaves every internal control exactly where it was. Four in particular:
- Retrieval permissions. An unpermissioned index inside your own boundary still lets any user read everything — see AI access control.
- Log access. Prompt and output logs are the densest copy of sensitive material in the system and are frequently the most widely readable store in it.
- Cache isolation. A response cache keyed only on the question will hand one user's answer to another.
- Egress from the wider system. Private inference does not help if an agent in the same environment can make arbitrary outbound requests.
Privilege AI's own supervision model runs entirely offline for exactly this reason: material reviewed in a legal setting is privileged, and an offline path makes the statement about it simple. The surrounding controls — permissions, retrieval boundaries, review-only authority — are designed alongside it rather than assumed. See MKC2.
04 / VERIFYING THE CLAIM
How a reviewer can check it.Trace the persistence.
Where prompts, outputs, embeddings and caches are written, who can read each store, and how long each is kept.
Test the failure path.
What happens when the local model is unavailable. A silent fallback to a hosted provider would void the entire claim.
The third point is a real configuration risk rather than a theoretical one. Resilience features that fall back to an external provider when a local endpoint fails are convenient and they convert a verified boundary into an occasional one. In an environment where the boundary is the reason the deployment exists, the correct behaviour on failure is to stop and report.
05 / QUESTIONS
Asked by reviewers assessing the inference path.What is the difference between private inference and encrypted transmission?
+
Encryption protects data in transit to a third party who then decrypts and processes it. Private inference means there is no third party in the path. Both are useful; only the second changes who holds the plaintext.
Does a zero-retention agreement achieve the same thing?
+
It is a meaningful assurance about provider behaviour, not a change in where processing happens. For data that is confidential by default the distinction between an undertaking and a location often decides the decision on its own.
Can private inference run without any network access at all?
+
Yes. An air-gapped deployment has no outbound route, so exfiltration is structurally impossible rather than controlled. Model updates and monitoring then need a deliberate import process. MKC2 runs entirely offline.
Is private inference slower?
+
Not necessarily. Local inference removes the network round trip and the shared queue, which often makes latency lower and far more predictable. Throughput is bounded by your own hardware rather than by a provider's capacity.
What should a private inference architecture document state?
+
The processes that handle a request, the destinations they can reach, every store that persists any part of it, who can read each store, the retention period, and the behaviour when the local model is unavailable. That set of answers is what makes the claim verifiable.
A boundary
you can describe.
Privilege AI builds local inference paths with explicit hardware, network and operational constraints.