The inference host holds nothing.
No standing credentials to data stores, no write access to the corpus. It receives prompts and returns text.
Harden the room,
not just
the model.
AI infrastructure security is mostly ordinary infrastructure security applied to an environment with unusual dependencies and unusually sensitive logs.
READ THE LAYERS ↓SEGMENT · VERIFY · CONTAIN
01 / DEFINITION
The short answer first.What is secure AI infrastructure?
Secure AI infrastructure is the hardened environment an AI system runs in: segmented networks with controlled egress, verified model artefacts, isolated inference processes, protected secrets and restricted logs. It is the layer beneath model and application security.
Most of this is conventional practice. What makes it worth stating separately for AI is that the dependency surface is unusual — large binary artefacts from public hubs, fast-moving numerical libraries, GPU drivers — and that the logs are unusually sensitive, because they contain the prompts and the documents inside them.
02 / THE LAYERS
Controls that carry the most weight.| Layer | Concern | Control |
|---|---|---|
| Network | Outbound requests from an inference host | Segmentation with a default-deny egress policy and an allowlist |
| Model artefacts | Unverified weights; unsafe serialisation | Checksums, recorded provenance, safe formats, an internal registry |
| Runtime | A compromised dependency reaching the host | Containerisation, non-root execution, minimal images |
| Secrets | Credentials reachable by the inference process | Short-lived tokens from a secret manager; nothing in environment files |
| Logs | Prompts and documents in a widely readable store | Restricted access, redaction where possible, explicit retention |
| Retrieval tier | An index that bypasses source permissions | Per-document access metadata, filtered at query time |
Default-deny egress is the control with the best return, because it closes the exfiltration route that nearly every other AI-specific risk eventually uses: an injected instruction, a compromised dependency and a mistaken tool configuration all need a way out, and an allowlist removes it for all three at once.
03 / THE MODEL SUPPLY CHAIN
Treating weights as a dependency.A model file
is a dependency.
Weights downloaded from a public hub with no verification are an unreviewed component with deep access to the environment.
Four practices make this tractable, and none of them is expensive:
- An internal registry. Models enter once, through a controlled import, and are served from inside. Production hosts do not download from the internet.
- Checksums and recorded provenance. Which file, from where, verified when, by whom. This is also what makes an evaluation result attributable years later.
- Safe serialisation only. Some formats permit arbitrary code execution on load. Where a conversion is needed, do it once in an isolated environment.
- Pinned versions everywhere. A model referenced by a moving tag is a dependency that can change without a deployment.
The same discipline applies to the numerical and serving libraries around the model, which move quickly and carry substantial native code. Pinning and reviewing them is ordinary supply-chain hygiene applied to an unusually fast-moving ecosystem.
04 / CONTAINMENT
Designing for a compromised component.Retrieval and generation are separate.
Separate processes with separate identities, so a compromise of the generation path does not inherit the index's access.
Nothing shares a cache.
Caches keyed by identity and scoped per tenant, so a leak cannot propagate between users through an optimisation.
Air-gapping is the strongest form of containment and is worth considering wherever the material justifies it, because it converts a set of controls into a property of the environment: with no outbound route, exfiltration is impossible rather than prevented. MKC2 runs entirely offline for this reason. The trade is that updates, monitoring and support all become scheduled, deliberate activities. See on-premise AI.
05 / QUESTIONS
Asked while building the environment.What is the single most valuable AI infrastructure control?
+
Default-deny egress from inference and agent hosts. It closes the route that injection, dependency compromise and tool misconfiguration all rely on, and it is cheaper than any of the controls aimed at those individually.
Are model weights a security risk?
+
The artefact can be, depending on the serialisation format, and an unverified download is an unreviewed dependency. Verify checksums, prefer safe formats, record provenance and serve from an internal registry rather than pulling from a public hub at deploy time.
Should inference run in containers?
+
Generally yes, for isolation, reproducibility and least privilege, with GPU access passed through. Non-root execution and minimal images do most of the work; the containerisation itself is not the control.
How should AI logs be protected?
+
As the most sensitive store in the system, because that is what they are: prompts and retrieved documents in one place. Restrict read access more tightly than the source systems if anything, redact where feasible, and set an explicit retention period.
Does secure infrastructure remove the need for model-level controls?
+
No. Infrastructure bounds what a compromise can reach; it does nothing about a model producing wrong output or an agent taking a permitted action for a bad reason. Both layers are required. See AI agent security.
Build the room
around the model.
Privilege AI treats security boundaries as part of the system: network, retrieval, permissions and inference designed together.