Storage and bandwidth.
Weights, indexes and evaluation sets, plus enough read throughput that model loading and retrieval do not become the bottleneck.
Memory first.
Everything else
follows.
Private AI infrastructure planning is mostly arithmetic: weights, activations and a key-value cache that grows with every token of context.
READ THE DEFINITION ↓MEMORY · CACHE · CONCURRENCY
01 / DEFINITION
The short answer first.What is private AI infrastructure?
Private AI infrastructure is the compute, memory, storage and network capacity needed to run AI models inside an organisation's own boundary. The binding constraint is almost always GPU memory: model weights plus a key-value cache that grows linearly with context length and concurrency.
02 / THE MEMORY ARITHMETIC
The calculation that determines the hardware.Three things occupy GPU memory during inference, and only the first is commonly estimated.
Weights. Roughly two bytes per parameter at 16-bit precision, so an 8B model occupies about 16 GB before anything else. At 8-bit it is about 8 GB, and at 4-bit about 4 GB.
Key-value cache. Grows with every token in context, multiplied by every concurrent request. This is the term that surprises people: a deployment sized comfortably for short prompts can run out of memory as soon as users start pasting long documents, because the cache scales with context length and concurrency together.
Activations and overhead. Working memory for the forward pass, plus framework overhead and fragmentation. A practical planning margin is to leave headroom rather than fitting to the last gigabyte.
The useful way to plan is to fix the context length and concurrency you intend to support, size the cache for that, and treat the result as the real requirement. Sizing on weights alone consistently produces a deployment that works in testing and fails on the first real workload.
| Lever | Effect on memory | Cost of using it |
|---|---|---|
| Lower precision (quantisation) | Roughly halves weight memory per step down | Some quality loss, which must be measured on your task |
| Shorter context limit | Reduces cache growth proportionally | Long documents need chunking, which changes retrieval design |
| Lower concurrency | Reduces total cache linearly | Queueing; users wait at peak |
| Smaller model | Reduces everything | Capability, unless the task is narrow enough to fine-tune for |
03 / QUANTISATION IS AN EVALUATION QUESTION
Do not accept a quality claim you have not measured.Cheaper weights,
measured quality.
Quantisation makes a model fit. Whether it still performs is a question for your own held-out set, not for a general benchmark.
Published quantisation comparisons typically report general benchmark scores, which are a poor proxy for a specialised task. Degradation is uneven: it tends to affect long-context reasoning, rare vocabulary and calibrated confidence more than it affects short-form general knowledge — and calibration is exactly the property that decides whether a score can be used for routing.
The right procedure is to run the same held-out evaluation at each precision you are considering, report all four measurement families rather than accuracy alone, and choose deliberately. A model that loses one accuracy point and half its ranking quality has become substantially less useful while appearing almost unchanged. The method is in held-out evaluation and the metrics in LLM evaluation metrics.
04 / WHAT ELSE THE DEPLOYMENT NEEDS
The parts that are not the GPU.A retrieval tier.
Vector and keyword search run on CPU and RAM, sized separately from inference. Retrieval is frequently the slower half of a RAG system.
Operations.
Driver and firmware compatibility, capacity monitoring, a patching route for an isolated environment, and someone on call for hardware failure.
Two operational details are worth deciding early. Model updates in an isolated environment need a defined import process, with provenance and checksums, because a convenient download path is exactly the outbound route the boundary was built to remove. And capacity monitoring has to include cache pressure, not just utilisation: the failure mode of an undersized deployment is a request that cannot be admitted, which looks like an application error rather than a capacity problem.
05 / QUESTIONS
Asked when sizing a private deployment.How much GPU memory does a private LLM need?
+
Start from about two bytes per parameter at 16-bit precision for the weights, then add key-value cache for your intended context length times concurrency, then leave headroom for activations and fragmentation. The cache term dominates once long documents are in play, and it is the one usually omitted.
Can private AI run on CPU only?
+
For small models and low volume, yes, with latency measured in seconds per response. It is viable for batch work and generally not for interactive use. Anything with concurrent users wants GPU acceleration.
Does quantisation hurt quality?
+
Usually a little, sometimes a lot, and unevenly across capabilities. Long-context reasoning and calibration tend to degrade before short-form accuracy does, which matters if the score is used for routing. Measure at each precision on your own held-out set rather than relying on published comparisons.
How many concurrent users can one GPU serve?
+
It depends on model size, context length and how long responses are. The honest answer comes from a load test at your intended context length; estimates derived from short-prompt benchmarks are consistently optimistic because they understate cache pressure.
What breaks first in a private deployment?
+
Memory, under long-context load. The second most common is the retrieval tier, which is sized as an afterthought and then becomes the latency bottleneck once the corpus grows.
Size it
for the real workload.
Privilege AI works on local inference and on-premise infrastructure with explicit hardware, network and operational constraints.