Privilege AIMKC2 results ↗

Private AI infrastructure

Private AI infrastructure planning is mostly arithmetic: weights, activations and a key-value cache that grows with every token of context.

READ THE DEFINITION ↓MEMORY · CACHE · CONCURRENCY

01 / DEFINITION

The short answer first.

What is private AI infrastructure?

Private AI infrastructure is the compute, memory, storage and network capacity needed to run AI models inside an organisation's own boundary. The binding constraint is almost always GPU memory: model weights plus a key-value cache that grows linearly with context length and concurrency.

WORKED EXAMPLE

Size the infrastructure around a real operating shift

An engineering group plans to serve a local drafting model to several teams. A single-user demonstration runs quickly on the available accelerator. That says little about the Monday morning shift, when several long documents arrive together, embeddings are refreshed and operators need enough spare capacity to restore a failed worker. Infrastructure sizing should reproduce those conditions before the hardware decision is final.

Measure the queue as well as the model

Build a workload from representative input lengths, output lengths and simultaneous users. Record time to the first usable output, total completion time, memory consumption and the number of requests waiting. Compare interactive drafting with a background indexing job. An acceptable average can conceal a queue that grows throughout the shift. Check the slowest useful cases and identify what happens when the system reaches capacity.

Separate the resources used by inference, retrieval and document conversion. If a large import starves the model server, a scheduling limit or separate worker may resolve the problem more directly than a larger model host. Test the same workload with one worker unavailable and measure how quickly service recovers.

Give the operations owner a deployment inventory, approved network destinations, storage growth estimate and a procedure for restoring a known model version. Include the test inputs and measurement conditions so a future hardware change can be compared with the original result. A private infrastructure proposal is much easier to evaluate when it describes the actual workload, failure tolerance and people responsible for operating it.

02 / THE MEMORY ARITHMETIC

Three things occupy GPU memory during inference, and only the first is commonly estimated.

Weights. Roughly two bytes per parameter at 16-bit precision, so an 8B model occupies about 16 GB before anything else. At 8-bit it is about 8 GB, and at 4-bit about 4 GB.

Key-value cache. Grows with every token in context, multiplied by every concurrent request. This is the term that surprises people: a deployment sized comfortably for short prompts can run out of memory as soon as users start pasting long documents, because the cache scales with context length and concurrency together.

Activations and overhead. Working memory for the forward pass, plus framework overhead and fragmentation. A practical planning margin is to leave headroom rather than fitting to the last gigabyte.

The useful way to plan is to fix the context length and concurrency you intend to support, size the cache for that, and treat the result as the real requirement. Sizing on weights alone consistently produces a deployment that works in testing and fails on the first real workload.

Planning dimensions and the effect of each. Quantisation, context length and concurrency are the three levers available once the model is chosen.
LeverEffect on memoryCost of using it
Lower precision (quantisation)Roughly halves weight memory per step downSome quality loss, which must be measured on your task
Shorter context limitReduces cache growth proportionallyLong documents need chunking, which changes retrieval design
Lower concurrencyReduces total cache linearlyQueueing; users wait at peak
Smaller modelReduces everythingCapability, unless the task is narrow enough to fine-tune for

03 / QUANTISATION IS AN EVALUATION QUESTION

Cheaper weights,
measured quality.

Do not accept a quality claim you have not measured.

Quantisation makes a model fit. Whether it still performs is a question for your own held-out set, not for a general benchmark.

Published quantisation comparisons typically report general benchmark scores, which are a poor proxy for a specialised task. Degradation is uneven: it tends to affect long-context reasoning, rare vocabulary and calibrated confidence more than it affects short-form general knowledge — and calibration is exactly the property that decides whether a score can be used for routing.

The right procedure is to run the same held-out evaluation at each precision you are considering, report all four measurement families rather than accuracy alone, and choose deliberately. A model that loses one accuracy point and half its ranking quality has become substantially less useful while appearing almost unchanged. The method is in held-out evaluation and the metrics in LLM evaluation metrics.

04 / WHAT ELSE THE DEPLOYMENT NEEDS

↳ 01

Storage and bandwidth.

Weights, indexes and evaluation sets, plus enough read throughput that model loading and retrieval do not become the bottleneck.

↳ 02

A retrieval tier.

Vector and keyword search run on CPU and RAM, sized separately from inference. Retrieval is frequently the slower half of a RAG system.

↳ 03

Operations.

Driver and firmware compatibility, capacity monitoring, a patching route for an isolated environment, and someone on call for hardware failure.

Two operational details are worth deciding early. Model updates in an isolated environment need a defined import process, with provenance and checksums, because a convenient download path is exactly the outbound route the boundary was built to remove. And capacity monitoring has to include cache pressure, not just utilisation: the failure mode of an undersized deployment is a request that cannot be admitted, which looks like an application error rather than a capacity problem.

05 / QUESTIONS

How much GPU memory does a private LLM need?

Start from about two bytes per parameter at 16-bit precision for the weights, then add key-value cache for your intended context length times concurrency, then leave headroom for activations and fragmentation. The cache term dominates once long documents are in play, and it is the one usually omitted.

Can private AI run on CPU only?

For small models and low volume, yes, with latency measured in seconds per response. It is viable for batch work and generally not for interactive use. Anything with concurrent users wants GPU acceleration.

Does quantisation hurt quality?

Usually a little, sometimes a lot, and unevenly across capabilities. Long-context reasoning and calibration tend to degrade before short-form accuracy does, which matters if the score is used for routing. Measure at each precision on your own held-out set rather than relying on published comparisons.

How many concurrent users can one GPU serve?

It depends on model size, context length and how long responses are. The honest answer comes from a load test at your intended context length; estimates derived from short-prompt benchmarks are consistently optimistic because they understate cache pressure.

What breaks first in a private deployment?

Memory, under long-context load. The second most common is the retrieval tier, which is sized as an afterthought and then becomes the latency bottleneck once the corpus grows.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Share the expected workload, network boundary and operational owner for the private system you are planning.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Share the expected workload, network boundary and operational owner for the private system you are planning.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Size it
for the real workload.

Privilege AI works on local inference and on-premise infrastructure with explicit hardware, network and operational constraints.