Admission control, not silent queueing.
When capacity is exhausted, refuse clearly and quickly. An unbounded queue turns a capacity problem into a timeout cascade.
MKC2 results ↗Downloading a model takes minutes. Serving it reliably to concurrent users, at a context length real documents require, is the engineering.
01 / DEFINITION
The short answer first.An on-premise LLM deployment is a language model served from an organisation's own hardware through an internal API. It consists of the weights, an inference server that batches and schedules requests, and the surrounding integration, monitoring and capacity management.
WORKED EXAMPLE
A publishing team wants an on-premise LLM to tag long manuscripts and return structured metadata. A developer's short prompt fits easily in accelerator memory, but an ordinary manuscript is much longer and several editors may submit work at once. Build the acceptance test around the manuscript workflow before deciding that a particular host is sufficient.
Select representative document lengths, output schemas and concurrency levels. Record the model revision, quantisation, serving runtime, context limits and hardware configuration. Test single requests and overlapping requests. Measure valid completed records per operating period, alongside latency and memory use. A high token rate does not help editors if the service repeatedly rejects malformed metadata or drops distinctions near the end of a manuscript.
Vary the workload deliberately. A burst of short articles should not prevent a long manuscript from finishing, and a background reindex should not make interactive editing unusable. Identify what the queue does when capacity is exhausted. A bounded refusal with a clear retry route may be preferable to uncontrolled memory pressure that takes down every active request.
Stage the new weights, tokenizer and runtime locally. Run the old and new versions against the same held-out manuscripts and compare both extraction accuracy and service performance. Confirm that no component reaches out to download missing files during startup. The operator should be able to identify every artefact from a manifest and verify that it belongs to the approved release.
Restore the previous version after a deliberately failed acceptance test. Check that existing users can finish their work and that the matching tokenizer and prompt configuration were restored with the weights. Keep the measured capacity, failure cases and rollback procedure together. Those records let the organisation distinguish a model-quality problem from a serving or hardware constraint when the next release behaves differently.
Include the model revision and actual context length beside each capacity figure so later comparisons remain interpretable.
02 / THE SERVING LAYER
Running a model in a script and serving it to users are different problems. The serving layer handles concurrency, memory management for the key-value cache, scheduling, streaming, and admission control when capacity is exhausted. The single feature that matters most is continuous batching: new requests join the batch as earlier ones finish, rather than waiting for a whole batch to complete.
The difference is large. Naive per-request execution leaves the GPU idle between requests and under-uses it during them; continuous batching keeps it saturated and multiplies effective throughput for concurrent workloads. Any serving choice should be assessed on whether it does this well before anything else.
| Approach | Suits | Trade-off |
|---|---|---|
| Dedicated inference server with continuous batching | Interactive multi-user services | More moving parts; a real deployment to operate |
| Lightweight local runtime | Single-user tools, desktops, small pilots | Limited concurrency; not a shared service |
| Custom framework code | Research and unusual model architectures | You implement batching and scheduling yourself |
| Batch offline processing | Bulk document review with no latency requirement | No interactive use; throughput optimised instead |
03 / THROUGHPUT AGAINST LATENCY
Larger batches raise total throughput and lengthen the wait for each individual response. The right setting depends on who is waiting.
A supervision model reviewing every document an organisation produces overnight is a throughput workload: batch aggressively, accept per-request latency, maximise documents per hour. An assistant a person is watching is a latency workload: smaller batches, prioritise time to first token, accept lower total throughput. Running both on one deployment with one configuration disappoints both.
Two practical measures follow. Separate the queues, so a bulk job cannot starve interactive requests. And set an explicit context limit rather than allowing the model's maximum: a single 128k-token request can consume the cache budget of many normal ones, which is how an otherwise healthy deployment starts refusing admissions. Chunking long documents in the retrieval layer is usually both cheaper and better for quality — see RAG evaluation.
04 / OPERATING IT
When capacity is exhausted, refuse clearly and quickly. An unbounded queue turns a capacity problem into a timeout cascade.
A process that is alive but cannot allocate cache memory will pass a port check and fail every request.
Weights, quantisation, serving version, decoding parameters. Without them no evaluation result can be attributed to a cause.
One further detail is worth building early: no silent fallback. Resilience features that route to an external provider when the local endpoint fails will, in an environment chosen for its data boundary, quietly void the reason the deployment exists. The correct behaviour on failure is to stop and report. See private AI inference.
05 / QUESTIONS
It lets new requests join an in-flight batch as earlier ones complete, instead of waiting for the whole batch to finish. For concurrent workloads it is the difference between a GPU that is mostly idle and one that is saturated, and it is the first thing to check in any serving stack.
It is a pragmatic choice: client libraries, tooling and application code work unchanged, and it keeps the option of swapping the backend. The compatibility surface is a convention rather than a standard, so pin the behaviour you depend on and test it.
Chunk in the retrieval layer and set an explicit context limit on the serving layer. Very long contexts consume cache memory disproportionately and frequently reduce answer quality, so the maximum supported context is rarely the right operating configuration.
Only with separate queues and different batching settings, and ideally separate capacity. A single configuration optimised for neither leaves interactive users waiting behind bulk work.
Requests and tokens per second, time to first token, cache utilisation and admission rejections, plus GPU memory and temperature. Cache pressure and rejections are the capacity signals that actually predict user-visible failure.
WORK WITH PRIVILEGE AI
Serve a model in-house, properly.
We set up language models to run on your own infrastructure and connect them to your applications.
Planned around your constraints.
Hardware, network and operating needs are worked out up front.
Your data stays with you.
Prompts and documents do not need to leave your environment.
Talk with Privilege AI
Share the model workload, concurrency target and update restrictions for the host you intend to operate.
CONTACT PRIVILEGE AI
Share the model workload, concurrency target and update restrictions for the host you intend to operate.
Privilege AI builds local inference and application integration with explicit hardware, network and operational constraints.