Serving model inference
Training a model is a project; serving it is an operation. Turning a trained model into a fast, reliable endpoint is its own discipline — batching, memory, and keeping the GPU busy without falling over.
Inference is the part that runs forever: every request a user makes hits the model. Unlike training, it is latency-sensitive and must stay up. A Python script in a loop is not a serving strategy.
Use a serving runtime
Dedicated runtimes — vLLM, TGI, Triton and the like — handle the hard parts: loading the model once, managing GPU memory, and batching requests. They exist because the naive approach wastes most of the card. Start with one rather than building your own.
Batching is where the throughput is
A GPU is most efficient processing many inputs at once. Serving runtimes batch incoming requests automatically, trading a few milliseconds of latency for several times the throughput. Tune the batch window to your latency budget — tighter for interactive, looser for bulk.
Memory is the ceiling
The model weights plus the working memory per request must fit in VRAM. Too many concurrent requests and you hit out-of-memory errors under exactly the load you most wanted to serve. Know your per-request memory and cap concurrency below the cliff — see sharing a GPU.
Scaling is not instant
You cannot spin a GPU up in a second the way you can a web worker — loading a model takes time and GPUs are a fixed resource. Plan capacity for your peak rather than assuming you can autoscale out of it. This is a place where knowing your capacity matters.
Watch latency and GPU use together
Monitor request latency alongside GPU utilisation and memory. Rising latency with the GPU not yet full means a batching or CPU bottleneck; rising latency with the GPU full means you are out of capacity. See monitoring your own server.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base