Running a local LLM
You do not have to send every prompt to someone else’s API. On a server with a capable GPU, Ollama runs open-weight language models locally — your data stays on your hardware, and there is no per-token meter running.
Ollama makes running an open-weight model about as simple as ollama run. The hard limits are not software — they are the GPU’s memory and your expectations. Get those right and a dedicated GPU server becomes a private inference endpoint.
VRAM decides which models fit
A model must fit in GPU memory to run fast. Roughly, a quantised model needs a bit more VRAM than its file size. A 24 GB card comfortably runs models up to the mid-tens of billions of parameters quantised; larger ones spill to system RAM and slow to a crawl. Pick the model to the card, not the other way round. See GPU drivers and CUDA.
Quantisation is the lever
Quantised models trade a little quality for a lot less memory. A 4-bit version of a model runs where the full-precision one would not fit at all, and for most tasks the difference is hard to notice. Start quantised and only go bigger if the quality is not there.
It speaks an API
Ollama exposes an HTTP API, so your own applications can call it exactly like a hosted one — without the data ever leaving the box. Keep that endpoint on your internal network or behind authentication; an open inference API on the public internet will be found and used. See a firewall that does not lock you out.
Cold starts and keeping models warm
Loading a large model into VRAM takes seconds. Ollama keeps a model resident for a while after use; if you need instant responses, keep it warm rather than letting it unload between requests.
Why run it here
Privacy and cost. Prompts and documents that cannot go to a third-party API can go to a model on your own server, and a GPU you rent by the month has no per-token bill. See do you need a GPU server?.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base