GPU drivers and CUDA
A GPU server is useless until the driver, CUDA and your application agree on versions. Getting that stack right once — and not letting an unattended kernel update break it — is most of the battle.
GPU work depends on a chain of matching versions: the NVIDIA driver in the kernel, the CUDA toolkit, and the framework your application uses. Get one out of step and the card is either invisible or crashing. It is finicky once and then stable — if you protect it.
Install the driver cleanly
Use the packaged driver for your distribution rather than the raw .run installer where you can — it survives updates better. After install, confirm the card is seen:
nvidia-smi
That output — the model, the driver version, the CUDA version it supports — is your ground truth. If it does not appear, nothing above it will work.
The kernel-update trap
The NVIDIA driver is built against the running kernel. An unattended kernel update can leave the driver unbuilt after the next reboot, and suddenly nvidia-smi fails. Use DKMS so the driver rebuilds automatically, and reboot deliberately — see upgrading the kernel safely.
Containers keep versions apart
If you run several GPU applications wanting different CUDA versions, the NVIDIA Container Toolkit lets each run in its own container with its own CUDA, sharing the one host driver. That is far saner than fighting versions on the host. See Docker.
Check it is really on the GPU
A misconfigured framework will quietly fall back to the CPU and run a hundred times slower while looking like it works. Watch nvidia-smi during a job — if GPU utilisation stays at zero, your code is not using the card you are paying for.
Heat is real here
GPUs under load draw serious power and dump serious heat. In our racks that is our problem to cool, but if you colocate your own GPU hardware, say so — see temperature and cooling.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base