NVIDIA CUDA Issues

[RossMadness]

Hello there.

I’m running into an issue on Ubuntu 24.04 with an NVIDIA GPU. I’ve created an Ubuntu container and added the setting for nvidia.runtime=true, as well as adding the gpu device.

The instance sees the gpu in /dev/ and I am able to successfully run nvidia-smi within the instance and get the correct driver and CUDA version in the output.

However, processes within the instance can’t utilize cuda. I’ve tried to do some digging and found this previous forum post:

Nvidia & gpu seems not complete

I noticed that setting gpu delegation to container via gpu and nvidia.runtime: "true" is not complete as it lacks the directory /proc/driver/nvidia/gpu mount from the host. moreover when i added this mount via: nvdock1: path: /proc/driver/nvidia/gpus source: /proc/driver/nvidia/gpus type: disk cuda started to work correctly but after restart of the guest it seems that this disk is attempted to be mounted before the essential part of the gpu/nvidia delegation is run, so in result …

Looking at the path /proc/driver/nvidia/ shows the gpus folder is missing just like the OP’s scenario in the other post. I attempted to mount the path in the same way using

 path: /proc/driver/nvidia/gpus
    source: /proc/driver/nvidia/gpus
    type: disk

However the gpus folder doesn’t appear. I tried making the folder and it doesn’t populate with the same contents as on the host.

Has there been any progress to resolving this issue? I’ve probably missed a step somewhere, but I’ve been going over every scrap of documentation or forum posts that I can find. Any help is much appreciated.

[turtle0x1]

You should install cuda inside the container if you want to use nvcc ETC (whisper.cpp, llama.cpp, ETC), the exact error / problem your getting would be helpful.