Docker in LXC with GPU not working!

LXD
1

I have a GPU application docker image. Now I want to run that docker container with GPU support inside an LXC container but I am unable to do so.

The way I run is this:

lxc launch ubuntu plex -c nvidia.runtime=true -c nvidia.driver.capabilities=compute,utility
lxc config set plex security.nesting true
lxc exec plex sudo apt-get update
lxc exec plex sudo apt-get install docker.io
lxc exec plex docker run ubuntu ls
lxc config device add plex gpu gpu id=0
lxc exec plex docker run --gpus all ubuntu ls

This last line execution of a docker container with gpus parameter returns this:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
ERRO[0000] error waiting for container: context canceled

I am able to see GPUs when I run nvidia-smi inside the LXC container.

Then I installed nvidia-container-toolkit in the LXC container and then again tried running container with gpus, I get the following error:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: write error: /sys/fs/cgroup/devices/docker/41155b577716bc9b26bf64e1f930f40c5a81dbdd4eeb8d831f42c0b8728b00a5/devices.allow: operation not permitted\\\\n\\\"\"": unknown.

I have even tried setting security.privileged=true on the main LXC container. Then when I try to reboot the container after setting privileged condition, it doesn’t reboot and returns the following error:

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart plex /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/plex/lxc.conf:

I am just scratching my head and still haven’t been able to figure out the solution to make it work. Don’t know what permissions are required for this to work. Any solution will be helpful.

Cheers!

2

Is there anybody here who can help? I guess the post has been ignored by the devs here! This is annoying…

3

Hi!

This is a LXD question and I changed the tag to LXD (this is just housekeeping, not related to receiving an answer quicker or not).

In general, it is an issue when you have container-inside-container (i.e. nesting), and you end up with something that does not work. Especially when you have different implementations.

You mention that you have installed nvidia-container-toolkit in the container. How exactly did you do that? Ideally, LXD should provide the necessary files automatically for you.

The error is on

nvidia-container-cli: mount error: write error: /sys/fs/cgroup/devices/docker/41155b577716bc9b26bf64e1f930f40c5a81dbdd4eeb8d831f42c0b8728b00a5/devices.allow: operation not permitted\\\\n\\\"\"": unknown.

As a first step, can you create the plex container as a privileged container and check whether it works?

4

Thanks @simos for adding in your response.

I installed nvidia-container-toolkit following this nvidia guide for Ubuntu: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html

If I don’t add the nvidia-container-toolkit then it displays this error while running a docker container with GPUs:

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
ERRO[0000] error waiting for container: context canceled

I ran a couple of more experiments as suggested, below is the outcome on both of them:

1st experiment:
Added security and nvidia parameters during launch:

lxc launch ubuntu plex -c security.nesting=true -c security.privileged=true -c nvidia.runtime=true -c nvidia.driver.capabilities=compute,utility

Got this output:

Creating plex

The local image 'ubuntu' couldn't be found, trying 'ubuntu:' instead.

Starting plex

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart plex /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/plex/lxc.conf

This failed to start during the launch itself.

2nd experiment:
Launched container with only security parameters and then added nvidia config after that as follows:

lxc launch ubuntu plex -c security.nesting=true -c security.privileged=true
lxc config set plex nvidia.runtime true
lxc config set plex nvidia.driver.capabilities compute
lxc config device add plex gpu gpu id=0
lxc exec plex nvidia-smi

On running nvidia-smi in the LXC container, it asked me to install nvidia driver:

Command 'nvidia-smi' not found, but can be installed with:

apt install nvidia-utils-390         # version 390.138-0ubuntu0.20.04.1, or
apt install nvidia-utils-440         # version 440.100-0ubuntu0.20.04.1
apt install nvidia-utils-450         # version 450.80.02-0ubuntu0.20.04.2
apt install nvidia-utils-450-server  # version 450.80.02-0ubuntu0.20.04.3
apt install nvidia-340               # version 340.108-0ubuntu2
apt install nvidia-utils-435         # version 435.21-0ubuntu7
apt install nvidia-utils-418-server  # version 418.152.00-0ubuntu0.20.04.1
apt install nvidia-utils-440-server  # version 440.95.01-0ubuntu0.20.04.1
apt install nvidia-utils-455         # version 455.38-0ubuntu0.20.04.1

I installed as follows in the LXC container:

sudo apt-get update
sudo apt install nvidia-utils-450

On running nvidia-smi again, it generated the following error:

Failed to initialize NVML: Unknown Error

Then I rebooted the lxc container to make sure the driver is installed properly, it then failed to reboot and then when I run lxc start plex, it displayed the same reboot error:

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart plex /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/plex/lxc.conf:

When I remove security privilege setting then the LXC container is able to start again. But then on running nvidia-smi command in the LXC container, I get this error:

Failed to initialize NVML: Driver/library version mismatch

Also, docker with GPUs doesn’t work then.

5

You shouldn’t install the NVIDIA tools in the container, that’s what’s causing the mismatch.

I suspect that the runtime option you’ve set is limiting what’s being exposed to the container a bit too much and is causing the nvidia-smi breakage.

We daily test normal NVIDIA passthrough and validate that smi works fine.

As for Docker, no idea what’s needed there. It may be that nvidia-docker would need modifications to work in such an environment.

6

In the first experiment, you used ubuntu as the container image name, which means that LXD would try to find the image local:ubuntu. Such an image did not exist, therefore LXD helped you here, and looked for ubuntu: (the ubuntu remote, and the default image), which means you are launching ubuntu:20.04.

You get a failure to start the container, and the message is a generic one. There should be more info in the LXD logs, and this part should be investigated. Further down your reply, you get the same error. Which hints that there some other issue that gets into the way.

When you enable nvidia.runtime, at least in a LXD profile, then LXD mounts for you the NVidia container image (which is also auto-downloaded by LXD). When I create such a container, I use a LXD profile. And in the created container, I get nvidia-smialready installed and available as soon as I get a shell into the container.

In the second experiment, you install manually an NVidia driver and you get a driver mismatch with the host.

I suggest to first add the nvidia lines to a LXD profile, then launch a container, and make sure that nvidia-smi works as soon as you get a shell into that container.

7

Thanks @stgraber and @simos for the response.

As mentioned by @stgraber I didn’t add nvidia runtime driver capabilities option this time and only passed the nvidia.runtime as follows:

lxc launch ubuntu plex -c security.nesting=true -c security.privileged=true -c nvidia.runtime=true

But again it failed to start and generated the forkstart error.

@simos then I ran the command lxc info --show-log local:plex to get the logs. Please find them below:

Name: plex
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/12/22 16:28 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc plex 20201222162802.463 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.plex"
lxc plex 20201222162802.465 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.plex"
lxc plex 20201222162802.573 ERROR    conf - conf.c:run_buffer:324 - Script exited with status 1
lxc plex 20201222162802.573 ERROR    conf - conf.c:lxc_setup:3374 - Failed to run mount hooks
lxc plex 20201222162802.573 ERROR    start - start.c:do_start:1218 - Failed to setup container "plex"
lxc plex 20201222162802.573 ERROR    sync - sync.c:__sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc plex 20201222162802.575 WARN     network - network.c:lxc_delete_network_priv:3185 - Failed to rename interface with index 0 from "eth0" to its initial name "veth36589e8c"
lxc plex 20201222162802.575 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:860 - Received container state "ABORTING" instead of "RUNNING"
lxc plex 20201222162802.575 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "plex"
lxc plex 20201222162802.575 WARN     start - start.c:lxc_abort:1013 - No such process - Failed to send SIGKILL via pidfd 31 for process 40894
lxc 20201222162802.725 WARN     commands - commands.c:lxc_cmd_rsp_recv:126 - Connection reset by peer - Failed to receive response for command "get_state"

This is not the case when I don’t add privileged option. If I run the container as follows:

lxc launch ubuntu plex -c security.nesting=true -c nvidia.runtime=true

Then it starts normally and I am able to see GPU inside the container with nvidia-smi command.

The moment I add privileged setting and then reboot to configure it I get the forkstart error.

@simos when you create the container through your LXD profile that shows nvidia-smi, is it privileged? Or is it able to run a docker container with GPUs inside it?

8

The container is unprivileged. Here is the baseline in creating a container with a working NVidia.

$ lxc create profile nvidia
...
$ lxc profile show nvidia
config:
  nvidia.driver.capabilities: all
  nvidia.runtime: "true"
description: ""
devices:
  mygpu:
    type: gpu
name: nvidia
used_by:
$ lxc launch ubuntu:20.04 --profile default --profile nvidia mycontainer
Creating mycontainer
Starting mycontainer
$ lxc shell mycontainer
root@ubuntu:~# nvidia-smi 
...
All OK
...
root@ubuntu:~#

From there on, you can try to setup Docker in the container. Add the nesting support when creating the container, obviously.

9

security.privileged is currently incompatible with the integration scripts provided by nvidia-container. Upstream was telling us there isn’t a particular reason it’s not working but they haven’t spent the time to make it work with such containers.

10

Thanks @simos and @stgraber for the response.

@simos the profile you shared worked for accepting GPU but wasn’t able to run nested docker containers with GPUs. Privilege mode is needed for that. But again privilege mode is incompatible with nvidia runtime capabilities and the container simply crashes on restart.

Have tried a lot and given enough time. I’ll let it be like this for now… will try again some other experiments in future maybe.

11

Just an update from my further deep-dive on it.

@simos and @stgraber I have figured out that nvidia.runtime and security.privileged mode don’t work together properly in LXD. It always ends up creating forkstart error on reboot. However this combination of runtime and security privilege works in Docker (if I had to run docker with GPU inside docker). That’s the only part causing problem.

But for my situation I needed nvidia runtime to work as I can’t be installing the GPU drivers repeatedly.

If there is a workaround then I’d be grateful to know about it. Cheers!

12

@stgraber and @simos any developments possible on this? To make a container have both nvidia runtime and security privilege at the same time?

13

It is not up to us, the limitation is coming from the LXC hook that NVIDIA themselves wrote/maintain.

14

In that case I think that the privilege mode of LXC and docker containers work differently.

Because I am able to run a docker container in privileged mode with GPU runtime. But on passing the nvidia runtime to LXC container with privilege mode it results in forkstart error.

Also, since I am in my early days of LXC, would you recommend any other way of providing privileges to LXC container other than security privilege setting? Maybe I can try that work around (if any).

15

Yes, Docker privileged and LXC privileged are not at all the same thing.

Docker is effectively always privileged in that containers do not use the user namespace, the privileged flag then makes things further unsafe by disabling a number of restrictions put in place to avoid container escape.

In LXD, all containers are unprivileged by default (use the user namespace), the privileged flag can be used to make them not use the user namespace and so get them to an equivalent of the default Docker (without privileged option).

I’m not sure what’s the best option in your case.
If you’re just trying to use Plex with NVIDIA based transcoding, then maybe your best option is to just install Plex directly in the LXD container rather than through a Docker package?

nvidia-docker in an unprivileged LXD container is apparently failing due to incorrect handling of devices cgroup permissions and possibly missing logic to bind-mount the needed device nodes into the Docker container.

nvidia-lxc is then failing when run inside a privileged LXD container.

In both cases it comes back to issues with nvidia-container, once in Docker and once in LXC but situations like this are enough of a corner case that I wouldn’t expect much energy being spent on this by NVIDIA.

In theory an alternative is to run a privileged LXD container without nvidia.runtime, only using the GPU passthrough, then installing the matching nvidia drivers and libraries in the container.

16

Thanks @stgraber for adding in further context on this. It’s pretty helpful.

With the GPU driver definitely it is working as expected but I was trying to get it started with runtime due to a particular solution that I am working on.

I also explored the uid and gid custom mapping as an attempt to add privileges and I bumped up the available mapping as you mentioned in your blog here: https://stgraber.org/2017/06/15/custom-user-mappings-in-lxd-containers/

But that again resulted in forkstart error on restart. I guess I’ll have to stick with the non runtime option only.

Anyways, thanks for the support :slight_smile: