Nested LXD instance not getting GPU (/dev/dri) - The "major" property may not be set when adding a device to a nested container

LXD
1

lxc instance in host gets GPU and acceleration in headless xrdp works.
But the nested lxc doesn’t accept GPU from its parent lxc instance:

Error: Failed preparing container for start: Failed to start device “gpu0”: The “major” property may not be set when adding a device to a nested container

I set lxc config set docker security.nesting=true on host’s instance ‘inst1’ and on ‘docker’ inside lxc instance.

In Lxc instance ‘inst1’:
root@inst1:~# ls -all /dev/dri
total 0
drwxr-xr-x 2 root root 80 Mar 29 13:30 .
drwxr-xr-x 10 root root 540 Mar 29 13:32 …
crw-rw---- 1 root root 226, 0 Mar 29 13:30 card0
crw-rw---- 1 root root 226, 128 Mar 29 13:30 renderD128

root@inst1:~# lxc info --resources
…
GPU:
NUMA node: 0
Vendor: VMware (15ad)
Product: SVGA II Adapter (0405)
PCI address: 0000:00:0f.0
Driver: vmwgfx (2.15.0.0)
DRM:
ID: 0
Card: card0 (226:0)
Control: controlD64 (226:0)
Render: renderD128 (226:128)

root@inst1:~# lxc list
±-------±--------±---------------------±-----±----------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±-------±--------±---------------------±-----±----------±----------+
| docker | RUNNING | 172.17.0.1 (docker0) | | CONTAINER | 0 |
| | | 10.25.151.160 (eth0) | | | |
±-------±--------±---------------------±-----±----------±----------+
root@inst1:~# lxc stop docker
root@inst1:~# lxc config device add docker gpu0 gpu
Device gpu0 added to docker
root@inst1:~# lxc start docker
Error: Failed preparing container for start: Failed to start device “gpu0”: The “major” property may not be set when adding a device to a nested container
Try lxc info --show-log docker for more info
root@inst1:~#

/dev/dri in nested docker:

root@inst1:~# docker run --device=/dev/dri -it dockernested /bin/bash
root@324e45f33efc:/# ls -all /dev/dri
total 0
drwxr-xr-x 2 root root 80 Mar 29 21:05 .
drwxr-xr-x 6 root root 380 Mar 29 21:05 …
crw-rw---- 1 root root 226, 0 Mar 29 13:30 card0
crw-rw---- 1 root root 226, 128 Mar 29 13:30 renderD128
root@324e45f33efc:/#

2

Yeah, that’s going to to need some pretty specific extra logic to handle properly.
It may be easier for you to just use unix-char devices for each of the entries under /dev that you care about.

Basically for GPUs, LXD normally creates a new copy of the char device, this cannot be done inside a container. Instead we’d need to track down the device and bind-mount it, but this can come with complications around permissions.