Cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:332 - Operation not permitted

LXD
1

I’m trying out unified cgroups (e.g. no device cgroup). I’m running a self compiled 5.10 kernel with LXD 4.0.9 (also self compiled). (for reasons ™). User namespace is not enabled in this kernel.

When I try to start a privileged container that has a unix-char device forwarded to it, the container fails to start (does not get an IP). The log shows:

$ lxc info --show-log deb11
Name: deb11
Location: none
Remote: unix://
Architecture: x86_64
Created: 2022/06/20 13:51 CEST
Status: Running
Type: container
Profiles: default, kmx_latencydev_pf
Pid: 24324
Ips:
  eth0:	inet6	fe80::216:3eff:fe3b:f9d1	vethed5469fd
  lo:	inet	127.0.0.1
  lo:	inet6	::1
Resources:
  Processes: 1
  Network usage:
    eth0:
      Bytes received: 260B
      Bytes sent: 656B
      Packets received: 2
      Packets sent: 8
    lo:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0

Log:

lxc deb11 20220620121909.989 ERROR    utils - utils.c:open_devnull:1232 - Operation not permitted - Can't open /dev/null
lxc deb11 20220620121909.989 WARN     start - start.c:do_start:1394 - Failed to ajust stdio permissions

The containers configuration:

$ lxc config show -e deb11
architecture: x86_64
config:
  boot.autostart: "false"
  image.architecture: amd64
  image.description: Debian bullseye amd64 (20220620_05:25)
  image.os: Debian
  image.release: bullseye
  image.serial: "20220620_05:25"
  image.type: squashfs
  image.variant: default
  security.privileged: "true"
  volatile.base_image: 0e74bd2b1cd760553cc80c902c88fbf9ed0091461deac01bd2ef6bf491eed692
  volatile.eth0.host_name: vethed5469fd
  volatile.eth0.hwaddr: 00:16:3e:3b:f9:d1
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 6e631b5b-c6f6-4159-b114-2054c9868632
devices:
  cpu_dma_latency:
    gid: "997"
    path: /dev/cpu_dma_latency
    type: unix-char
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
- kmx_latencydev_pf
stateful: false
description: ""

When I remove the unix-char device, the container starts alright.

If I now start a 2nd container that has the same unix-char device forwarded to it, the container starts but I get an error inside the log:

$ lxc init -c security.privileged=true -c boot.autostart=false images:debian/11 deb11-2nd
Creating deb11-2nd

$ lxc profile add deb11-2nd kmx_latencydev_pf 
Profile kmx_latencydev_pf added to deb11-2nd

$ lxc start deb11-2nd 

$ lxc info --show-log deb11-2nd                                                          
Name: deb11-2nd
Location: none
Remote: unix://
Architecture: x86_64
Created: 2022/06/20 14:25 CEST
Status: Running
Type: container
Profiles: default, kmx_latencydev_pf
Pid: 25915
Ips:
  lo:	inet	127.0.0.1
  lo:	inet6	::1
  eth0:	inet	10.0.4.65	vetha8a7a62a
  eth0:	inet6	fe80::216:3eff:fe02:78b2	vetha8a7a62a
Resources:
  Processes: 1
  Network usage:
    lo:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
    eth0:
      Bytes received: 742B
      Bytes sent: 2.32kB
      Packets received: 3
      Packets sent: 23

Log:

lxc deb11-2nd 20220620122522.944 ERROR    cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:332 - Operation not permitted - Failed to load bpf program: (null)


$ lxc profile show kmx_latencydev_pf 
config: {}
description: add cpu_dma latency control to container
devices:
  cpu_dma_latency:
    gid: "997"
    path: /dev/cpu_dma_latency
    type: unix-char
name: kmx_latencydev_pf
used_by:
- /1.0/instances/deb11-2nd
2

Are there any dependencies between LXD versions and systemd versions? I compiled 4.0.9 and sprinkled Printfs all over the place (systemd is 249.11-0ubuntu3.1, Ubuntu 22.04)

LXD seems to think that I only have the “pid” cgroup controller, even though:

$ cat /sys/fs/cgroup/cgroup.controllers 
cpuset cpu io pids

LXD seems to look at /proc/self/cgroup and then the last entry. It finds that only the pid controller is available:

$ cat /proc/self/cgroup 
0::/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-785e94cb-4c8d-41bc-ad71-4fcddbe89b85.scope
$ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-785e94cb-4c8d-41bc-ad71-4fcddbe89b85.scope/cgroup.controllers
pids

I guess I’ll give it another try tomorrow to find out why a container with a unix-char device does not start up.

3

I got it running. It seems LXD thinks that there is no support for the device cgroup (=> ebpf in unified scenario). This causes the container to fail to start up if I have a unix-char device forwarded to it (I have no idea why).

LXD thinks that there is no support for eBPF because the relevant query for “cgroup2” API extension returns false. This is caused by go-lxc bindings failing to handle the “~git” part in liblxc5. Fixing this in my go-lxc bindings also fixes the problem I’m seeing.

The function runtimeLiblxcVersionAtLeast seems to be duplicated inside LXD and go-lxc bindings. Only LXD carries a fix for the version of liblxc5 in Ubuntu 22.04.

4

Can you open a bug report please on github?

5
1 Like