Failed to run: Failed to create cgroup at_mnt

Incus
1

Hi, currently following the getting started tutorial. I’m on archlinux.

lion@main :: ~ » incus launch images:ubuntu/22.04 first
Creating first
Starting first                                
Error: Failed to run: /usr/bin/incusd forkstart first /var/lib/incus/containers /var/log/incus/first/lxc.conf: exit status 1

lion@main :: ~ 1 » incus info --show-log local:first
Name: first
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2024/01/12 13:30 CET
Last Used: 2024/01/12 13:30 CET

Log:

lxc first 20240112123033.376 ERROR    cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_mount:2139 - No such file or directory - Failed to create cgroup at_mnt 24()
lxc first 20240112123033.376 ERROR    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:852 - No such file or directory - Failed to mount "/sys/fs/cgroup"
lxc first 20240112123033.376 ERROR    conf - ../src/lxc/conf.c:lxc_setup:4433 - Failed to setup remaining automatic mounts
lxc first 20240112123033.376 ERROR    start - ../src/lxc/start.c:do_start:1272 - Failed to setup container "first"
lxc first 20240112123033.376 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc first 20240112123033.398 WARN     network - ../src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from "eth0" to its initial name "veth9f8f49ef"
lxc first 20240112123033.399 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:878 - Received container state "ABORTING" instead of "RUNNING"
lxc first 20240112123033.399 ERROR    start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "first"
lxc first 20240112123033.399 WARN     start - ../src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 19 for process 5222
lxc 20240112123033.573 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20240112123033.573 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

I’ve set the usermod -v 1000000-1000999999 -w 1000000-1000999999 root already. It refuses to start.

My user is in the incus group. Doing this as root produces the same error.

2

Can you show ls -lh /sys/fs/cgroup?

3 
lion@main :: ~ » sudo ls -lh /sys/fs/cgroup
[sudo] password for root: 
total 0
-r--r--r--  1 root root 0 Jan 12 18:08 cgroup.controllers
-rw-r--r--  1 root root 0 Jan 12 18:08 cgroup.max.depth
-rw-r--r--  1 root root 0 Jan 12 18:08 cgroup.max.descendants
-rw-r--r--  1 root root 0 Jan 12 18:08 cgroup.pressure
-rw-r--r--  1 root root 0 Jan 12 18:08 cgroup.procs
-r--r--r--  1 root root 0 Jan 12 18:08 cgroup.stat
-rw-r--r--  1 root root 0 Jan 12 13:26 cgroup.subtree_control
-rw-r--r--  1 root root 0 Jan 12 18:08 cgroup.threads
-rw-r--r--  1 root root 0 Jan 12 18:08 cpu.pressure
-r--r--r--  1 root root 0 Jan 12 18:08 cpuset.cpus.effective
-r--r--r--  1 root root 0 Jan 12 13:22 cpuset.mems.effective
-r--r--r--  1 root root 0 Jan 12 18:08 cpu.stat
-r--r--r--  1 root root 0 Jan 12 18:08 cpu.stat.local
drwxr-xr-x  2 root root 0 Jan 12 13:21 dev-hugepages.mount
drwxr-xr-x  2 root root 0 Jan 12 13:21 dev-mqueue.mount
drwxr-xr-x  2 root root 0 Jan 12 13:21 init.scope
-rw-r--r--  1 root root 0 Jan 12 18:08 io.cost.model
-rw-r--r--  1 root root 0 Jan 12 18:08 io.cost.qos
-rw-r--r--  1 root root 0 Jan 12 18:08 io.pressure
-rw-r--r--  1 root root 0 Jan 12 18:08 io.prio.class
-r--r--r--  1 root root 0 Jan 12 18:08 io.stat
-rw-r--r--  1 root root 0 Jan 12 18:08 irq.pressure
drwxr-xr-x  2 root root 0 Jan 12 18:08 lxc.pivot
drwxr-xr-x  2 root root 0 Jan 12 13:21 machine.slice
-r--r--r--  1 root root 0 Jan 12 18:08 memory.numa_stat
-rw-r--r--  1 root root 0 Jan 12 18:08 memory.pressure
--w-------  1 root root 0 Jan 12 13:28 memory.reclaim
-r--r--r--  1 root root 0 Jan 12 18:08 memory.stat
-r--r--r--  1 root root 0 Jan 12 18:08 misc.capacity
-r--r--r--  1 root root 0 Jan 12 18:08 misc.current
dr-xr-xr-x  4 root root 0 Jan 12 13:21 net_cls
drwxr-xr-x  2 root root 0 Jan 12 13:21 proc-sys-fs-binfmt_misc.mount
drwxr-xr-x  2 root root 0 Jan 12 13:21 sys-fs-fuse-connections.mount
drwxr-xr-x  2 root root 0 Jan 12 13:21 sys-kernel-config.mount
drwxr-xr-x  2 root root 0 Jan 12 13:21 sys-kernel-debug.mount
drwxr-xr-x  2 root root 0 Jan 12 13:21 sys-kernel-tracing.mount
drwxr-xr-x 47 root root 0 Jan 12 13:44 system.slice
drwxr-xr-x  3 root root 0 Jan 12 13:22 user.slice
4

Can you show cat /proc/self/mountinfo too?

5

Hi, I’m currently facing the same issue on a clean incus install (v0.5-1) on arch.

root@system ~ % cat /proc/self/mountinfo                                                                                                                                                       2024-01-27 20:46:49
22 27 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
23 27 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:6 - sysfs sys rw
24 27 0:5 / /dev rw,nosuid,relatime shared:2 - devtmpfs dev rw,size=6990568k,nr_inodes=1747642,mode=755,inode64
25 27 0:23 / /run rw,nosuid,nodev,relatime shared:12 - tmpfs run rw,mode=755,inode64
26 23 0:24 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:7 - efivarfs efivarfs rw
27 1 254:2 / / rw,relatime shared:1 - ext4 /dev/mapper/arch-root rw
28 23 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
29 24 0:25 / /dev/shm rw,nosuid,nodev shared:3 - tmpfs tmpfs rw,inode64
30 24 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
31 23 0:27 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:9 - cgroup2 cgroup2 rw
32 23 0:28 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:10 - pstore pstore rw
33 23 0:29 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:11 - bpf bpf rw,mode=700
34 22 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=37,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13354
35 23 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:14 - debugfs debugfs rw
36 24 0:31 / /dev/hugepages rw,nosuid,nodev,relatime shared:15 - hugetlbfs hugetlbfs rw,pagesize=2M
37 24 0:19 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:16 - mqueue mqueue rw
38 23 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw
39 23 0:32 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:18 - configfs configfs rw
40 23 0:33 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:19 - fusectl fusectl rw
42 27 0:34 / /tmp rw,nosuid,nodev shared:42 - tmpfs tmpfs rw,nr_inodes=1048576,inode64
44 27 259:1 / /boot rw,relatime shared:44 - vfat /dev/nvme0n1p1 rw,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
46 34 0:35 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:46 - binfmt_misc binfmt_misc rw
91 31 0:46 / /sys/fs/cgroup/net_cls rw,relatime shared:270 - cgroup net_cls rw,net_cls
436 25 0:59 / /run/user/1000 rw,nosuid,nodev,relatime shared:575 - tmpfs tmpfs rw,size=1402844k,nr_inodes=350711,mode=700,uid=1000,gid=1000,inode64
485 436 0:60 / /run/user/1000/gvfs rw,nosuid,nodev,relatime shared:618 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
687 436 0:61 / /run/user/1000/doc rw,nosuid,nodev,relatime shared:636 - fuse.portal portal rw,user_id=1000,group_id=1000
93 27 0:44 / /var/lib/lxcfs rw,nosuid,nodev,relatime shared:258 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
268 27 0:45 / /var/lib/incus/shmounts rw,relatime shared:279 - tmpfs tmpfs rw,size=100k,mode=711,inode64
319 27 0:52 / /var/lib/incus/guestapi rw,relatime shared:291 - tmpfs tmpfs rw,size=100k,mode=755,inode64
6

That’s almost certainly your problem.

We’ve seen this mess in the past with some VPNs like mullvad.
This kind of mixing of cgroup1 on top of cgroup2 isn’t a supportable environment.

Can you try umount -l /sys/fs/cgroup/net_cls and see if that sort it out?

1 Like
7

Unmounting it worked for me, thank you :slight_smile:

8

Hi, I found this thread and can confirm that this was a case on my end too (I have Mullvad installed on Arch Linux).

The fix described here works just fine, although I found out that Mullvad now allows redefining the mountpoint by using TALPID_NET_CLS_MOUNT_DIR environment variable for the mullvad-daemon service [1].

One can set this variable via systemctl edit mullvad-daemon.service (check further docs in the Mullvad repo if needed).

[1] Add info about net_cls env var by pinkisemils · Pull Request #3660 · mullvad/mullvadvpn-app · GitHub