Ubuntu 22.04 LXD fails to launch container with mullvad VPN

LXD
1

Host: Ubuntu 22.04

$ sudo snap --version
snap 2.56.2+22.04ubuntu1
snapd 2.56.2+22.04ubuntu1
series 16
ubuntu 22.04
kernel 5.15.0-46-generic

$ lxd --version
5.4

$ lxc launch ubuntu:focal node1
Creating node1
Starting node1
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart node1 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/node1/lxc.conf:
Try lxc info --show-log local:node1 for more info

$ lxc info --show-log local:node1
Name: node1
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/08/18 12:29 EDT
Last Used: 2022/08/18 12:29 EDT

Log:

lxc node1 20220818162914.531 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc node1 20220818162914.531 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc node1 20220818162914.532 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc node1 20220818162914.532 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc node1 20220818162914.616 ERROR cgfsng - …/src/src/lxc/cgroups/cgfsng.c:cgfsng_mount:2131 - No such file or directory - Failed to create cgroup at_mnt 24()
lxc node1 20220818162914.616 ERROR conf - …/src/src/lxc/conf.c:lxc_mount_auto_mounts:851 - No such file or directory - Failed to mount “/sys/fs/cgroup”
lxc node1 20220818162914.616 ERROR conf - …/src/src/lxc/conf.c:lxc_setup:4396 - Failed to setup remaining automatic mounts
lxc node1 20220818162914.616 ERROR start - …/src/src/lxc/start.c:do_start:1272 - Failed to setup container “node1”
lxc node1 20220818162914.616 ERROR sync - …/src/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc node1 20220818162914.619 WARN network - …/src/src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from “eth0” to its initial name “veth917cfd30”
lxc node1 20220818162914.619 ERROR lxccontainer - …/src/src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state “ABORTING” instead of “RUNNING”
lxc node1 20220818162914.619 ERROR start - …/src/src/lxc/start.c:__lxc_start:2107 - Failed to spawn container “node1”
lxc node1 20220818162914.619 WARN start - …/src/src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 19 for process 4187
lxc node1 20220818162919.727 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc node1 20220818162919.727 WARN conf - …/src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc 20220818162919.760 ERROR af_unix - …/src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220818162919.760 ERROR commands - …/src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command “get_state”

I just upgrade this system from 20.04 to 22.04

Any ideas how to fix this?

Brian

2

Same thing happens if I try to create 22.04 LXD container…

$ lxc launch ubuntu:22.04 cn1
Creating cn1
Starting cn1
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart cn1 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/cn1/lxc.conf:
Try lxc info --show-log local:cn1 for more info

3

Can you show cat /proc/self/mountinfo ?

4

@stgraber

$ cat /proc/self/mountinfo

24 29 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw
25 29 0:23 / /proc rw,nosuid,nodev,noexec,relatime shared:12 - proc proc rw
26 29 0:5 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=32843688k,nr_inodes=8210922,mode=755,inode64
27 26 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
28 29 0:25 / /run rw,nosuid,nodev,noexec,relatime shared:5 - tmpfs tmpfs rw,size=6575800k,mode=755,inode64
29 1 8:5 / / rw,relatime shared:1 - ext4 /dev/sda5 rw,errors=remount-ro
30 24 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
31 26 0:26 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw,inode64
32 28 0:27 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k,inode64
33 24 0:28 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:9 - cgroup2 cgroup2 rw
34 24 0:29 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:10 - pstore pstore rw
35 24 0:30 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:11 - bpf bpf rw,mode=700
36 25 0:31 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=31900
37 26 0:32 / /dev/hugepages rw,relatime shared:14 - hugetlbfs hugetlbfs rw,pagesize=2M
38 26 0:20 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:15 - mqueue mqueue rw
39 24 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:16 - debugfs debugfs rw
40 24 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw
41 24 0:33 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:18 - fusectl fusectl rw
42 24 0:21 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:19 - configfs configfs rw
64 28 0:34 / /run/credentials/systemd-sysusers.service ro,nosuid,nodev,noexec,relatime shared:20 - ramfs none rw,mode=700
43 29 7:0 / /snap/bare/5 ro,nodev,relatime shared:21 - squashfs /dev/loop0 ro,errors=continue
44 29 7:1 / /snap/bitwarden/74 ro,nodev,relatime shared:22 - squashfs /dev/loop1 ro,errors=continue
45 29 7:2 / /snap/chromium/2051 ro,nodev,relatime shared:23 - squashfs /dev/loop2 ro,errors=continue
46 29 7:3 / /snap/core/13425 ro,nodev,relatime shared:24 - squashfs /dev/loop3 ro,errors=continue
47 29 7:4 / /snap/core18/2538 ro,nodev,relatime shared:25 - squashfs /dev/loop4 ro,errors=continue
48 29 7:5 / /snap/core20/1593 ro,nodev,relatime shared:26 - squashfs /dev/loop5 ro,errors=continue
49 29 7:6 / /snap/core20/1611 ro,nodev,relatime shared:27 - squashfs /dev/loop6 ro,errors=continue
50 29 7:7 / /snap/firefox/1635 ro,nodev,relatime shared:28 - squashfs /dev/loop7 ro,errors=continue
51 29 7:8 / /snap/firefox/1670 ro,nodev,relatime shared:29 - squashfs /dev/loop8 ro,errors=continue
52 29 7:9 / /snap/gnome-3-28-1804/161 ro,nodev,relatime shared:30 - squashfs /dev/loop9 ro,errors=continue
53 29 7:10 / /snap/gnome-3-38-2004/112 ro,nodev,relatime shared:31 - squashfs /dev/loop10 ro,errors=continue
54 29 7:11 / /snap/gtk-common-themes/1535 ro,nodev,relatime shared:32 - squashfs /dev/loop11 ro,errors=continue
55 29 7:12 / /snap/lxd/23367 ro,nodev,relatime shared:33 - squashfs /dev/loop12 ro,errors=continue
56 29 7:13 / /snap/plexmediaserver/307 ro,nodev,relatime shared:34 - squashfs /dev/loop13 ro,errors=continue
57 29 7:14 / /snap/plexmediaserver/311 ro,nodev,relatime shared:35 - squashfs /dev/loop14 ro,errors=continue
58 29 7:15 / /snap/snap-store/582 ro,nodev,relatime shared:36 - squashfs /dev/loop15 ro,errors=continue
135 29 7:16 / /snap/snapd/16292 ro,nodev,relatime shared:37 - squashfs /dev/loop16 ro,errors=continue
139 29 7:17 / /snap/snapd-desktop-integration/14 ro,nodev,relatime shared:38 - squashfs /dev/loop17 ro,errors=continue
142 29 8:2 / /boot/efi rw,relatime shared:48 - vfat /dev/sda2 rw,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
1256 28 0:25 /snapd/ns /run/snapd/ns rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,size=6575800k,mode=755,inode64
1298 1256 0:4 mnt:[4026532635] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
1299 1256 0:4 mnt:[4026532636] /run/snapd/ns/plexmediaserver.mnt rw - nsfs nsfs rw
2184 1256 0:4 mnt:[4026532903] /run/snapd/ns/snapd-desktop-integration.mnt rw - nsfs nsfs rw
2586 33 0:79 / /sys/fs/cgroup/net_cls rw,relatime shared:1045 - cgroup net_cls rw,net_cls
842 28 0:48 / /run/user/1000 rw,nosuid,nodev,relatime shared:499 - tmpfs tmpfs rw,size=6575796k,nr_inodes=1643949,mode=700,uid=1000,gid=1000,inode64
1116 842 0:49 / /run/user/1000/gvfs rw,nosuid,nodev,relatime shared:519 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
1157 842 0:75 / /run/user/1000/doc rw,nosuid,nodev,relatime shared:698 - fuse.portal portal rw,user_id=1000,group_id=1000
899 29 8:1 / /media/bmullan/bcd721a1-a930-48cb-af57-e0a1de560989 rw,nosuid,nodev,relatime shared:503 - ext4 /dev/sda1 rw,errors=remount-ro
1377 29 8:48 / /media/bmullan/SSD-480GB rw,nosuid,nodev,relatime shared:772 - ext4 /dev/sdd rw,errors=remount-ro
2306 29 0:60 / /media/bmullan/default rw,nosuid,nodev,relatime shared:949 - btrfs /dev/disk/by-uuid/4609be62-0c29-4087-a365-2fd442de147a rw,ssd,space_cache,subvolid=5,subvol=/
3082 29 8:17 / /media/bmullan/643f9ad5-bf0f-49f2-b792-52e2cc2bd424 rw,nosuid,nodev,relatime shared:1080 - ext4 /dev/sdb1 rw,errors=remount-ro
3555 1256 0:4 mnt:[4026532991] /run/snapd/ns/firefox.mnt rw - nsfs nsfs rw
1156 29 7:18 / /snap/chromium/2064 ro,nodev,relatime shared:500 - squashfs /dev/loop18 ro,errors=continue
4048 1256 0:4 mnt:[4026534249] /run/snapd/ns/chromium.mnt rw - nsfs nsfs rw
4350 29 0:125 / /var/snap/lxd/common/ns rw,relatime - tmpfs tmpfs rw,size=1024k,mode=700,inode64
4423 4350 0:4 mnt:[4026534479] /var/snap/lxd/common/ns/shmounts rw - nsfs nsfs rw
4326 4350 0:4 mnt:[4026532635] /var/snap/lxd/common/ns/mntns rw - nsfs nsfs rw
4426 36 0:141 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:797 - binfmt_misc binfmt_misc rw

5

anyone have any ideas what might be causing this ?

thanks

6 
 2586 33 0:79 / /sys/fs/cgroup/net_cls rw,relatime shared:1045 - cgroup net_cls rw,net_cls

That’s the issue.

We’ve seen it occasionally but haven’t yet found the root cause for this.
It’s a cgroup1 controller mounted on top of a cgroup2 tree, that’s a very bad idea and causes a bunch of issues.

A temporary fix is to run umount -l /sys/fs/cgroup/net_cls but that will not survive a reboot.

7

Oh man…

Thanks Stephanie.

I’ll use the workaround till it gets figured out.

Is there a Bug ID I can watch?

Brian

8

I got tired of having to remember to run this everytime I restart my system:

umount -l /sys/fs/cgroup/net_cls

So I reinstalled 22.04.1 from scratch and after “sudo lxd init” I tried again
to create a new container:

lxc launch ubuntu:22.04 cn1

But the problem

 2586 33 0:79 / /sys/fs/cgroup/net_cls rw,relatime shared:1045 - cgroup net_cls rw,net_cls

is still there?

If this is “cgroup1 controller mounted on top of a cgroup2 tree”

Is there any Bug filed about this that I can follow??

thanks
brian

9

See HELP ! HELP ! HELP ! Cgroup2 related issue on ubuntu jammy - #14 by tomp