Cannot start containers on Arch after upgrade to lxc 1:4.0.11-2 with LTS kernel

lyderic · October 28, 2021, 6:52am

Hi,

After the last upgrade of the ‘lxc’ package on Arch linux (from 1:4.0.10-2 to 1:4.0.11-2), lxd doesn’t start the containers anymore (i.e. lxd start fails).

The problem is encountered only on Arch machines with the LTS kernel, the machines with standard kernel are not affected.

Versions:

⏹ [lyderic@nuc ~]$ pacman -Q lxc lxd linux-lts
lxc 1:4.0.11-2
lxd 4.19-1
linux-lts 5.10.75-1

Reproduce:

⏹ [lyderic@nuc ~]$ lxc list bob
+------+---------+------+------+-----------+-----------+
| NAME |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+------+---------+------+------+-----------+-----------+
| bob  | STOPPED |      |      | CONTAINER | 4         |
+------+---------+------+------+-----------+-----------+
⏹ [lyderic@nuc ~]$ lxc start bob
Error: Failed to run: /usr/bin/lxd forkstart bob /var/lib/lxd/containers /var/log/lxd/bob/lxc.conf: 
Try `lxc info --show-log bob` for more info
⏹ [lyderic@nuc ~]$ lxc info --show-log bob
Name: bob
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2021/06/29 08:34 BST
Last Used: 2021/10/28 07:36 BST

Snapshots:
[...]

Log:

lxc bob 20211028063616.909 ERROR    start - start.c:core_scheduling:1575 - Invalid argument - The kernel does not support core scheduling
lxc bob 20211028063616.910 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state "ABORTING" instead of "RUNNING"
lxc bob 20211028063616.911 ERROR    start - start.c:__lxc_start:2073 - Failed to spawn container "bob"
lxc bob 20211028063616.911 WARN     start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 17 for process 998
lxc 20211028063621.989 ERROR    af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20211028063621.989 ERROR    commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors

Please note that the above works perfectly on a machine that has a standard kernel, i.e. a machine with the following versions:

$ pacman -Q lxc lxd linux
lxc 1:4.0.11-2
lxd 4.19-1
linux 5.14.14.arch1-1

Many thanks for looking into it. For the moment, to make it work, I downgraded the lxc package as follows:

⏹ [lyderic@nuc ~]$ sudo pacman -U "/var/cache/pacman/pkg/lxc-1:4.0.10-2-x86_64.pkg.tar.zst"
loading packages...
warning: downgrading package lxc (1:4.0.11-2 => 1:4.0.10-2)
[...]
⏹ [lyderic@nuc ~]$ pacman -Q lxc lxd linux-lts
lxc 1:4.0.10-2
lxd 4.19-1
linux-lts 5.10.75-1

lyderic · October 28, 2021, 6:59am

Please note that this problem is reported as being fixed here. However, it seems to apply only to the standard kernel, not the LTS kernel.

Many thanks.

lyderic · October 28, 2021, 7:40am

If I understand correctly, reading the bug report on Github, the issue is fixed indeed, but has not yet been packaged for Arch.

So, contrary to what I thought, the most up-do-date lxc package so far (4.0.11-2) still contains the bug and we should stick to 4.0.10-2 when using LTS kernel until next release of the lxc package (which I supposed will be 4.0.11-3?)

If that’s confirmed, then please consider this as fixed.

broizter · October 28, 2021, 8:41am

It’s the lxd package that needs an update. Lxc added a feature that lxd didn’t support properly. I think the arch maintainer decided to wait for the next lxd release instead of patching the current version. It should be out in a week or so.

lyderic · October 28, 2021, 12:15pm

Excellent. Many thanks for confirming.
L.

toby63 · October 28, 2021, 3:39pm

See:

github.com/lxc/lxd

Cannot start any container - Arch Linux - kernel 5.10.75-1-lts

opened 09:06AM - 23 Oct 21 UTC

closed 12:58PM - 25 Oct 21 UTC

pixrl

Bug

# Required information * Distribution: Arch Linux * Distribution version: …rolling release * `lxc-start --version` ``` 4.0.11 ``` * `lxc-checkconfig` ``` LXC version 4.0.11 --- Namespaces --- Namespaces: enabled Utsname namespace: enabled Ipc namespace: enabled Pid namespace: enabled User namespace: enabled newuidmap is not installed newgidmap is not installed Network namespace: enabled --- Control groups --- Cgroups: enabled Cgroup namespace: enabled Cgroup v1 mount points: Cgroup v2 mount points: /sys/fs/cgroup Cgroup v1 systemd controller: missing Cgroup v1 freezer controller: missing Cgroup ns_cgroup: required Cgroup device: enabled Cgroup sched: enabled Cgroup cpu account: enabled Cgroup memory controller: enabled Cgroup cpuset: enabled --- Misc --- Veth pair device: enabled, loaded Macvlan: enabled, not loaded Vlan: enabled, not loaded Bridges: enabled, loaded Advanced netfilter: enabled, not loaded CONFIG_NF_NAT_IPV4: missing CONFIG_NF_NAT_IPV6: missing CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded FUSE (for use with lxcfs): enabled, loaded --- Checkpoint/Restore --- checkpoint restore: enabled CONFIG_FHANDLE: enabled CONFIG_EVENTFD: enabled CONFIG_EPOLL: enabled CONFIG_UNIX_DIAG: enabled CONFIG_INET_DIAG: enabled CONFIG_PACKET_DIAG: enabled CONFIG_NETLINK_DIAG: enabled File capabilities: Note : Before booting a new kernel, you can check its configuration usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig ``` * `uname -a` ``` Linux zen 5.10.75-1-lts lxc/lxc#1 SMP Wed, 20 Oct 2021 11:02:09 +0000 x86_64 GNU/Linux ``` * `cat /proc/self/cgroup` ``` 0::/user.slice/user-1000.slice/session-1.scope ``` * `cat /proc/1/mounts` ``` proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 dev /dev devtmpfs rw,nosuid,relatime,size=1972868k,nr_inodes=493217,mode=755,inode64 0 0 run /run tmpfs rw,nosuid,nodev,relatime,mode=755,inode64 0 0 efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0 /dev/sda2 / btrfs rw,relatime,ssd,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/ 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13628 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /tmp tmpfs rw,nosuid,nodev,nr_inodes=409600,inode64 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 /boot vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro 0 0 lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0 tmpfs /var/lib/lxd/shmounts tmpfs rw,relatime,size=100k,mode=711,inode64 0 0 tmpfs /var/lib/lxd/devlxd tmpfs rw,relatime,size=100k,mode=755,inode64 0 0 /dev/sda2 /var/lib/lxd/storage-pools/lxd btrfs rw,relatime,ssd,space_cache,user_subvol_rm_allowed,subvolid=261,subvol=/var/lib/lxd/storage-pools/lxd 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=396524k,nr_inodes=99131,mode=700,uid=1000,gid=1000,inode64 0 0 ``` # Issue description Cannot start any container after kernel upgrade probably. # Steps to reproduce 1. Trying to use `lxc start` to start container and getting an error # Information to attach ``` core@zen ~> lxc info --show-log pihole Name: pihole Status: STOPPED Type: container Architecture: x86_64 Created: 2021/09/26 13:17 CEST Last Used: 2021/10/23 10:53 CEST Log: lxc pihole 20211023085322.793 ERROR cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:332 - Operacja niedozwolona - Failed to load bpf program: (null) lxc pihole 20211023085322.149 ERROR start - start.c:core_scheduling:1570 - Zły argument - The kernel does not support core scheduling lxc pihole 20211023085322.150 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state "ABORTING" instead of "RUNNING" lxc pihole 20211023085322.150 ERROR start - start.c:__lxc_start:2068 - Failed to spawn container "pihole" lxc pihole 20211023085322.150 WARN start - start.c:lxc_abort:1038 - Nie ma takiego procesu - Failed to send SIGKILL via pidfd 17 for process 968 lxc 20211023085327.269 ERROR af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Połączenie zerwane przez drugą stronę - Failed to receive response lxc 20211023085327.269 ERROR commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors ```

and https://github.com/lxc/lxd/issues/9419#issuecomment-950657812

broizter · October 28, 2021, 4:55pm

I saw your comment on GitHub which seems to have been deleted now, but Arch hasn’t gotten the update to lxc and lxd so thats why it doesn’t work yet.

toby63 · October 28, 2021, 6:00pm

Yes, I assumed originally that only the LXC patch was needed, but then I realized that somewhere in the thread there are more patches mentioned for LXD.
Thats why I removed the comment again .

Thanks anyway for the info.

I guess that the Arch maintainers will include that, when they have the time.

lyderic · November 6, 2021, 5:09pm

For info, I just updated two machines with LTS kernels to the latest lxd and now all is working.

The following package combination is working for me:

$ pacman -Q lxc lxd lxcfs linux-lts
lxc 1:4.0.11-2
lxd 4.20-1
lxcfs 4.0.11-1
linux-lts 5.10.77-3