Containers do not launch on Linux 5.12

C0rn3j · May 6, 2021, 2:39pm

[ 2842.768889] lxdbr0: port 3(veth8a78ee33) entered blocking state
[ 2842.768922] lxdbr0: port 3(veth8a78ee33) entered disabled state
[ 2842.769238] device veth8a78ee33 entered promiscuous mode
[ 2842.769303] audit: type=1700 audit(1620311921.275:480): dev=veth8a78ee33 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[ 2842.769679] audit: type=1300 audit(1620311921.275:480): arch=c000003e syscall=46 success=yes exit=40 a0=3 a1=7ffcdbe2be00 a2=0 a3=55a65eb042a0 items=0 ppid=2237 pid=6269 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ip" exe="/usr/bin/ip" key=(null)
[ 2842.769695] audit: type=1327 audit(1620311921.275:480): proctitle=6970006C696E6B007365740064657600766574683861373865653333006D6173746572006C7864627230
[ 2843.104997] audit: type=1334 audit(1620311921.608:481): prog-id=74 op=LOAD
[ 2843.105014] audit: type=1300 audit(1620311921.608:481): arch=c000003e syscall=321 success=yes exit=14 a0=5 a1=7ffc62ac1840 a2=78 a3=7f832d373010 items=0 ppid=1 pid=6280 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="lxd" exe="/usr/bin/lxd" key=(null)
[ 2843.105035] audit: type=1327 audit(1620311921.608:481): proctitle=5B6C7863206D6F6E69746F725D202F7661722F6C69622F6C78642F636F6E7461696E6572732077656B616E33
[ 2843.105124] audit: type=1334 audit(1620311921.608:482): prog-id=74 op=UNLOAD

Doesn’t look like it.
For the record, I reproduced this on two systems.

brauner · May 6, 2021, 2:46pm

Wait, can you do

getcap /usr/bin/newuidmap
getcap /usr/bin/newgidmap

please?

C0rn3j · May 6, 2021, 2:47pm

/usr/bin/newuidmap cap_setuid=ep
/usr/bin/newgidmap cap_setgid=ep

brauner · May 6, 2021, 2:47pm

Strange my version of Archlinux doesn’t even have those bits set. Weird.

brauner · May 6, 2021, 3:06pm

Are there any Archlinux specific patches? Security modules?

brauner · May 6, 2021, 3:51pm

@Foxboron I’m testing a fix in LXC. I think we can treat this as a LXC regression as other tools won’t be affected by this very likely.

Foxboron · May 6, 2021, 3:53pm

I wanted to try reproduce it with an lxd Arch vm, but qemu doesn’t actually launch either Please do pass me patches and I’ll do package builds with them for testing on our end!

Thanks for the quick response

(This is just a testament that I should probably leave lxd in testing for a week going forward)

brauner · May 6, 2021, 3:55pm

Oh? What error are you seeing with qemu? @stgraber might helper there.

Foxboron · May 6, 2021, 3:58pm

qemu-system-x86_64:/var/log/lxd/test/qemu.conf:50: memdev=mem0 is ambiguous

I’m suspecting there are more issues with the soft-deprecation of the qemu.conf going on?

λ ~ » lxc start test 
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited -- /usr/bin/qemu-system-x86_64 -S -name test -uuid
4e74621a-32e9-4945-a4e4-d5ec40b9b42c -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-reboot
-no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig
/var/log/lxd/test/qemu.conf -pidfile /var/log/lxd/test/qemu.pid -D /var/log/lxd/test/qemu.log -chroot
/var/lib/lxd/virtual-machines/test -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas nobody: char device
redirected to /dev/pts/7 (label console)
: Process exited with a non-zero value
Try `lxc info --show-log test` for more info

λ ~ » lxc info --show-log test
Name: test
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/05/06 15:49 UTC
Status: Stopped
Type: virtual-machine
Profiles: default

Log:

qemu-system-x86_64:/var/log/lxd/test/qemu.conf:50: memdev=mem0 is ambiguous

brauner · May 6, 2021, 3:58pm

Very likely. This has been quite the pita.

stgraber · May 6, 2021, 4:07pm

Is that QEMU 6.0? If so, we haven’t actually done any testing/validation on that version yet and given the list of upstream changes, we indeed expect some breakage.

Foxboron · May 6, 2021, 4:11pm

Ack. I’ll try take a few stabs at the changes and see if this is easily solveable else just recommend people hold back qemu. Sadly pushed lxd before there was a new major version of qemu in the repositories

brauner · May 6, 2021, 4:20pm

@Foxboron this should fix it:

brauner · May 6, 2021, 4:20pm

I actually compiled this on archlinux and tested it. I haven’t had an Archlinux box in a while that was fun!

Foxboron · May 6, 2021, 4:32pm

Patch for the package, and test packages uploaded for @C0rn3j to test as well (includes a 4.0.9 update to lxc).

https://pkgbuild.com/~foxboron/repos/loose-pkgs/

Works on my machine at least

diff --git a/lxc/PKGBUILD b/lxc/PKGBUILD
index 4129195..4c56d51 100644
--- a/lxc/PKGBUILD
+++ b/lxc/PKGBUILD
@@ -1,4 +1,5 @@
 # Maintainer: Sergej Pupykin <pupykin.s+arch@gmail.com>
+# Maintainer: Morten Linderud <foxboron@archlinux.org>
 # Contributor: Andrea Zucchelli <zukka77@gmail.com>
 # Contributor: Daniel Micay <danielmicay@gmail.com>
 # Contributor: Jonathan Liu <net147@gmail.com>
@@ -6,7 +7,7 @@
 
 pkgname=lxc
 epoch=1
-pkgver=4.0.8
+pkgver=4.0.9
 pkgrel=1
 pkgdesc="Linux Containers"
 arch=('x86_64')
@@ -23,11 +24,13 @@ backup=('etc/lxc/default.conf'
        'etc/default/lxc')
 validpgpkeys=('602F567663E593BCBD14F338C638974D64792D67')
 source=("https://linuxcontainers.org/downloads/lxc/$pkgname-${pkgver}.tar.gz"{,.asc}
+       "https://patch-diff.githubusercontent.com/raw/lxc/lxc/pull/3827.patch"
        "lxc.tmpfiles.d"
        "lxc.service"
        "lxc-auto.service")
-sha256sums=('ac55852bcc7b828e82cfa4afec794dd030a030c728b7b56c0c6b671fb5af0b9b'
+sha256sums=('1fcf0610e9140eceb4be2334eb537bb9c5a213faea77c793ab3c62b86f37e52b'
             'SKIP'
+            '05031e6e12768a0928adbf2c745f11a8d9152428e0bf3ea81eee1855f27ca540'
             '10e4f661872f773bf3122a2f9f2cb13344fea86a4ab72beecb4213be4325c479'
             'bbe7e0447bc3bf5f75f312c34d647f5218024731628a5e8633b1ea1801ebe16b'
             'b31f8d6b301ab9901b43f2696bcd0babb32b96e4a59fab63a2d642e43bf26bb3')
@@ -37,6 +40,7 @@ prepare() {
   sed -i \
     -e 's|"\\"-//Davenport//DTD DocBook V3.0//EN\\""|"\\"-//OASIS//DTD DocBook XML\\" \\"https://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd\\""|' \
     configure.ac
+    patch -Np1 < "$srcdir/3827.patch"
 }
 
 build() {

brauner · May 6, 2021, 4:48pm

Merged and pushed to upstream stable-4.0. If you wouldn’t mind cherry-picking this patch that would be highly appreciated.

brauner · May 6, 2021, 4:50pm

That’s

in master
and

in stable-4.0. Should apply cleanly to 4.0.8 and 4.0.9.

C0rn3j · May 6, 2021, 7:51pm

Confirming that with this package I can now run containers on 5.12, thanks everyone!

Foxboron · May 6, 2021, 8:59pm

Pushed a new package to Arch with the patch. Thanks again for the fast response!

brauner · May 7, 2021, 7:14am

Excellent thank you very much.

Fwiw, you likely want to also pick up

as a patch to shadow utils (newuidmap). We will be releasing a new
version of this package soon too. This will take care that root users
using newuidmap to map host uid 0 where newuidmap is compiled with
capability support don’t see any issues.

Christian