Default bridge: no network on Archlinux, working on Ubuntu container

LXD
1

I’m trying to start a unprivileged Archlinux container on a Manjaro host. All of the settings are default or set according to the arch wiki.

My problem is that no matter what, the Archlinux container does not get an ipv4 address assigned. An Ubuntu container, on the other hand, works out of the box.

`lxc launch images:ubuntu/20.04 hlos-ubuntu`  # gets ipv4, internet works out of the box
`lxc launch images:archlinux hlos-arch`  # no ipv4 -> no internet

+-------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|    NAME     |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+-------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| hlos-arch   | RUNNING |                      | fd42:9baf:9d56:538d:216:3eff:fe6a:ef95 (eth0) | CONTAINER | 0         |
+-------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| hlos-ubuntu | RUNNING | 10.25.199.228 (eth0) | fd42:9baf:9d56:538d:216:3eff:fe2f:3a57 (eth0) | CONTAINER | 0         |
+-------------+---------+----------------------+-----------------------------------------------+-----------+-----------+

There are A LOT of wrong leads on the internet because there was a bug in systemd for a while, which prevented networking in lxc.

My best guess (after working trough dozens of google results, github issues and countless mailing lists) that there is something broken in the archlinux container.
journalctl inside the container has some leads but looking them up lead me back to old/fixed issues.

Some lines of interst:

Jun 16 19:44:04 hlos-arch systemd-udevd[50]: Failed to chown '/dev/net/tun' 0 0: Operation not permitted
Jun 16 19:44:04 hlos-arch systemd-udevd[50]: Failed to apply permissions on static device nodes: Operation not permitted
Jun 16 19:44:04 hlos-arch systemd[1]: Started udev Kernel Device Manager.
Jun 16 19:44:04 hlos-arch systemd-udevd[58]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
...
Jun 16 19:44:04 hlos-arch systemd[59]: systemd-networkd.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Permission denied
Jun 16 19:44:04 hlos-arch systemd[59]: systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Permission denied
...
Jun 16 19:44:04 hlos-arch systemd[1]: Starting Network Service...
Jun 16 19:44:04 hlos-arch systemd[61]: systemd-logind.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Permission denied
Jun 16 19:44:04 hlos-arch systemd[61]: systemd-logind.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-logind: Permission denied

Could somebody please verify, if the Archlinux container works for them? Any hint is greatly appreciated.

2

Just tested the Arch image on an Ubuntu host and it works just fine here.
What LXD version is this?

One workaround for such issue would be to set security.nesting=true on the container but it’s not something I’ve needed to do here so still wondering why it’s behaving differently on your system.

3

For the time being, i switched back to using a VM to continue my work, but I’m still curious what’s going on here.
LXD version is 4.2

4

Looks a lot like an apparmor failure to me.
dmesg would probably show some denials

5

Good call! I wonder why this is only an issue with the Archlinux container…

[ 7345.930676] lxdbr0: port 1(veth6d57013c) entered blocking state
[ 7345.930882] lxdbr0: port 1(veth6d57013c) entered disabled state
[ 7345.933167] device veth6d57013c entered promiscuous mode
[ 7345.933182] audit: type=1700 audit(1592405655.520:406): dev=veth6d57013c prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
[ 7345.948519] IPv6: ADDRCONF(NETDEV_CHANGE): vethbdb942a5: link becomes ready
[ 7345.948583] lxdbr0: port 1(veth6d57013c) entered blocking state
[ 7345.948584] lxdbr0: port 1(veth6d57013c) entered forwarding state
[ 7346.025935] audit: type=1400 audit(1592405655.614:407): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-hlos-arch_</var/lib/lxd>" pid=84363 comm="apparmor_parser"
[ 7346.087598] eth0: renamed from vethbdb942a5
[ 7346.103718] lxdbr0: port 1(veth6d57013c) entered disabled state
[ 7346.105624] lxdbr0: port 1(veth6d57013c) entered blocking state
[ 7346.105625] lxdbr0: port 1(veth6d57013c) entered forwarding state
[ 7346.484299] audit: type=1400 audit(1592405656.074:408): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84441 comm="(d-logind)" flags="ro, remount, noatime, bind"
[ 7346.485522] audit: type=1400 audit(1592405656.074:409): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84442 comm="(networkd)" flags="ro, remount, noatime, bind"
[ 7346.492335] audit: type=1400 audit(1592405656.080:410): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84445 comm="(d-logind)" flags="ro, remount, noatime, bind"
[ 7346.494926] audit: type=1400 audit(1592405656.084:411): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84446 comm="(networkd)" flags="ro, remount, noatime, bind"
[ 7346.500453] audit: type=1400 audit(1592405656.087:412): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84449 comm="(d-logind)" flags="ro, remount, noatime, bind"
[ 7346.503002] audit: type=1400 audit(1592405656.090:413): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84450 comm="(networkd)" flags="ro, remount, noatime, bind"
[ 7346.508783] audit: type=1400 audit(1592405656.097:414): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84453 comm="(d-logind)" flags="ro, remount, noatime, bind"
[ 7346.511098] audit: type=1400 audit(1592405656.104:415): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-hlos-arch_</var/lib/lxd>" name="/run/systemd/unit-root/" pid=84454 comm="(networkd)" flags="ro, remount, noatime, bind"

is there a proper fix for this?

6

Our apparmor policy for the container specifically allows that set of flags and path in the base profile so it’s quite weird…

Can you look for the relevant profile in /var/snap/lxd/common/lxd/security and confirm that you see the ro,remount,noatime,bind mount entries in there?

7

It’s not a snap installation. Output of /var/lib/lxd/security/apparmor/profiles/lxd-hlos-arch here. Is this what you need?

8

Yeah, the profile looks correct with the exact thing you tried to do being allowed according to it… This feels like an apparmor-parser or kernel bug in this case.

The security.nesting=true workaround should be fine in this case but you probably should file a bug against apparmor in your distro.

9

Thank you for your quick response. I can confirm that the workaround indeed works.
I filed an issue for manjaro (hopefully in the right place)