You mean the tcpdump output?
$ sudo tcpdump port bootpc -i eno2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno2, link-type EN10MB (Ethernet), capture size 262144 bytes
Yes that is good, and also for the same time inside the container.
What we’re trying to do is ascertain is the container making DHCP requests, and where are they getting lost.
P.s. I usually run my tcpdump with the “-nn” flag to disable host and port name lookups, so I get raw IP and port numbers.
There is no output inside the container when I restart systemd-networkd again and again:
# tcpdump port bootpc -nn -vv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
also, systemd-networkd could not be started. It seems that the container does not make DHCP requests?
● systemd-networkd.service - Network Service
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2019-11-25 09:52:03 UTC; 2min 26s ago
Docs: man:systemd-networkd.service(8)
Process: 119 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=226/NAMESPACE)
Main PID: 119 (code=exited, status=226/NAMESPACE)
Nov 25 09:52:03 archlinux-demo systemd[1]: systemd-networkd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Nov 25 09:52:03 archlinux-demo systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Nov 25 09:52:03 archlinux-demo systemd[1]: Stopped Network Service.
Nov 25 09:52:03 archlinux-demo systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Nov 25 09:52:03 archlinux-demo systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Nov 25 09:52:03 archlinux-demo systemd[1]: Failed to start Network Service.
Nov 25 09:52:04 archlinux-demo systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Nov 25 09:52:04 archlinux-demo systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Nov 25 09:52:04 archlinux-demo systemd[1]: Failed to start Network Service.
OK so we are getting clearer on the problem now, the issue is that systemd-networkd cannot start without security.nesting=true being set.
So we can say that something that systemd-networkd is trying to do inside Arch container is being blocked without that setting.
Perhaps check your AppArmor logs to see what is being blocked when security.nesting=false and you try to start systemd-networkd.
Oh, after a while, there are some output inside the container:
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:53:15.576023 IP (tos 0x0, ttl 128, id 16734, offset 0, flags [none], proto UDP (17), length 328)
172.21.17.196.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from c8:d3:ff:40:c5:11, length 300, xid 0xe54bbbed, Flags [none] (0x0000)
Client-IP 172.21.17.196
Client-Ethernet-Address c8:d3:ff:40:c5:11
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Inform
Client-ID Option 61, length 7: ether c8:d3:ff:40:c5:11
Hostname Option 12, length 9: "yadfsejie"
Vendor-Class Option 60, length 8: "MSFT 5.0"
Parameter-Request Option 55, length 13:
Subnet-Mask, Domain-Name, Default-Gateway, Domain-Name-Server
Netbios-Name-Server, Netbios-Node, Netbios-Scope, Router-Discovery
Static-Route, Classless-Static-Route, Classless-Static-Route-Microsoft, Vendor-Option
Option 252
09:53:47.371857 IP (tos 0x0, ttl 128, id 32662, offset 0, flags [none], proto UDP (17), length 328)
172.21.17.148.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from c8:d3:ff:40:cb:b9, length 300, xid 0xe81ca296, Flags [none] (0x0000)
Client-IP 172.21.17.148
Client-Ethernet-Address c8:d3:ff:40:cb:b9
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Inform
Client-ID Option 61, length 7: ether c8:d3:ff:40:cb:b9
Hostname Option 12, length 4: "wang"
Vendor-Class Option 60, length 8: "MSFT 5.0"
Parameter-Request Option 55, length 13:
Subnet-Mask, Domain-Name, Default-Gateway, Domain-Name-Server
Netbios-Name-Server, Netbios-Node, Netbios-Scope, Router-Discovery
Static-Route, Classless-Static-Route, Classless-Static-Route-Microsoft, Vendor-Option
Option 252
09:55:07.606466 IP (tos 0x0, ttl 128, id 17135, offset 0, flags [none], proto UDP (17), length 328)
172.21.17.196.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from c8:d3:ff:40:c5:11, length 300, xid 0x462097a9, Flags [none] (0x0000)
Client-IP 172.21.17.196
Client-Ethernet-Address c8:d3:ff:40:c5:11
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Inform
Client-ID Option 61, length 7: ether c8:d3:ff:40:c5:11
Hostname Option 12, length 9: "fewfe"
Vendor-Class Option 60, length 8: "MSFT 5.0"
Parameter-Request Option 55, length 13:
Subnet-Mask, Domain-Name, Default-Gateway, Domain-Name-Server
Netbios-Name-Server, Netbios-Node, Netbios-Scope, Router-Discovery
Static-Route, Classless-Static-Route, Classless-Static-Route-Microsoft, Vendor-Option
Option 252
09:55:32.210099 IP (tos 0x0, ttl 128, id 67, offset 0, flags [none], proto UDP (17), length 328)
172.21.17.148.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from c8:d3:ff:40:cb:b9, length 300, xid 0x7a2f3198, Flags [none] (0x0000)
Client-IP 172.21.17.148
Client-Ethernet-Address c8:d3:ff:40:cb:b9
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Inform
Client-ID Option 61, length 7: ether c8:d3:ff:40:cb:b9
Hostname Option 12, length 4: "wang"
Vendor-Class Option 60, length 8: "MSFT 5.0"
Parameter-Request Option 55, length 13:
Subnet-Mask, Domain-Name, Default-Gateway, Domain-Name-Server
Netbios-Name-Server, Netbios-Node, Netbios-Scope, Router-Discovery
Static-Route, Classless-Static-Route, Classless-Static-Route-Microsoft, Vendor-Option
Option 252
But according to the MAC, it’s not related to the container.
I know nothing about AppArmor. How could I check the log? Inside the container or the host system?
What version of systemd do you run?
Yes that is likely something else on your network. Which is encouraging at least it shows your MACVLAN is working. Just need to get your systemd-networkd working and hopefully will fix it.
I noticed you originally said systemd 237.
Although what is the version of systemd inside the container?
inside the container:
# systemctl --version
systemd 243 (243.162-2-arch)
+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid
in the host:
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
I wonder if you’re being affected by:
Looks like a partial fix was added in LXD 3.0.4.
dmesg gives me some logs similiar to the issue you mentioned:
[4574283.407848] device eth0 entered promiscuous mode
[4574297.859562] audit: type=1400 audit(1574675523.369:2743): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=211113 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4574297.905973] audit: type=1400 audit(1574675523.417:2744): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=211122 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4574297.953784] audit: type=1400 audit(1574675523.465:2745): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=211131 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4574297.998512] audit: type=1400 audit(1574675523.509:2746): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=211142 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4574298.030893] audit: type=1400 audit(1574675523.541:2747): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=211156 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4574536.139449] device eth0 left promiscuous mode
[4578823.084339] audit: type=1400 audit(1574680048.594:2748): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=226164 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4578823.190308] audit: type=1400 audit(1574680048.698:2749): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=226172 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4578823.268265] audit: type=1400 audit(1574680048.778:2750): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=226179 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4578823.364248] audit: type=1400 audit(1574680048.874:2751): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=226197 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
[4578823.456734] audit: type=1400 audit(1574680048.966:2752): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-archlinux-demo_</var/lib/lxd>" name="/run/systemd/unit-root/tmp/" pid=226209 comm="(networkd)" flags="ro, nosuid, nodev, remount, bind"
My host system is Ubuntu 18.04 with lxd 3.0.3 from the apt repo, not the snap repo. The version of lxd in apt repo will not bump during 18.04 lifetime, right? I could not use snap repo, snap has no mirrors, we suffer from poor network situation installing pkg from snap.
@stgraber is LXD 3.0.4 likely to be available as a deb in Ubuntu 18.04 at some point?
Any chance to upgrade to latest version? Or host in ppa?
Probably at some point, yes, though we may end up skipping and going straight to 3.0.5.
Anyone like myself stumbling across this topic after a similar issue cropped up back in early December 2020, I can confirm that the advice in this comment above about setting security.nesting=true still applies today with LXD 4.x and the latest:archlinux image, e.g.
lxc launch images:archlinux -c security.nesting=true
or
lxc init images:archlinux $container
lxc config set $container security.nesting true
Note: I have no idea why this works or what the security ramifications are but it suffices for the use case I have.
1 Like