Containers get no IPv4

LXD
1

My containers don’t get IPv4, only IPv6. They used to get IPs, but has recently stopped getting them and I have no idea why.

I’ve ensured my Debian Buster system is using /usr/sbin/iptables-legacy (by running update-alternatives --set iptables /usr/sbin/iptables-legacy):

root@quanah ~ # ls /etc/alternatives/iptableslrwxrwxrwx 1 root root 25 Mar 26 06:47 /etc/alternatives/iptables -> /usr/sbin/iptables-legacy

I have nothing listening on 53 on the host, so LXD’s dnsmasq should have no problems.

root@quanah ~ # netstat -nlp --tcp | grep -w 53
tcp        0      0 10.107.206.1:53         0.0.0.0:*               LISTEN      351/dnsmasq         tcp6       0      0 fd42:71d3:6e74:8047::53 :::*                    LISTEN      351/dnsmasq         
tcp6       0      0 fe80::216:3eff:fed0::53 :::*                    LISTEN      351/dnsmasq

The one you see in the netstat listing is LXD’s:

~ $ ps auxww | grep -w 351     
lxd        351  0.0  0.0  43640  3476 ?        Ss   07:15   0:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.107.206.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.107.206.2,10.107.206.254,1h --listen-address=fd42:71d3:6e74:8047::1 --enable-ra --dhcp-range ::,constructor:lxdbr0,ra-stateless,ra-names -s lxd -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd

I have tried restarting the network, then snap and then, for good measure, lxd:

# ifdown wlp4s0
# ifup wlp4s0
# systemctl restart snapd
# snap restart lxd

The most interesting bit in /var/log/syslog to my eyes is:

Mar 26 07:15:08 quanah systemd-udevd[377]: Could not generate persistent MAC address for vethb5794188: No such file or directory
Mar 26 07:15:08 quanah systemd-udevd[32691]: Could not generate persistent MAC address for veth3012792f: No such file or directory

One error per container.

To rule out the firewall, I’ve turned off my own rules and can only see the ones set up by LXD:

root@quanah ~ # iptables -LChain INPUT (policy ACCEPT)
target     prot opt source               destination         ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain /* generated for LXD network lxdbr0 */ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps /* generated for LXD network lxdbr0 */

Chain FORWARD (policy ACCEPT)target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* generated for LXD network lxdbr0 */
ACCEPT     all  --  anywhere             anywhere             /* generated for LXD network lxdbr0 */
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:domain /* generated for LXD network lxdbr0 */ACCEPT     udp  --  anywhere             anywhere             udp spt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps /* generated for LXD network lxdbr0 */

Versions

~ $ grep PRETTY /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
~ $ snap --version 
snap    2.49.1
snapd   2.49.1
series  16
debian  10
kernel  4.19.0-14-amd64
~ $ lxd --version
4.12
~ $

Any help would be much appreciated.

2

Did you try manually running dhclient -v eth0 in the container?

3

No, but I tried it now. It doesn’t get an IP (the container runs Ubuntu 20.04.2 LTS):

root@dev:~# dhclient -v eth0
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/00:16:3e:c8:ad:28
Sending on   LPF/eth0/00:16:3e:c8:ad:28
Sending on   Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 4 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 7 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 19 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 9 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 11 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 9 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 16 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 12 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 18 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 10 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 15 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 17 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 19 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 19 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 20 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 20 (xid=0xe7cdc950)
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 14 (xid=0xe7cdc950)
No DHCPOFFERS received.
No working leases in persistent database - sleeping.
4

Okay, can you try to tcpdump your bridge on the host to see if those DHCP requests are making it through?

5

Looks like it, doesn’t it?

root@quanah ~ # tcpdump -i lxdbr0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:43:49.375466 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:c8:ad:28 (oui Unknown), length 300
16:43:52.882939 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:c8:ad:28 (oui Unknown), length 300
16:43:54.447331 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:c8:ad:28 (oui Unknown), length 287
6

Ok, anything in /var/log/syslog which would explain why dnsmasq is ignoring it?

7

Seems to be apparmor who is spoiling the party:

Mar 26 17:17:16 quanah kernel: [564990.251192] audit: type=1400 audit(1616775436.181:581): apparmor="DENIED" operation="open" namespace="root//lxd-dev_<var-snap-lxd-common-lxd>" profile="/{,usr/}sbin/dhclient" name="/proc/783/task/784/comm" pid=15864 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=1000000 ouid=1000000
Mar 26 17:17:16 quanah kernel: [564990.251307] audit: type=1400 audit(1616775436.181:582): apparmor="DENIED" operation="open" namespace="root//lxd-dev_<var-snap-lxd-common-lxd>" profile="/{,usr/}sbin/dhclient" name="/proc/783/task/785/comm" pid=15864 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=1000000 ouid=1000000
Mar 26 17:17:16 quanah kernel: [564990.251355] audit: type=1400 audit(1616775436.181:583): apparmor="DENIED" operation="open" namespace="root//lxd-dev_<var-snap-lxd-common-lxd>" profile="/{,usr/}sbin/dhclient" name="/proc/783/task/786/comm" pid=15864 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=1000000 ouid=1000000
8

That’s unlikely to be the issue, if apparmor was blocking things, you wouldn’t see the request come out, if apparmor is blocking handling the response, you’d still see the response in tcpdump.

9

Mhmm, those DENIED messages for dhclient sounded spot on, but if that’s not it, that’s a pity since there are no other messages being written to /var/log/syslog when I run dhclient -v eth0 inside the container.

10

Does systemctl reload snap.lxd.daemon cause dnsmasq to get restarted?
And if so, does that help?

11

That’s a negative:

systemctl reload snap.lxd.daemon

Did not restart dnsmasq to restart. It still runs with the same PID and start time.

12

Hmm, kill dnsmasq and reload LXD to have it get respawned maybe?

13 
# killall dnsmasq
# systemctl reload snap.lxd.daemon

did indeed start a new dnsmasq process. Running dhclient -v eth0 inside the container still didn’t yield an IP, though.

14

I wonder if you’re dealing with a conflict between nft and iptables somehow.
After you reconfigured your iptables, did you reboot your system to ensure that everything went through the legacy iptables?

Can you show nft list ruleset as root (may require nftables to be installed)?

15

No, I haven’t rebooted (in a couple of weeks), so it’s of course possible something lingers in a hidden corner.

Anyways, after apt install nftables, it shows:

table ip filter {
        chain INPUT {
                type filter hook input priority 0; policy drop;
                iifname "lo" ip saddr 127.0.0.1 ip daddr 127.0.0.1 counter packets 108 bytes 5536 accept
                iifname "lo" ip saddr 127.0.0.1 ip daddr 127.0.1.1 counter packets 20209 bytes 2099193 accept
                ct state related,established counter packets 1630073 bytes 2058579198 accept
                meta l4proto icmp icmp type destination-unreachable counter packets 8 bytes 2655 accept
                meta l4proto icmp icmp type time-exceeded counter packets 0 bytes 0 accept
                meta l4proto icmp icmp type echo-request counter packets 0 bytes 0 accept
                meta l4proto icmp icmp type echo-reply counter packets 0 bytes 0 accept
                meta l4proto tcp tcp dport 22 counter packets 7 bytes 420 accept
                meta l4proto tcp tcp dport 80 counter packets 0 bytes 0 accept
                meta l4proto tcp tcp dport 443 counter packets 0 bytes 0 accept
                meta l4proto tcp tcp dport 3142 counter packets 0 bytes 0 accept
                meta l4proto tcp tcp dport 8081 counter packets 0 bytes 0 accept
                meta l4proto tcp tcp dport 8899 counter packets 0 bytes 0 accept
                meta l4proto udp udp dport 60000-61000 counter packets 303 bytes 126385 accept
                counter packets 97767 bytes 18908458 drop
        }

        chain FORWARD {
                type filter hook forward priority 0; policy drop;
        }

        chain OUTPUT {
                type filter hook output priority 0; policy accept;
        }
}
table ip nat {
        chain PREROUTING {
                type nat hook prerouting priority -100; policy accept;
        }

        chain INPUT {
                type nat hook input priority 100; policy accept;
        }

        chain POSTROUTING {
                type nat hook postrouting priority 100; policy accept;
        }

        chain OUTPUT {
                type nat hook output priority -100; policy accept;
        }
}

Of course, I could reboot and see if it’s working then, but I’d love to find out exactly what’s keeping it from giving out IPs as I may run into this later and a reboot is rather inconvenient.

16

Yeah, so your nft based firewall is dropping stuff as you can see in the output above.

Combining both nft and xtables based firewalling usually leads to a very bad day as the application order is not deterministic and if either rejects something, you’re done and left with little to no indication of what happened.

In this case, it’s showing that all forwarding is being dropped and input into the host is also severely restricted (doesn’t allow 67/68 as would be required for DHCP).

17

I see. How could I have arrived at this spot? I’ve never used nft (didn’t even have the nftables package installed).

18

But you were using the non-legacy iptables before you used update-alternatives to switch over to legacy.

non-legacy iptables is nft (not using the command but uses the kernel interface directly) whereas legacy iptables is the regular xtables interface.

So your system started with iptables using nft, loaded its rules in nft, you then switched to legacy which then caused a mixup of rules in both.

1 Like
19

That explains it. Thank you SO much for your help @stgraber, you rock.

For completeness and for anyone who arrives at this thread searching for the solution to the same problem, I’ll mention that to remedy this, I did:

root@quanah ~ # nft flush ruleset

I could then do:

root@dev:~# dhclient -v eth0

inside the container, which instantly gave me an IP.

1 Like