Containers not picking up DHCP from bridge

infinitesteps · July 23, 2019, 2:30am

Guests won’t receive DHCP offers. The network is bridged. When I set the guest IP and gateway manually (inside the guest using ip) the network functions correctly. I have verified using tcpdump that DHCP offers are arriving on the bridge interface (and seem to also arrive on the guest eth0 interface, but the client can’t see them). I have tried multiple OSs (debian and ubuntu) and multiple DHCP clients (dhclient [isc-dhcp-client], and pump).

$ lxc --version
3.15

$ uname -a
Linux baz 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux

Host: Debian GNU/Linux 10 (buster)
Guest: (multiple) images:debian/10, images:ubuntu/14.04
Guest config (debian):
…
volatile.eth0.host_name: vethIH2BWO
volatile.eth0.hwaddr: 52:54:00:ee:14:64
…
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
…

Bridge config:
config: {}
description: “”
name: br0
type: bridge
used_by:

/1.0/containers/foo
managed: false
status: “”
locations: []

gpatel-fr · July 23, 2019, 5:04am

this is not clear, either packets arrive or they don’t :-).
If you can see them with tcpdump running inside the container, they are coming in. That should mean that:

either they are coming from a process running on the host, like the default dnsmasq from the default lxd bridge, because it has direct access to lxdbr0
or you have setup a dhcp relay (routing don’t work for dhcp AFAIK)
Both options should work. If as one can suspect the second possibility is the one you use (because the first one is well, just strange, why replace dnsmasq, the default lxd tool ? it works out of the box), maybe you did not setup the dhcp relay correctly.

tomp · July 23, 2019, 7:54am

@infinitesteps can you run the lxc config show --expanded <container name> so we can see the profile config too.

As @gpatel-fr suggested, it would be good to confirm whether you are seeing the packets arrive inside the container.

Can you also advise where the DHCP server is running, is it off host and the br0 bridge is connected to an external network, or is it running locally and listening on br0?

infinitesteps · July 23, 2019, 12:06pm

My apologies. I should have been more careful with the packet inspection on the guest. I was a bit weary. I just reran packet capture (from within the guest) during a dhclient discover request and confirmed that return packets are not making to the guest eth0 interface.

Additionally I ran the same test except captured (from within the host) the virtual nic on the host (vethYB6MF1) that corresponds to the guest. Same result - DHCP offer packets are not making it to this interface.

For good measure, I reran the same test except captured on the host br0 interface. The DHCP offer packets are making it to this interface and seem to be correct (the destination mac address is correct). I hadn’t study DHCP before and noticed that during this debugging that DHCP offer packets have layer 3 IP data in them - which I found odd since the guest doesn’t have an IP address yet. I guess these are “unicast” packets? Including this here in case it is a clue - do the virtual interfaces need to be more promiscuous?

The dhcp server is external to this system - guests are joining a broader network.

Config data…

$ lxc config show --expanded foo
architecture: x86_64
config:
image.architecture: amd64
image.description: Debian buster amd64 (20190720_05:24)
image.os: Debian
image.release: buster
image.serial: “20190720_05:24”
volatile.base_image: bed9e862faaa94ee41eee34c747f5dfbf4e952016e5862e9580b8f17a143a98d
volatile.eth0.host_name: vethYB6MF1
volatile.eth0.hwaddr: 52:54:00:ee:14:64
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.power: RUNNING
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:

default
stateful: false
description: “”

$ lxc profile show default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: default
type: disk
name: default
used_by:

/1.0/containers/foo
/1.0/containers/foo/Initial Minimal Install
/1.0/containers/foo/Base packages installed

Thanks for the help!

tomp · July 23, 2019, 1:49pm

Thanks, that helps to build the picture.

Can you also advise where the DHCP server is running?

Is it off host and is the br0 bridge is connected to an external network, or is it running locally and listening on br0?

tomp · July 23, 2019, 1:50pm

Sorry just saw your bottom part of the post

So br0 is bridged to eth0 on the host.

You can see offer packets arriving at br0, but not being relayed onto the host veth interface.

tomp · July 23, 2019, 1:52pm

Can you confirm the bridge is working by going into the container and manually configuring an IP on the same subnet as the external network, just using the ip a command and check you can ping hosts that are not on the same host (i.e the DHCP server’s address).

We need to figure out if the bridge is fundamentally working.

infinitesteps · July 23, 2019, 3:04pm

Confirmed. When I manually set the ip and gateway the guest can use the network normally (pinging google.com works for example) except the dhclient still cannot receive DHCP offers.

tomp · July 23, 2019, 3:40pm

@infinitesteps can you send me the output of from the host please:

  bridge  link show  br0

infinitesteps · July 23, 2019, 4:46pm

Sure thing…

bridge  link show  br0
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2 
40: vethYB6MF1@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2

tomp · July 23, 2019, 5:14pm

Thanks, also output from:

ebtables -L

infinitesteps · July 23, 2019, 5:16pm

No problemo…

ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

infinitesteps · July 23, 2019, 6:34pm

Additional info. Just to be sure that the DHCP server wasn’t somehow sending malformed offers, I tested dhclient from the host. dhclient ran just fine on the host - it requested and received/processed offers.

tomp · July 24, 2019, 12:19pm

@infinitesteps can you send me the PCAP you’re seeing on the bridge for DHCP? thomas dot parrott at canonical dot com

Thanks

rkelleyrtp · July 24, 2019, 12:33pm

Is this a physical or virtualized server? If virtualized, what hypervisor are you running?

infinitesteps · July 24, 2019, 12:38pm

DM sent. Thanks!

infinitesteps · July 24, 2019, 12:39pm

The host is a vmware (VMware ESXi, 6.7.0, 13006603) virtual machine connected to a virtual switch that is promiscuous. The virtual switch is connected to a physical NIC that is attached to a VLAN.

rkelleyrtp · July 24, 2019, 12:45pm

By chance, have you also enabled MAC Address changes and Forged Transmits? I had to do this on my ESX 6.7 DVSwitch to get LXC containers running properly.

infinitesteps · July 24, 2019, 3:20pm

Thanks Ron. I just confirmed:

Security
Promiscuous mode 	Accept
MAC address changes 	Accept
Forged transmits 	Accept

gpatel-fr · July 24, 2019, 6:33pm

While I have to say that it’s a massive shot in the dark, how about
sudo brctl stp br0 on
if it breaks something instead of fixing it, you can always reverse it