Last time I turned on STP accidentally, I frustrated the network admins. So unless this is more than a shot in the dark, I think I’ll hold off on trying this.
I’ve taken a look at your packet traces, and everything seems fine, not seeing anything that you havent already described, so was as expected.
Have you got any iptables rules in place btw?
Can you provide a dump of iptables -L -v -n thanks
also the output of:
bridge fdb
Yep! Thanks again for the help!
# iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
# Warning: iptables-legacy tables present, use iptables-legacy to see them
# iptables-legacy -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
# bridge fdb
c8:d9:d2:18:53:3d dev ens192 master br0
c4:34:6b:73:62:b1 dev ens192 master br0
c8:d9:d2:18:4f:8a dev ens192 master br0
c8:d9:d2:19:73:90 dev ens192 master br0
c8:d9:d2:18:50:ef dev ens192 master br0
c8:d9:d2:18:4f:8f dev ens192 master br0
c8:d9:d2:18:4f:28 dev ens192 master br0
70:5a:0f:44:51:3c dev ens192 master br0
ec:8e:b5:27:83:6c dev ens192 master br0
e0:69:95:f5:17:55 dev ens192 master br0
70:5a:0f:48:c0:b5 dev ens192 master br0
70:5a:0f:3d:3f:c2 dev ens192 master br0
c8:d9:d2:18:4b:1d dev ens192 master br0
70:5a:0f:48:c0:82 dev ens192 master br0
c8:d9:d2:18:4e:ce dev ens192 master br0
a0:48:1c:a6:1d:a0 dev ens192 master br0
74:46:a0:af:83:cd dev ens192 master br0
c8:d9:d2:18:51:cb dev ens192 master br0
70:5a:0f:42:24:32 dev ens192 master br0
c4:34:6b:73:5f:0d dev ens192 master br0
00:26:73:79:15:4f dev ens192 master br0
c8:d9:d2:18:4f:16 dev ens192 master br0
3c:52:82:78:f8:61 dev ens192 master br0
c4:34:6b:61:fe:b2 dev ens192 master br0
70:5a:0f:42:23:e5 dev ens192 master br0
e0:69:95:f5:14:bf dev ens192 master br0
70:5a:0f:44:52:b8 dev ens192 master br0
3c:52:82:78:fb:ce dev ens192 master br0
00:26:73:40:5a:05 dev ens192 master br0
70:5a:0f:42:24:3f dev ens192 master br0
10:60:4b:79:32:b9 dev ens192 master br0
c4:34:6b:73:5e:be dev ens192 master br0
70:5a:0f:42:23:7d dev ens192 master br0
74:46:a0:9d:cb:23 dev ens192 master br0
70:5a:0f:48:c1:8a dev ens192 master br0
88:51:fb:6e:97:0f dev ens192 master br0
8c:dc:d4:27:fb:a0 dev ens192 master br0
74:46:a0:9d:f0:a6 dev ens192 master br0
74:46:a0:af:83:99 dev ens192 master br0
74:46:a0:9d:f0:9e dev ens192 master br0
3c:52:82:70:fb:ce dev ens192 master br0
74:46:a0:9d:f0:92 dev ens192 master br0
88:51:fb:6e:be:3e dev ens192 master br0
48:0f:cf:49:86:88 dev ens192 master br0
00:26:73:42:12:ed dev ens192 master br0
18:60:24:87:07:19 dev ens192 master br0
8c:dc:d4:2b:e7:ee dev ens192 master br0
78:7b:8a:c2:be:9a dev ens192 master br0
f4:ce:46:3c:51:37 dev ens192 master br0
ac:16:2d:39:76:aa dev ens192 master br0
c8:d9:d2:18:4c:54 dev ens192 master br0
18:60:24:87:07:17 dev ens192 master br0
c8:d9:d2:18:4f:a2 dev ens192 master br0
48:0f:cf:42:db:9d dev ens192 master br0
14:58:d0:ea:e0:00 dev ens192 master br0
00:50:56:af:a9:13 dev ens192 vlan 1 master br0 permanent
00:50:56:af:a9:13 dev ens192 master br0 permanent
01:00:5e:00:00:01 dev ens192 self permanent
33:33:00:00:00:01 dev br0 self permanent
01:00:5e:00:00:01 dev br0 self permanent
33:33:ff:af:a9:13 dev br0 self permanent
fe:9b:e7:32:d6:a8 dev vethHFL13L vlan 1 master br0 permanent
fe:9b:e7:32:d6:a8 dev vethHFL13L master br0 permanent
33:33:00:00:00:01 dev vethHFL13L self permanent
01:00:5e:00:00:01 dev vethHFL13L self permanent
33:33:ff:32:d6:a8 dev vethHFL13L self permanent
I can’t see the veth MAC in the forward table: 52:54:00:ee:14:64
I’m wondering if this is something to do with the VLAN, perhaps there is some tagging left on the packet that means the linux bridge is not routing it down to the veth pair.
Is the physical connection a trunked connection or an untagged connection?
It is trunked.
To test I moved the host to another untagged connection and had the same result. However, once (and only once) AFTER adding the ip and gateway manually to the guest, then pinging google, then manually running dhclient, I received offers (!?). I thought we had made some progress but then after rebooting the guest couldn’t replicate the behavior.
I think that the veth is showing in the bridge fdb…
# sudo bridge fdb > /tmp/a
# lxc start foo
# sudo bridge fdb > /tmp/b
# diff /tmp/a /tmp/b
0a1,2
> 10:60:4b:79:32:b7 dev ens192 master br0
> 70:5a:0f:44:51:ca dev ens192 master br0
36,40d37
< 50:65:f3:4b:bc:74 dev ens192 master br0
< 88:51:fb:6e:24:bc dev ens192 master br0
< 74:46:a0:9d:cb:20 dev ens192 master br0
< 8c:dc:d4:21:44:7f dev ens192 master br0
< 8c:dc:d4:27:fb:a5 dev ens192 master br0
66a64,68
> fe:6a:c0:a7:d6:2a dev vethHBH2WU vlan 1 master br0 permanent
> fe:6a:c0:a7:d6:2a dev vethHBH2WU master br0 permanent
> 33:33:00:00:00:01 dev vethHBH2WU self permanent
> 01:00:5e:00:00:01 dev vethHBH2WU self permanent
> 33:33:ff:a7:d6:2a dev vethHBH2WU self permanent
# ip a show dev vethHBH2WU
9: vethHBH2WU@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether fe:6a:c0:a7:d6:2a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::fc6a:c0ff:fea7:d62a/64 scope link
valid_lft forever preferred_lft forever
I configured a new host machine using Ubuntu 18.04 LTS. Same result.
4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux
FWIW I have already seen/read some similar behaviour, the reason was that using tcpdump set the adapter in promiscuous mode.
By host machine, I suspect you mean VM? If so, this really smells like some sort of vSwitch issue. Especially since you had to first ping out from the container to the network to get the arp table updated. Are you using standard vSwitches or Distributed vSwitches?
As a test, try installing another DHCP server on the same physical host as your test VM. Then, plumb both VMs (DHCP server and test VM) on a new VLAN and retry the DHCP request.
What does your /etc/network/interfaces file look like? Maybe there is some sort of VLAN or MTU issue? And, any chance you have duplicate MACs in your network?
Thanks for the tip. To test this, I just set both veth… and eth0 (in the guest) to promiscuous. No change.
“…first ping out from the container…” - I’d agree with you except it didn’t work on subsequent tries. So it may have been a coincidence.
“Are you using standard vSwitches or Distributed vSwitches?” Using standard virtual switches.
interfaces file looks normal:
auto br0
iface br0 inet static
bridge_ports ens192
bridge_fd 0
bridge_maxwait 0
bridge_stp off
address 10.121.1.30/16
gateway 10.121.30.1
dns-nameservers 10.120.160.13 10.120.160.14 10.120.160.1
The DHCP packets are making it to the virtual machine (on br0 and ens192) just not to the vethxxx nic (and subsequently not the container eth0 nic). For some reason the bridge is not passing the packets on.
Sorry, I see what you mean now. Yes. The guest mac address 52:54:00:ee:14:64 is not in the fdb until after I manually add an IP/gateway in the guest. Then it shows up there:
$ sudo bridge fdb |grep 64
52:54:00:ee:14:64 dev vethJTC9DP master br0
Seems like the broadcast packets (a la. DHCP request) should have established that entry without me manually configuring an IP address?
Even after the MAC address shows up as and FDB entry, the DHCP packets still don’t make it back to the guest.
The VLAN tagging doesn’t seem to be present on captured packets but that might be because they are stripped by the kernel. (?)
I will be away from this setup until next Friday so won’t be able to engage in deep debugging until then. I am still interested in getting to the bottom of the problem. I am less concerned about DHCP specifically as we could work around this with static addresses. I am more concerned that this is a symptom of something else (ex. why aren’t the ARP tables getting updated on the bridge)
Looks like a known issue with vmware? I worked around the problem by using a second nic and a macvlan instead. Described here:
https://www.claudiokuenzler.com/blog/556/network-connectivity-problem-lxc-in-vmware-vm-veth
1 Like
@infinitesteps I’m glad you got it sorted, and good to resurrect that old post as it still seems to be relevant. Thanks