Issues with LXD/C networking [Ubuntu 18.04 on VMware vSphere]

cjchristopher · October 15, 2020, 6:44am

As far as I can tell, none of the other plethora of posts necessarily helps me here - so apologies in advance if this is inadvertently a double post.

I’m running LXD and several containers on an Ubuntu 18.04 running on a VMware vSphere. I do not have access to the cloud infrastructure, so I believe some of the VMware mac address filtering issues mentioned around the place might be afflicting me? I have sudo on the VM, but it is managed above me as well.

I configured the default lxdbr0 with lxd init, and the containers can indeed DNS resolve, but I can’t otherwise do anything (e.g. apt install - everything times out).

$ lxc network list      
+--------+----------+---------+----------------+---------------------------+-------------+---------+
|  NAME  |   TYPE   | MANAGED |      IPV4      |           IPV6            | DESCRIPTION | USED BY |
+--------+----------+---------+----------------+---------------------------+-------------+---------+
| eth0   | physical | NO      |                |                           |             | 0       |
+--------+----------+---------+----------------+---------------------------+-------------+---------+
| lxdbr0 | bridge   | YES     | 10.47.110.1/24 | fd42:e0b7:79dd:cf0d::1/64 |             | 4       |
+--------+----------+---------+----------------+---------------------------+-------------+---------+

and

$ lxc network show lxdbr0
config:
  ipv4.address: 10.47.110.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:e0b7:79dd:cf0d::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- [redacted]
managed: true
status: Created
locations:
- none

and

$ lxc config show --expanded [name]
architecture: x86_64
config:
  boot.autostart: "1"
  image.architecture: amd64
  image.description: Ubuntu bionic amd64 (20201014_07:42)
  image.os: Ubuntu
  image.release: bionic
  image.serial: "20201014_07:42"
  image.type: squashfs
  volatile.base_image: e081c41658861afc947610b82c67ee2f423116bfd1e8f4153f0e171b69250127
  volatile.eth0.host_name: vethe88da6c8
  volatile.eth0.hwaddr: 00:16:3e:e5:e2:b7
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

And trying to decipher my networking - using netplan, this is the only file - I see no evidence elsewhere of the bridge, although I’m definitely not experienced here, so I’m probably looking in the wrong places.

$ cat /etc/netplan/01-network-manager-all.yaml
---
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    eth0:
      dhcp4: true
      dhcp6: true

lastly, iptables:

$ iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       icmp -f  *      *       0.0.0.0/0            0.0.0.0/0            /* 000 drop fragmented icmp */
    7   488 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            /* 001 accept any unfragmented icmp */
 1149  110K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0            /* 002 accept all to lo interface */
    0     0 REJECT     all  --  !lo    *       0.0.0.0/0            127.0.0.0/8          /* 003 reject local traffic not on loopback interface */ reject-with icmp-port-unreachable
 2094 8475K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED /* 005 accept related established rules */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22 /* 006 allow SSH access */
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport sports 67 multiport dports 68 /* 007 allow DHCP IPv4 */
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            224.0.0.251          multiport dports 5353 /* 008 allow MULTICAST mDNS */
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            239.255.255.250      multiport dports 1900 /* 009 allow MULTICAST UPnP */
    0     0 LOG_ACCEPT  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25 /* 800 TEST SMTP BLOCK */
37302 4400K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 999 accept all */

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 107 packets, 656K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain LOG_ACCEPT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 100 log accept traffic */ LOG flags 0 level 6 prefix "INPUT:ACCEPT:"
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 101 accept */

Chain LOG_DROP (7 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 1/sec burst 5 /* 100 log drop traffic */ LOG flags 0 level 4 prefix "INPUT:DROP:"
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 101 drop */

Chain RATE_LIMIT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: up to 25/sec burst 50 mode srcip /* 000 throttle a SYN-Flood */
    0     0 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 1/sec burst 5 /* 001 log DOS Attack/SYN scan */ LOG flags 0 level 6 prefix "[DOS/SYN SCAN DROPPED] "

Chain TCP_FLAGS (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LOG_DROP   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x3F/0x00 /* 000 All of the TCP flag bits are clear */
    0     0 LOG_DROP   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x03/0x03 /* 001 TCP flags SYN and FIN both set */
    0     0 LOG_DROP   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x06 /* 002 TCP flags SYN and RST are both set */
    0     0 LOG_DROP   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x05/0x05 /* 003 TCP flags FIN and RST are both set */
    0     0 LOG_DROP   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x11/0x01 /* 004 TCP flag FIN is only bit set, without ACK */
    0     0 LOG_DROP   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x18/0x08 /* 005 TCP flag PSH is only bit set, without ACK */
    0     0 LOG_DROP   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x30/0x20 /* 006 TCP flag URG is only bit set, without ACK */
    0     0 RATE_LIMIT  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x16/0x02 /* 007 Filtering for SYN-Flood */

any help highly appreciated!

edit - from within a container:

# systemd-resolve www.archive.org
www.archive.org: 207.241.224.2
-- Information acquired via protocol DNS in 164.8ms.
-- Data is authenticated: no

# ping www.archive.org
PING www.archive.org (207.241.224.2) 56(84) bytes of data.

^C
--- www.archive.org ping statistics ---
86 packets transmitted, 0 received, 100% packet loss, time 87021ms

from host:

$ ip a show
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:86:c0:8c brd ff:ff:ff:ff:ff:ff
    inet 152.83.108.53/21 brd 152.83.111.255 scope global dynamic noprefixroute eth0
       valid_lft 74919sec preferred_lft 74919sec
    inet6 2405:b000:601:17::108:53/128 scope global dynamic noprefixroute 
       valid_lft 85723sec preferred_lft 85723sec
    inet6 fe80::250:56ff:fe86:c08c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:48:61:0d brd ff:ff:ff:ff:ff:ff
    inet 10.47.110.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:e0b7:79dd:cf0d::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe48:610d/64 scope link 
       valid_lft forever preferred_lft forever
5: vethe88da6c8@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 0a:ce:dc:5a:e4:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: veth398aedb4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 7a:61:88:12:57:23 brd ff:ff:ff:ff:ff:ff link-netnsid 1
9: veth59e7fa37@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 4e:c6:d7:f0:5c:ec brd ff:ff:ff:ff:ff:ff link-netnsid 2

turtle0x1 · October 15, 2020, 7:46am

99% sure the network card has to be in Promiscuous mode - which you can only set through vcenter / vsphere (i had a similar issue but 2/3 years ago now so hard to remeber)

I was setting up a bridge though you seem to be doing somethinh else

tomp · October 15, 2020, 8:15am

In this case, it doesn’t appear you are trying to connect the containers to the external network via a bridge (where the MAC filtering issue you’ve read about would become an issue).

Instead it looks like you’re just using a normal private bridge lxdbr0 and will have outbound traffic from the containers to the external network NATted to the IP address (and importantly MAC address) of the VM itself. In which case MAC filtering shouldn’t be an issue, and its likely something else.

Please can you show the output of ip a and ip r from both the LXD host and inside the container?

Also, when you do a ping from the container to the external network, can you run a sudo tcpdump -nn -e -i eth0 on the LXD host at the same time and see if you see the packets leaving the external interface with the MAC address and source IP of the eth0 interface?

cjchristopher · October 15, 2020, 10:36am

Indeed, this is what I was trying to do.
This started as a basically clean 18.04 installation, which is running snap version 4.6 of both client and server.
All that I’ve really done is run lxd initwith almost all defaults (except answering yes when asked about exposing lxd to the internet), and then create a few 18.04 containers which I’ve then tried to install some packages on.

So, requested info:
On the host:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:86:c0:8c brd ff:ff:ff:ff:ff:ff
    inet 152.83.108.53/21 brd 152.83.111.255 scope global dynamic noprefixroute eth0
       valid_lft 62468sec preferred_lft 62468sec
    inet6 2405:b000:601:17::108:53/128 scope global dynamic noprefixroute 
       valid_lft 85872sec preferred_lft 85872sec
    inet6 fe80::250:56ff:fe86:c08c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:48:61:0d brd ff:ff:ff:ff:ff:ff
    inet 10.47.110.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:e0b7:79dd:cf0d::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe48:610d/64 scope link 
       valid_lft forever preferred_lft forever
5: vethe88da6c8@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 0a:ce:dc:5a:e4:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: veth398aedb4@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 7a:61:88:12:57:23 brd ff:ff:ff:ff:ff:ff link-netnsid 1
9: veth59e7fa37@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 4e:c6:d7:f0:5c:ec brd ff:ff:ff:ff:ff:ff link-netnsid 2

$ ip r
default via 152.83.104.1 dev eth0 proto dhcp metric 100 
10.47.110.0/24 dev lxdbr0 proto kernel scope link src 10.47.110.1 
152.83.104.0/21 dev eth0 proto kernel scope link src 152.83.108.53 metric 100

and inside a container:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
4: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:e5:e2:b7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.47.110.6/24 brd 10.47.110.255 scope global dynamic eth0
       valid_lft 2407sec preferred_lft 2407sec
    inet6 fd42:e0b7:79dd:cf0d:216:3eff:fee5:e2b7/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3177sec preferred_lft 3177sec
    inet6 fe80::216:3eff:fee5:e2b7/64 scope link 
       valid_lft forever preferred_lft forever

# ip r
default via 10.47.110.1 dev eth0 proto dhcp src 10.47.110.6 metric 100 
10.47.110.0/24 dev eth0 proto kernel scope link src 10.47.110.6 
10.47.110.1 dev eth0 proto dhcp scope link src 10.47.110.6 metric 100

re the packet capture… there is a lot of output - I think mostly internal network traffic given the addresses. I tried grepping for the eth0 ipv4 and got nothing, the ipv6 and saw what looks like mostly internal traffic. When grepping for google’s ipv4 (as resolved inside the container), and ipv6 (as given by the host), I also saw nothing whilst pinging from the container.

tomp · October 15, 2020, 10:45am

Can you show the output of iptables-save on the LXD host please?

cjchristopher · October 15, 2020, 10:57am

$ iptables-save
# Generated by iptables-save v1.6.1
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [3561:8951087]
:LOG_ACCEPT - [0:0]
:LOG_DROP - [0:0]
:RATE_LIMIT - [0:0]
:TCP_FLAGS - [0:0]
-A INPUT -p icmp -f -m comment --comment "000 drop fragmented icmp" -j DROP
-A INPUT -p icmp -m comment --comment "001 accept any unfragmented icmp" -j ACCEPT
-A INPUT -i lo -m comment --comment "002 accept all to lo interface" -j ACCEPT
-A INPUT -d 127.0.0.0/8 ! -i lo -m comment --comment "003 reject local traffic not on loopback interface" -j REJECT --reject-with icmp-port-unreachable
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "005 accept related established rules" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -m comment --comment "006 allow SSH access" -j ACCEPT
-A INPUT -p udp -m multiport --sports 67 -m multiport --dports 68 -m comment --comment "007 allow DHCP IPv4" -j ACCEPT
-A INPUT -d 224.0.0.251/32 -p udp -m multiport --dports 5353 -m comment --comment "008 allow MULTICAST mDNS" -j ACCEPT
-A INPUT -d 239.255.255.250/32 -p udp -m multiport --dports 1900 -m comment --comment "009 allow MULTICAST UPnP" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 25 -m comment --comment "800 TEST SMTP BLOCK" -j LOG_ACCEPT
-A INPUT -m comment --comment "999 accept all" -j ACCEPT
-A LOG_ACCEPT -m comment --comment "100 log accept traffic" -j LOG --log-prefix "INPUT:ACCEPT:" --log-level 6
-A LOG_ACCEPT -m comment --comment "101 accept" -j ACCEPT
-A LOG_DROP -p tcp -m limit --limit 1/sec -m comment --comment "100 log drop traffic" -j LOG --log-prefix "INPUT:DROP:"
-A LOG_DROP -m comment --comment "101 drop" -j DROP
-A RATE_LIMIT -p tcp -m hashlimit --hashlimit-upto 25/sec --hashlimit-burst 50 --hashlimit-mode srcip --hashlimit-name synattack -m comment --comment "000 throttle a SYN-Flood" -j RETURN
-A RATE_LIMIT -p tcp -m limit --limit 1/sec -m comment --comment "001 log DOS Attack/SYN scan" -j LOG --log-prefix "[DOS/SYN SCAN DROPPED] " --log-level 6
-A TCP_FLAGS -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -m comment --comment "000 All of the TCP flag bits are clear" -j LOG_DROP
-A TCP_FLAGS -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -m comment --comment "001 TCP flags SYN and FIN both set" -j LOG_DROP
-A TCP_FLAGS -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -m comment --comment "002 TCP flags SYN and RST are both set" -j LOG_DROP
-A TCP_FLAGS -p tcp -m tcp --tcp-flags FIN,RST FIN,RST -m comment --comment "003 TCP flags FIN and RST are both set" -j LOG_DROP
-A TCP_FLAGS -p tcp -m tcp --tcp-flags FIN,ACK FIN -m comment --comment "004 TCP flag FIN is only bit set, without ACK" -j LOG_DROP
-A TCP_FLAGS -p tcp -m tcp --tcp-flags PSH,ACK PSH -m comment --comment "005 TCP flag PSH is only bit set, without ACK" -j LOG_DROP
-A TCP_FLAGS -p tcp -m tcp --tcp-flags ACK,URG URG -m comment --comment "006 TCP flag URG is only bit set, without ACK" -j LOG_DROP
-A TCP_FLAGS -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -m comment --comment "007 Filtering for SYN-Flood" -j RATE_LIMIT
COMMIT
# Generated by iptables-save v1.6.1
*mangle
:PREROUTING ACCEPT [91265:29229952]
:INPUT ACCEPT [89057:29040070]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [5412:12746189]
:POSTROUTING ACCEPT [5412:12746189]
COMMIT
# Generated by iptables-save v1.6.1
*nat
:PREROUTING ACCEPT [22699:4042583]
:INPUT ACCEPT [20491:3852701]
:OUTPUT ACCEPT [746:65194]
:POSTROUTING ACCEPT [746:65194]
COMMIT

tomp · October 15, 2020, 11:02am

Thats interesting, there are none of the rules that LXD adds there.

Do you use nftables? Can you show the output of nft list ruleset?

tomp · October 15, 2020, 11:02am

The issue is something has wiped the firewall rules that LXD adds, and has removed the NAT rules LXD adds. Try reloading LXD and then run iptables-save, you should see some rules added by LXD.

cjchristopher · October 15, 2020, 11:07am

Well that makes a lot of sense then. I suspect it’s because the VM is also managed via puppet by the cloud administrators. As far as I can tell, nftables has not been used (system prompts me to install it).

systemctl reload snap.lxd.daemon yes? I’ve done that and don’t see any changes to the output of iptables-save unfortunately.

cjchristopher · October 15, 2020, 11:14am

Ah. A full restart of the daemon reinserted the rules, but they are gone again within a minute, it seems.

Edit: seems inconsistent actually. restarted again and it seems to be persisting a little longer…

tomp · October 15, 2020, 11:22am

Yeah, you’ll need to disable puppet or get the admin to add the rules LXD adds to the puppet managed ruleset.

cjchristopher · October 15, 2020, 11:24am

Cheers. Will see if I can convince them to do that for me - have marked a solution!

Thanks much for your assist.

cjchristopher · October 16, 2020, 5:22am

Out of curiosity - where are those lxd firewall rules actually written to? I’m trying to pin down what is being overwritten by puppet.

tomp · October 16, 2020, 7:59am

LXD just adds the rules directly to the running iptables or nftables system. Its not written to disk.

Each time LXD restarts it re-adds them if needed.

So if something else on the system (puppet) is periodically resetting them this would explain why it resets at different intervals.