No DNS on bridge network? advice needed!

JeToJedno · April 20, 2021, 8:47am

I have 2 network: lxdbr0 & lxdfan0 on a small cluster, and I have no DNS on lxdbr0 for no reason I can see. Can someone point me to what to change / investigate.

This was working before the upgrade to 4.13, but I was working on another part of this setup and so can’t be certain whether the only change is the LXD version or whether I inadvertently changed something.

$ for X in lxdbr0 lxdfan0 ; do lxc profile list | grep "${X}" | awk -F '|' '{print $2 " " $3 }' | sed -r -e 's/^\s+//' -e 's/\s+$//' -e 's/\s+/ /g' && for Y in albans ; do lxc launch images:ubuntu/bionic test-${X}-${Y} --target ${Y} -p default $( [ 'default' = "${X}" ] || echo "-p ${X}" ) --ephemeral || break ; done ; done ; sleep 3 && lxc list
lxdbr0 Use LXD bridge network
Creating test-lxdbr0-albans
Starting test-lxdbr0-albans
lxdfan0 Use LXD fan network    
Creating test-lxdfan0-albans
Starting test-lxdfan0-albans
+---------------------+---------+---------------------+------+-----------------------+-----------+----------+
|        NAME         |  STATE  |        IPV4         | IPV6 |         TYPE          | SNAPSHOTS | LOCATION |
+---------------------+---------+---------------------+------+-----------------------+-----------+----------+
| gateway             | STOPPED |                     |      | CONTAINER             | 0         | albans   |
+---------------------+---------+---------------------+------+-----------------------+-----------+----------+
| test-lxdbr0-albans  | RUNNING | 10.1.218.212 (eth0) |      | CONTAINER (EPHEMERAL) | 0         | albans   |
+---------------------+---------+---------------------+------+-----------------------+-----------+----------+
| test-lxdfan0-albans | RUNNING | 240.0.215.93 (eth0) |      | CONTAINER (EPHEMERAL) | 0         | albans   |
+---------------------+---------+---------------------+------+-----------------------+-----------+----------+



$ K='ping' ; for X in $( lxc list '^test-' | tail --lines=+3 | grep '^|' | awk -F '|' '{print $2}' ) ; do echo "$X $( lxc config show ${X} | yq -j -I 0 e '.profiles' - ) [$( lxc exec ${X} -- ip -4 -o address show dev eth0 | awk '{print $4}' | tr '[:space:]' ' ' | sed -r -e 's/^\s+//' -e 's/\s+$//' -e 's/\s+/,/g' )]" | logger --stderr --tag "lxd-test-$K" 2>&1 ; Z="$( dig +short www.google.com | head --lines=1 )" ; ( lxc exec ${X} -- systemd-resolve www.google.com && lxc exec ${X} -- ping -c3 -n -q -A -w1 "${Z}" && lxc exec ${X} -- ping -c3 -n -q -A -w1 www.google.com ) 2>&1 | logger --skip-empty --stderr --tag "lxd-test-$K" 2>&1 ; done
<13>Apr 20 08:24:25 lxd-test-ping: test-lxdbr0-albans ["default","lxdbr0"] [10.1.218.212/16]
<13>Apr 20 08:24:26 lxd-test-ping: www.google.com: resolve call failed: All attempts to contact name servers or networks failed
<13>Apr 20 08:26:22 lxd-test-ping: test-lxdfan0-albans ["default","lxdfan0"] [240.0.215.93/8]
<13>Apr 20 08:26:22 lxd-test-ping: www.google.com: 216.239.38.120
<13>Apr 20 08:26:22 lxd-test-ping: -- Information acquired via protocol DNS in 2.2ms.
<13>Apr 20 08:26:22 lxd-test-ping: -- Data is authenticated: no
<13>Apr 20 08:26:23 lxd-test-ping: PING 216.239.38.120 (216.239.38.120) 56(84) bytes of data.
<13>Apr 20 08:26:23 lxd-test-ping: --- 216.239.38.120 ping statistics ---
<13>Apr 20 08:26:23 lxd-test-ping: 3 packets transmitted, 3 received, 0% packet loss, time 14ms
<13>Apr 20 08:26:23 lxd-test-ping: rtt min/avg/max/mdev = 5.973/6.400/6.801/0.344 ms, ipg/ewma 7.009/6.663 ms
<13>Apr 20 08:26:23 lxd-test-ping: PING www.google.com (216.239.38.120) 56(84) bytes of data.
<13>Apr 20 08:26:23 lxd-test-ping: --- www.google.com ping statistics ---
<13>Apr 20 08:26:23 lxd-test-ping: 3 packets transmitted, 3 received, 0% packet loss, time 12ms
<13>Apr 20 08:26:23 lxd-test-ping: rtt min/avg/max/mdev = 6.170/6.437/6.708/0.238 ms, ipg/ewma 6.341/6.266 ms

$ lxc network show lxdbr0 && echo && lxc profile show lxdbr0
config:
  ipv4.address: 10.1.0.1/16
  ipv4.firewall: "false"
  ipv4.nat: "true"
  ipv6.address: none
description: Default local LXD network
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/test-lxdbr0-albans
- /1.0/profiles/lxdbr0
managed: true
status: Created

config: {}
description: Use LXD bridge network
devices:
  eth0:
    name: eth0
    network: lxdbr0
    security.mac_filtering: "true"
    type: nic
name: lxdbr0
used_by:
- /1.0/instances/test-lxdbr0-albans

$ lxc network show lxdfan0 && echo && lxc profile show lxdfan0
config:
  bridge.mode: fan
  dns.domain: lxd
  dns.mode: managed
  fan.overlay_subnet: 240.0.0.0/8
  fan.underlay_subnet: 10.2.0.0/16
  ipv4.firewall: "false"
  ipv4.nat: "true"
description: Default LXC cluster network
name: lxdfan0
type: bridge
used_by:
- /1.0/instances/gateway
- /1.0/instances/test-lxdfan0-albans
- /1.0/profiles/default
- /1.0/profiles/gateway-create
- /1.0/profiles/lxdfan0
managed: true
status: Created

config: {}
description: Use LXD fan network
devices:
  eth0:
    name: eth0
    network: lxdfan0
    security.mac_filtering: "true"
    type: nic
name: lxdfan0
used_by:
- /1.0/instances/test-lxdfan0-albans

$ ip l && echo && ip a && echo && ip r
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: dmz0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether b8:a3:86:70:cc:e6 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether f0:4d:a2:25:df:d0 brd ff:ff:ff:ff:ff:ff
5: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:71:69:b0 brd ff:ff:ff:ff:ff:ff
6: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ee:93:aa:ec:4b:1b brd ff:ff:ff:ff:ff:ff
7: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fa:e5:11:3a:aa:ee brd ff:ff:ff:ff:ff:ff
8: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:4e:67:17 brd ff:ff:ff:ff:ff:ff
10: veth6241a9ef@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 12:ca:46:df:56:32 brd ff:ff:ff:ff:ff:ff link-netnsid 0
12: vethedb2b39b@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP mode DEFAULT group default qlen 1000
    link/ether 3e:20:15:fc:3b:eb brd ff:ff:ff:ff:ff:ff link-netnsid 1

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: dmz0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether b8:a3:86:70:cc:e6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.157.215/24 brd 192.168.157.255 scope global dmz0
       valid_lft forever preferred_lft forever
    inet 10.1.0.215/16 brd 10.1.255.255 scope global dmz0
       valid_lft forever preferred_lft forever
    inet 10.2.0.215/16 brd 10.2.255.255 scope global dmz0
       valid_lft forever preferred_lft forever
    inet 10.3.0.215/16 brd 10.3.255.255 scope global dmz0
       valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether f0:4d:a2:25:df:d0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.57.215/24 brd 192.168.57.255 scope global eth0
       valid_lft forever preferred_lft forever
5: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:71:69:b0 brd ff:ff:ff:ff:ff:ff
    inet 240.0.215.1/8 scope global lxdfan0
       valid_lft forever preferred_lft forever
6: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether ee:93:aa:ec:4b:1b brd ff:ff:ff:ff:ff:ff
7: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether fa:e5:11:3a:aa:ee brd ff:ff:ff:ff:ff:ff
8: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:4e:67:17 brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.1/16 scope global lxdbr0
       valid_lft forever preferred_lft forever
10: veth6241a9ef@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 12:ca:46:df:56:32 brd ff:ff:ff:ff:ff:ff link-netnsid 0
12: vethedb2b39b@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether 3e:20:15:fc:3b:eb brd ff:ff:ff:ff:ff:ff link-netnsid 1

default via 192.168.157.253 dev dmz0 proto static
10.1.0.0/16 dev dmz0 proto kernel scope link src 10.1.0.215
10.1.0.0/16 dev lxdbr0 proto kernel scope link src 10.1.0.1
10.2.0.0/16 dev dmz0 proto kernel scope link src 10.2.0.215
10.3.0.0/16 dev dmz0 proto kernel scope link src 10.3.0.215
192.168.57.0/24 dev eth0 proto kernel scope link src 192.168.57.215
192.168.157.0/24 dev dmz0 proto kernel scope link src 192.168.157.215
240.0.0.0/8 dev lxdfan0 proto kernel scope link src 240.0.215.1

$ ps -Aly x | grep '\bdnsmasq\b'
S   105    2523    1758  0  80   0  3724 12493 -      ?          0:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdfan0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=240.0.215.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdfan0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdfan0/dnsmasq.hosts --dhcp-range 240.0.215.2,240.0.215.254,1h -s lxd -S /lxd/240.0.215.1#1053 --rev-server=240.0.0.0/8,240.0.215.1#1053 --conf-file=/var/snap/lxd/common/lxd/networks/lxdfan0/dnsmasq.raw -u lxd -g lxd
S   105    3508    1758  0  80   0  3748 12493 -      ?          0:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.1.0.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.1.0.2,10.1.255.254,1h -s lxd -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd

both dnsmasq.raw files are empty.

# journalctl -b -o short-precise -e
Apr 20 08:24:25.509070 albans systemd[1816]: Started snap.lxd.lxc.666f947a-bb2f-4e4e-9a6d-5d393f05e066.scope.
Apr 20 08:24:25.633464 albans systemd[1816]: snap.lxd.lxc.666f947a-bb2f-4e4e-9a6d-5d393f05e066.scope: Succeeded.
Apr 20 08:24:25.694527 albans systemd[1816]: Started snap.lxd.lxc.d52f2592-da66-42c6-8846-e0fa1aa58de0.scope.
Apr 20 08:24:25.709138 albans systemd[1816]: Started snap.yq.yq.d9be6454-50c9-4f18-92c7-c58bad95b882.scope.
Apr 20 08:24:25.714000 albans audit[5431]: AVC apparmor="DENIED" operation="capable" profile="/snap/snapd/11588/usr/lib/snapd/snap-confine" pid=5431 comm="snap-confine" capability=4  capname="fsetid"
Apr 20 08:24:25.718724 albans systemd[1]: tmp-snap.rootfs_gY7LeR.mount: Succeeded.
Apr 20 08:24:25.720685 albans kernel: audit: type=1400 audit(1618907065.714:50): apparmor="DENIED" operation="capable" profile="/snap/snapd/11588/usr/lib/snapd/snap-confine" pid=5431 comm="snap-confine" capability=4  capname="fsetid"
Apr 20 08:24:25.724819 albans systemd[1816]: tmp-snap.rootfs_gY7LeR.mount: Succeeded.
Apr 20 08:24:25.780808 albans systemd[1816]: snap.lxd.lxc.d52f2592-da66-42c6-8846-e0fa1aa58de0.scope: Succeeded.
Apr 20 08:24:25.933296 albans systemd[1816]: snap.yq.yq.d9be6454-50c9-4f18-92c7-c58bad95b882.scope: Succeeded.
Apr 20 08:24:25.980640 albans systemd[1816]: Started snap.lxd.lxc.be1addb1-c925-4637-8c60-3a6daf9fce23.scope.
Apr 20 08:24:26.181721 albans systemd[1816]: snap.lxd.lxc.be1addb1-c925-4637-8c60-3a6daf9fce23.scope: Succeeded.
Apr 20 08:24:26.182785 albans lxd-test-ping[5428]: test-lxdbr0-albans ["default","lxdbr0"] [10.1.218.212/16]
Apr 20 08:24:26.280634 albans systemd[1816]: Started snap.lxd.lxc.e3284d34-1fa2-493f-ab10-71e7d60ee2e1.scope.
Apr 20 08:24:26.412514 albans kernel: 'lxd: 'IN=lxdbr0 OUT= PHYSIN=veth6241a9ef MAC=00:16:3e:4e:67:17:00:16:3e:2a:20:21:08:00 SRC=10.1.218.212 DST=10.1.0.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=64286 DF PROTO=UDP SPT=46048 DPT=53 LEN=51 
Apr 20 08:24:26.412793 albans kernel: 'lxd: 'IN=lxdbr0 OUT= PHYSIN=veth6241a9ef MAC=00:16:3e:4e:67:17:00:16:3e:2a:20:21:08:00 SRC=10.1.218.212 DST=10.1.0.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=64286 DF PROTO=UDP SPT=46048 DPT=53 LEN=51 
Apr 20 08:24:26.412844 albans kernel: 'lxd: 'IN=lxdbr0 OUT= PHYSIN=veth6241a9ef MAC=00:16:3e:4e:67:17:00:16:3e:2a:20:21:08:00 SRC=10.1.218.212 DST=10.1.0.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=64286 DF PROTO=UDP SPT=46048 DPT=53 LEN=51 
Apr 20 08:24:26.412886 albans kernel: 'lxd: 'IN=lxdbr0 OUT= PHYSIN=veth6241a9ef MAC=00:16:3e:4e:67:17:00:16:3e:2a:20:21:08:00 SRC=10.1.218.212 DST=10.1.0.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=64286 DF PROTO=UDP SPT=46048 DPT=53 LEN=51 
Apr 20 08:24:30.640512 albans kernel: rpfilter_DROP: IN=lxdbr0 OUT= PHYSIN=veth6241a9ef MAC=33:33:00:00:00:02:00:16:3e:2a:20:21:86:dd SRC=fe80:0000:0000:0000:0216:3eff:fe2a:2021 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=261257 PROTO=ICMPv6 TYPE=133 CODE=0 
Apr 20 08:24:41.196525 albans kernel: rpfilter_DROP: IN=lxdfan0 OUT= PHYSIN=vethedb2b39b MAC=33:33:00:00:00:02:00:16:3e:4c:9f:23:86:dd SRC=fe80:0000:0000:0000:0216:3eff:fe4c:9f23 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=249043 PROTO=ICMPv6 TYPE=133 CODE=0 
Apr 20 08:25:34.120506 albans kernel: rpfilter_DROP: IN=lxdbr0 OUT= PHYSIN=veth6241a9ef MAC=33:33:00:00:00:02:00:16:3e:2a:20:21:86:dd SRC=fe80:0000:0000:0000:0216:3eff:fe2a:2021 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=261257 PROTO=ICMPv6 TYPE=133 CODE=0 
Apr 20 08:25:49.700500 albans kernel: rpfilter_DROP: IN=lxdfan0 OUT= PHYSIN=vethedb2b39b MAC=33:33:00:00:00:02:00:16:3e:4c:9f:23:86:dd SRC=fe80:0000:0000:0000:0216:3eff:fe4c:9f23 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=249043 PROTO=ICMPv6 TYPE=133 CODE=0 
Apr 20 08:26:22.444340 albans lxd-test-ping[5538]: www.google.com: resolve call failed: All attempts to contact name servers or networks failed
Apr 20 08:26:22.452466 albans systemd[1816]: snap.lxd.lxc.e3284d34-1fa2-493f-ab10-71e7d60ee2e1.scope: Succeeded.
Apr 20 08:26:22.511613 albans systemd[1816]: Started snap.lxd.lxc.b002bfe5-b789-4593-a086-17978e962938.scope.
Apr 20 08:26:22.519423 albans systemd[1816]: Started snap.yq.yq.d8c644be-47b0-4bca-b7ee-3041472cb0ec.scope.
Apr 20 08:26:22.583549 albans systemd[1816]: snap.lxd.lxc.b002bfe5-b789-4593-a086-17978e962938.scope: Succeeded.
Apr 20 08:26:22.585120 albans systemd[1816]: snap.yq.yq.d8c644be-47b0-4bca-b7ee-3041472cb0ec.scope: Succeeded.
Apr 20 08:26:22.632059 albans systemd[1816]: Started snap.lxd.lxc.c62f4e41-9330-4891-aead-92a8fea0bca6.scope.
Apr 20 08:26:22.779049 albans systemd[1816]: snap.lxd.lxc.c62f4e41-9330-4891-aead-92a8fea0bca6.scope: Succeeded.
Apr 20 08:26:22.781259 albans lxd-test-ping[5665]: test-lxdfan0-albans ["default","lxdfan0"] [240.0.215.93/8]
Apr 20 08:26:22.844567 albans systemd[1816]: Started snap.lxd.lxc.720eaf5b-285d-4770-9626-24c8b00d1d5c.scope.
Apr 20 08:26:22.994073 albans kernel: 'lxd: 'IN=lxdfan0 OUT= PHYSIN=vethedb2b39b MAC=00:16:3e:71:69:b0:00:16:3e:4c:9f:23:08:00 SRC=240.0.215.93 DST=240.0.215.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=3372 DF PROTO=UDP SPT=57247 DPT=53 LEN=51 
Apr 20 08:26:22.994283 albans kernel: 'lxd: 'IN=lxdfan0 OUT= PHYSIN=vethedb2b39b MAC=00:16:3e:71:69:b0:00:16:3e:4c:9f:23:08:00 SRC=240.0.215.93 DST=240.0.215.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=3372 DF PROTO=UDP SPT=57247 DPT=53 LEN=51 
Apr 20 08:26:22.994964 albans lxd-test-ping[5768]: www.google.com: 216.239.38.120
Apr 20 08:26:22.995016 albans lxd-test-ping[5768]: -- Information acquired via protocol DNS in 2.2ms.
Apr 20 08:26:22.995029 albans lxd-test-ping[5768]: -- Data is authenticated: no
Apr 20 08:26:23.001477 albans systemd[1816]: snap.lxd.lxc.720eaf5b-285d-4770-9626-24c8b00d1d5c.scope: Succeeded.
Apr 20 08:26:23.045971 albans systemd[1816]: Started snap.lxd.lxc.c3bc56fb-32ff-499e-af6a-7f6f62c93ddb.scope.
Apr 20 08:26:23.270849 albans lxd-test-ping[5768]: PING 216.239.38.120 (216.239.38.120) 56(84) bytes of data.
Apr 20 08:26:23.284625 albans lxd-test-ping[5768]: --- 216.239.38.120 ping statistics ---
Apr 20 08:26:23.284658 albans lxd-test-ping[5768]: 3 packets transmitted, 3 received, 0% packet loss, time 14ms
Apr 20 08:26:23.284676 albans lxd-test-ping[5768]: rtt min/avg/max/mdev = 5.973/6.400/6.801/0.344 ms, ipg/ewma 7.009/6.663 ms
Apr 20 08:26:23.294136 albans systemd[1816]: snap.lxd.lxc.c3bc56fb-32ff-499e-af6a-7f6f62c93ddb.scope: Succeeded.
Apr 20 08:26:23.338563 albans systemd[1816]: Started snap.lxd.lxc.f11c05f8-7adb-4251-80e2-2291e460ad82.scope.
Apr 20 08:26:23.467389 albans kernel: 'lxd: 'IN=lxdfan0 OUT= PHYSIN=vethedb2b39b MAC=00:16:3e:71:69:b0:00:16:3e:4c:9f:23:08:00 SRC=240.0.215.93 DST=240.0.215.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=3392 DF PROTO=UDP SPT=53913 DPT=53 LEN=51 
Apr 20 08:26:23.467558 albans kernel: 'lxd: 'IN=lxdfan0 OUT= PHYSIN=vethedb2b39b MAC=00:16:3e:71:69:b0:00:16:3e:4c:9f:23:08:00 SRC=240.0.215.93 DST=240.0.215.1 LEN=71 TOS=0x00 PREC=0x00 TTL=64 ID=3392 DF PROTO=UDP SPT=53913 DPT=53 LEN=51 
Apr 20 08:26:23.474618 albans lxd-test-ping[5768]: PING www.google.com (216.239.38.120) 56(84) bytes of data.
Apr 20 08:26:23.487918 albans lxd-test-ping[5768]: --- www.google.com ping statistics ---
Apr 20 08:26:23.487959 albans lxd-test-ping[5768]: 3 packets transmitted, 3 received, 0% packet loss, time 12ms
Apr 20 08:26:23.487980 albans lxd-test-ping[5768]: rtt min/avg/max/mdev = 6.170/6.437/6.708/0.238 ms, ipg/ewma 6.341/6.266 ms
Apr 20 08:26:23.494916 albans systemd[1816]: snap.lxd.lxc.f11c05f8-7adb-4251-80e2-2291e460ad82.scope: Succeeded.
Apr 20 08:27:42.000508 albans kernel: rpfilter_DROP: IN=lxdbr0 OUT= PHYSIN=veth6241a9ef MAC=33:33:00:00:00:02:00:16:3e:2a:20:21:86:dd SRC=fe80:0000:0000:0000:0216:3eff:fe2a:2021 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=261257 PROTO=ICMPv6 TYPE=133 CODE=0 
Apr 20 08:28:06.232502 albans kernel: rpfilter_DROP: IN=lxdfan0 OUT= PHYSIN=vethedb2b39b MAC=33:33:00:00:00:02:00:16:3e:4c:9f:23:86:dd SRC=fe80:0000:0000:0000:0216:3eff:fe4c:9f23 DST=ff02:0000:0000:0000:0000:0000:0000:0002 LEN=56 TC=0 HOPLIMIT=255 FLOWLBL=249043 PROTO=ICMPv6 TYPE=133 CODE=0 
Apr 20 08:29:00.197170 albans systemd[1816]: Started snap.lxd.lxc.bede0ba2-8c6d-4e55-8d0f-bdc18373d7c8.scope.
Apr 20 08:29:00.308959 albans systemd[1816]: snap.lxd.lxc.bede0ba2-8c6d-4e55-8d0f-bdc18373d7c8.scope: Succeeded.
Apr 20 08:29:25.076288 albans systemd[1816]: Started snap.lxd.lxc.0ba2a903-3c22-4926-98fc-17e05a8a6a16.scope.
Apr 20 08:29:25.122349 albans systemd[1816]: snap.lxd.lxc.0ba2a903-3c22-4926-98fc-17e05a8a6a16.scope: Succeeded.
Apr 20 08:29:46.098589 albans systemd[1816]: Started snap.lxd.lxc.70b11bc8-64e6-48cc-ac29-01c64ecfcdb3.scope.
Apr 20 08:29:46.206467 albans systemd[1816]: snap.lxd.lxc.70b11bc8-64e6-48cc-ac29-01c64ecfcdb3.scope: Succeeded.

# firewall-cmd --info-zone=lxd
lxd (active)
  target: default
  icmp-block-inversion: no
  interfaces: lxdbr0 lxdfan0
  sources: 10.1.0.1/16 10.1.0.215/16 10.2.0.0/16 240.0.215.1/8
  services: dhcp dns dns-over-tls ssh
  ports:
  protocols:
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
        rule priority="-20" family="ipv4" source address="192.168.57.0/24" log prefix="lxd.drop: " level="warning" limit value="5/s" drop
        rule priority="-20" family="ipv4" destination address="192.168.57.0/24" log prefix="lxd.drop: " level="warning" limit value="5/s" drop
        rule priority="-20" family="ipv4" source address="192.168.4.16/24" log prefix="lxd.drop: " level="warning" limit value="5/s" drop
        rule priority="-20" family="ipv4" destination address="192.168.4.16/24" log prefix="lxd.drop: " level="warning" limit value="5/s" drop
        rule priority="-20" family="ipv4" source address="192.168.157.0/24" log prefix="lxd.drop: " level="warning" limit value="5/s" drop
        rule priority="-20" family="ipv4" destination address="192.168.157.0/24" log prefix="lxd.drop: " level="warning" limit value="5/s" drop
        rule priority="-10" family="ipv4" source address="10.1.0.0/16" destination address="10.1.0.0/16" port port="8443" protocol="tcp" accept
        rule priority="-10" family="ipv4" source address="10.1.0.0/16" destination address="10.1.0.0/16" source-port port="8443" protocol="tcp" accept
        rule priority="-10" family="ipv4" source NOT address="10.1.0.0/16" destination address="10.1.0.0/16" port port="8443" protocol="tcp" log prefix="lxd.drop: " level="warning" limit value="5/s" drop
        rule priority="-10" family="ipv4" source address="10.1.0.0/16" destination not address="10.1.0.0/16" port port="8443" protocol="tcp" log prefix="lxd.drop: " level="warning" limit value="5/s" drop
        rule priority="-10" family="ipv4" protocol value="icmp" log prefix="lxd.icmp: " level="debug" limit value="5/s" accept
        rule family="ipv4" service name="dhcp" accept
        rule family="ipv4" service name="dns" accept
        rule family="ipv4" service name="dns-over-tls" accept
        rule family="ipv4" service name="ssh" accept
        rule family="ipv4" source address="10.1.0.1/16" destination address="10.1.0.1/16" accept
        rule family="ipv4" source address="10.1.0.1/16" log prefix="lxd: " level="debug" limit value="5/s" accept
        rule family="ipv4" destination address="10.1.0.1/16" log prefix="lxd: " level="debug" limit value="5/s" accept
        rule family="ipv4" source address="10.1.0.215/16" destination address="10.1.0.215/16" accept
        rule family="ipv4" source address="10.1.0.215/16" log prefix="lxd: " level="debug" limit value="5/s" accept
        rule family="ipv4" destination address="10.1.0.215/16" log prefix="lxd: " level="debug" limit value="5/s" accept
        rule family="ipv4" source address="10.2.0.0/16" destination address="10.2.0.0/16" accept
        rule family="ipv4" source address="10.2.0.0/16" log prefix="lxd: " level="debug" limit value="5/s" accept
        rule family="ipv4" destination address="10.2.0.0/16" log prefix="lxd: " level="debug" limit value="5/s" accept
        rule family="ipv4" source address="240.0.215.1/8" destination address="240.0.215.1/8" accept
        rule family="ipv4" source address="240.0.215.1/8" log prefix="lxd: " level="debug" limit value="5/s" accept
        rule family="ipv4" destination address="240.0.215.1/8" log prefix="lxd: " level="debug" limit value="5/s" accept

# sysctl -a | grep -e '\brp_' -e martian | grep -v -e eth
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.dmz0.log_martians = 1
net.ipv4.conf.dmz0.rp_filter = 2
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.lxdbr0.log_martians = 1
net.ipv4.conf.lxdbr0.rp_filter = 2
net.ipv4.conf.lxdfan0.log_martians = 1
net.ipv4.conf.lxdfan0.rp_filter = 2
net.ipv4.conf.lxdfan0-fan.log_martians = 0
net.ipv4.conf.lxdfan0-fan.rp_filter = 1
net.ipv4.conf.lxdfan0-mtu.log_martians = 0
net.ipv4.conf.lxdfan0-mtu.rp_filter = 1

tomp · April 20, 2021, 9:00am

Can you show output of this on host:

sudo ss -ulpn | grep 53

JeToJedno · April 20, 2021, 10:55am

UNCONN    0         0                   10.1.0.1:53              0.0.0.0:*       users:(("dnsmasq",pid=3508,fd=6))                                              
UNCONN    0         0                240.0.215.1:53              0.0.0.0:*       users:(("dnsmasq",pid=2523,fd=6))                                              
UNCONN    0         0              127.0.0.53%lo:53              0.0.0.0:*       users:(("systemd-resolve",pid=1388,fd=12))

DHCP is working in the bridge network as the container received an IP Address.

tomp · April 20, 2021, 10:58am

Can you try doing a dig @10.1.0.1 linuxcontainers.org inside an instance connected to lxdbr0, while at the same time doing a sudo tcpdump -nn -i lxdbr0 port 53 and see if you can see the DNS requests coming into lxdbr0 on the host?

JeToJedno · April 20, 2021, 11:08am

lxc exec test-lxdbr0-albans -- dig @10.1.0.1 linuxcontainers.org produced nothing in the tcpdump
lxc exec test-lxdbr0-albans -- systemd-resolve linuxcontainers.org produced a sequence of 11:05:38.412809 IP 10.1.218.212.33613 > 10.1.0.1.53: 64466+ A? linuxcontainers.org. (37) spaced at 5 sec intervals.

tomp · April 20, 2021, 11:10am

That’s pretty weird. And with the systemd-resolve, was that TCP or UDP traffic to 10.1.0.1?

tomp · April 20, 2021, 11:10am

Do you have dig installed in the container?

JeToJedno · April 20, 2021, 11:11am

I’m not that familiar with tcpdump … that line I pasted was verbatim, so I’m assuming tcp as it has no udp indicators.

tomp · April 20, 2021, 11:12am

Can you turn off your firewall and confirm that isn’t causing the problem.

JeToJedno · April 20, 2021, 11:13am

It’s from images:ubuntu/bionic, so no. A very minimal set of tools.

tomp · April 20, 2021, 11:15am

That’ll be why you got no output and no traffic then

JeToJedno · April 20, 2021, 11:16am

with no firewall,

$ lxc exec test-lxdbr0-albans -- systemd-resolve linuxcontainers.org
linuxcontainers.org: resolve call failed: Lookup failed due to system error: Connection timed out

give tcpdump output:

# tcpdump -nn -i lxdbr0 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:15:17.072069 IP 10.1.218.212.54102 > 10.1.0.1.53: Flags [S], seq 1529130747, win 64240, options [mss 1460,sackOK,TS val 1341401303 ecr 0,nop,wscale 7], length 0
11:15:18.084491 IP 10.1.218.212.54102 > 10.1.0.1.53: Flags [S], seq 1529130747, win 64240, options [mss 1460,sackOK,TS val 1341402315 ecr 0,nop,wscale 7], length 0
11:15:20.100500 IP 10.1.218.212.54102 > 10.1.0.1.53: Flags [S], seq 1529130747, win 64240, options [mss 1460,sackOK,TS val 1341404331 ecr 0,nop,wscale 7], length 0
11:15:24.356498 IP 10.1.218.212.54102 > 10.1.0.1.53: Flags [S], seq 1529130747, win 64240, options [mss 1460,sackOK,TS val 1341408587 ecr 0,nop,wscale 7], length 0

tomp · April 20, 2021, 11:17am

OK lets get dig installed in the instance and then we can try querying an upstream DNS resolver (such as 8.8.8.8) using dig @resolverIP linuxcontainers.org just to check there isn’t some upstream issue in your network.

tomp · April 20, 2021, 11:18am

Also with the firewall off, please confirm no firewall with sudo iptables-save and sudo nft list ruleset. Thanks

JeToJedno · April 20, 2021, 11:20am

sudo iptables-save
No output
sudo nft list ruleset
sudo: nft: command not found

JeToJedno · April 20, 2021, 11:22am

Give me 1 hour (or so - meeting looming, not to do with this) and I’ll recreate the containers with the ubuntu:bionic images - they, I think, have a full set of tools.

JeToJedno · April 20, 2021, 11:22am

and thanks.

tomp · April 20, 2021, 11:26am

You can just do apt install dnsutils too.

JeToJedno · April 20, 2021, 12:18pm

won’t work No DNS!

New containers created from ubuntu:bionic. They have dnsutils included out-of-the-box, so to speak.

JeToJedno · April 20, 2021, 1:36pm

Trial with dig available and firewall off.

lxc exec test-lxdbr0-albans -- dig @10.1.0.1 linuxcontainers.org

; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @10.1.0.1 linuxcontainers.org
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

tcpdump -nn -i lxdbr0 port 53                                                                                                         tcpdump: verbose output suppressed, use -v or -vv for full protocol decode                                                                                  listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:34:00.928354 IP 10.1.10.182.58513 > 10.1.0.1.53: 54093+ [1au] A? linuxcontainers.org. (60)
13:34:05.928300 IP 10.1.10.182.58513 > 10.1.0.1.53: 54093+ [1au] A? linuxcontainers.org. (60)
13:34:10.928397 IP 10.1.10.182.58513 > 10.1.0.1.53: 54093+ [1au] A? linuxcontainers.org. (60)

lxc exec test-lxdfan0-albans -- dig @10.1.0.1 linuxcontainers.org

; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> @10.1.0.1 linuxcontainers.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44920
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;linuxcontainers.org.           IN      A

;; ANSWER SECTION:
linuxcontainers.org.    772     IN      A       170.39.196.167

;; Query time: 0 msec
;; SERVER: 10.1.0.1#53(10.1.0.1)
;; WHEN: Tue Apr 20 13:36:08 UTC 2021
;; MSG SIZE  rcvd: 64

tcpdump -nn -i lxdfan0 port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lxdfan0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:36:08.005879 IP 240.0.215.28.47872 > 10.1.0.1.53: 44920+ [1au] A? linuxcontainers.org. (60)
13:36:08.006071 IP 10.1.0.1.53 > 240.0.215.28.47872: 44920 1/0/1 A 170.39.196.167 (64)