No internet access in LXD containers on Fedora 34 host (DNS issue? )

I followed the instructions here: Home · ganto/copr-lxc4 Wiki · GitHub to install the COPR for lxd (4.15),

  1. lxc launch images:ubuntu/16.04 (also tried, 18, 20, and fedora 34)
  2. lxc shell charmed-teal
lxc list
+--------------+---------+------+-----------------------------------------------+-----------+-----------+
|     NAME     |  STATE  | IPV4 |                     IPV6                      |   TYPE    | SNAPSHOTS |
+--------------+---------+------+-----------------------------------------------+-----------+-----------+
| charmed-teal | RUNNING |      | fd42:ce9f:b336:915b:216:3eff:feb9:dbd8 (eth0) | CONTAINER | 0         |
+--------------+---------+------+-----------------------------------------------+-----------+-----------+
root@charmed-teal:~# apt update
Err:1 http://archive.ubuntu.com/ubuntu xenial InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 https://esm.ubuntu.com/infra/ubuntu xenial-infra-security InRelease
  Could not resolve host: esm.ubuntu.com
Err:5 https://esm.ubuntu.com/infra/ubuntu xenial-infra-updates InRelease
  Could not resolve host: esm.ubuntu.com
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-security/InRelease  Could not resolve host: esm.ubuntu.com
W: Failed to fetch https://esm.ubuntu.com/infra/ubuntu/dists/xenial-infra-updates/InRelease  Could not resolve host: esm.ubuntu.com
W: Some index files failed to download. They have been ignored, or old ones used instead.

Based on other forum posts, I think this is a DNS issue? I followed the troubleshooting steps suggested:

❯ lxc info | grep 'firewall:'

  firewall: nftables
❯ sudo ss -ulpn
[sudo] password for michael: 
State  Recv-Q Send-Q                       Local Address:Port    Peer Address:Port Process                                     
UNCONN 0      0                               10.59.20.1:5353         0.0.0.0:*     users:(("nxserver.bin",pid=3519,fd=41))    
UNCONN 0      0                            192.168.122.1:5353         0.0.0.0:*     users:(("nxserver.bin",pid=3519,fd=40))    
UNCONN 0      0                              192.168.2.4:5353         0.0.0.0:*     users:(("nxserver.bin",pid=3519,fd=39))    
UNCONN 0      0                            100.109.9.108:5353         0.0.0.0:*     users:(("nxserver.bin",pid=3519,fd=38))    
UNCONN 0      0                            192.168.0.114:5353         0.0.0.0:*     users:(("nxserver.bin",pid=3519,fd=37))    
UNCONN 0      0                                  0.0.0.0:5353         0.0.0.0:*     users:(("nxserver.bin",pid=3519,fd=36))    
UNCONN 0      0                                  0.0.0.0:5353         0.0.0.0:*     users:(("avahi-daemon",pid=2953,fd=15))    
UNCONN 0      0                                  0.0.0.0:5355         0.0.0.0:*     users:(("systemd-resolve",pid=2921,fd=11)) 
UNCONN 0      0                                  0.0.0.0:21841        0.0.0.0:*                                                
UNCONN 0      0                            192.168.0.114:54774        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=74)) 
UNCONN 0      0                               10.59.20.1:41409        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=75)) 
UNCONN 0      0                                  0.0.0.0:41641        0.0.0.0:*     users:(("tailscaled",pid=3359,fd=11))      
UNCONN 0      0                                127.0.0.1:59371        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=71)) 
UNCONN 0      0                            192.168.0.114:48224        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=79)) 
UNCONN 0      0                                  0.0.0.0:32410        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=80)) 
UNCONN 0      0                                  0.0.0.0:32412        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=91)) 
UNCONN 0      0                                  0.0.0.0:32413        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=69)) 
UNCONN 0      0                                  0.0.0.0:32414        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=83)) 
UNCONN 0      0                                  0.0.0.0:48958        0.0.0.0:*     users:(("Plex Tuner Serv",pid=4186,fd=71)) 
UNCONN 0      0                               10.59.20.1:53           0.0.0.0:*     users:(("dnsmasq",pid=7999,fd=8))          
UNCONN 0      0                            192.168.122.1:53           0.0.0.0:*     users:(("dnsmasq",pid=3877,fd=5))          
UNCONN 0      0                            127.0.0.53%lo:53           0.0.0.0:*     users:(("systemd-resolve",pid=2921,fd=17)) 
UNCONN 0      0                           0.0.0.0%lxdbr0:67           0.0.0.0:*     users:(("dnsmasq",pid=7999,fd=4))          
UNCONN 0      0                           0.0.0.0%virbr0:67           0.0.0.0:*     users:(("dnsmasq",pid=3877,fd=3))          
UNCONN 0      0                                127.0.0.1:323          0.0.0.0:*     users:(("chronyd",pid=3037,fd=6))          
UNCONN 0      0                               10.59.20.1:33260        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=77)) 
UNCONN 0      0                               10.59.20.1:50679        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=84)) 
UNCONN 0      0                                  0.0.0.0:1901         0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=78)) 
UNCONN 0      0                            192.168.0.114:51683        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=73)) 
UNCONN 0      0                                  0.0.0.0:36566        0.0.0.0:*     users:(("avahi-daemon",pid=2953,fd=17))    
UNCONN 0      0                                127.0.0.1:37663        0.0.0.0:*     users:(("Plex Media Serv",pid=3557,fd=72)) 
UNCONN 0      0                                     [::]:5353            [::]:*     users:(("avahi-daemon",pid=2953,fd=16))    
UNCONN 0      0                                     [::]:5355            [::]:*     users:(("systemd-resolve",pid=2921,fd=13)) 
UNCONN 0      0                                     [::]:21841           [::]:*                                                
UNCONN 0      0                                     [::]:41641           [::]:*     users:(("tailscaled",pid=3359,fd=16))      
UNCONN 0      0                                     [::]:44051           [::]:*     users:(("avahi-daemon",pid=2953,fd=18))    
UNCONN 0      0                 [fd42:ce9f:b336:915b::1]:53              [::]:*     users:(("dnsmasq",pid=7999,fd=12))         
UNCONN 0      0        [fe80::216:3eff:fe52:a81d]%lxdbr0:53              [::]:*     users:(("dnsmasq",pid=7999,fd=10))         
UNCONN 0      0                                    [::1]:323             [::]:*     users:(("chronyd",pid=3037,fd=7))          
UNCONN 0      0                              [::]%lxdbr0:547             [::]:*     users:(("dnsmasq",pid=7999,fd=6))   
❯ sudo iptables-save
[sudo] password for michael: 
# Generated by iptables-save v1.8.7 on Wed Jun 30 17:48:12 2021
*nat
:PREROUTING ACCEPT [104:43929]
:INPUT ACCEPT [92:41527]
:OUTPUT ACCEPT [1327:103338]
:POSTROUTING ACCEPT [1321:102611]
:LIBVIRT_PRT - [0:0]
:ts-postrouting - [0:0]
-A POSTROUTING -j LIBVIRT_PRT
-A POSTROUTING -j ts-postrouting
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A ts-postrouting -m mark --mark 0x40000 -j MASQUERADE
COMMIT
# Completed on Wed Jun 30 17:48:12 2021
# Generated by iptables-save v1.8.7 on Wed Jun 30 17:48:12 2021
*mangle
:PREROUTING ACCEPT [100330:133623630]
:INPUT ACCEPT [100330:133623630]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [59245:5487583]
:POSTROUTING ACCEPT [60025:5581121]
:LIBVIRT_PRT - [0:0]
-A POSTROUTING -j LIBVIRT_PRT
-A LIBVIRT_PRT -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Wed Jun 30 17:48:12 2021
# Generated by iptables-save v1.8.7 on Wed Jun 30 17:48:12 2021
*raw
:PREROUTING ACCEPT [100668:133695185]
:OUTPUT ACCEPT [59606:5540210]
COMMIT
# Completed on Wed Jun 30 17:48:12 2021
# Generated by iptables-save v1.8.7 on Wed Jun 30 17:48:12 2021
*security
:INPUT ACCEPT [100594:133681173]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [59606:5540210]
COMMIT
# Completed on Wed Jun 30 17:48:12 2021
# Generated by iptables-save v1.8.7 on Wed Jun 30 17:48:12 2021
*filter
:INPUT ACCEPT [95419:130468799]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [54406:4756153]
:LIBVIRT_FWI - [0:0]
:LIBVIRT_FWO - [0:0]
:LIBVIRT_FWX - [0:0]
:LIBVIRT_INP - [0:0]
:LIBVIRT_OUT - [0:0]
:ts-forward - [0:0]
:ts-input - [0:0]
-A INPUT -j LIBVIRT_INP
-A INPUT -j ts-input
-A FORWARD -j LIBVIRT_FWX
-A FORWARD -j LIBVIRT_FWI
-A FORWARD -j LIBVIRT_FWO
-A FORWARD -j ts-forward
-A OUTPUT -j LIBVIRT_OUT
-A LIBVIRT_FWI -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWO -s 192.168.122.0/24 -i virbr0 -j ACCEPT
-A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable
-A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
-A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
-A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT
-A ts-forward -i tailscale0 -j MARK --set-xmark 0x40000/0xffffffff
-A ts-forward -m mark --mark 0x40000 -j ACCEPT
-A ts-forward -s 100.64.0.0/10 -o tailscale0 -j DROP
-A ts-forward -o tailscale0 -j ACCEPT
-A ts-input -s 100.109.9.108/32 -i lo -j ACCEPT
-A ts-input -s 100.115.92.0/23 ! -i tailscale0 -j RETURN
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
COMMIT
❯ sudo nft list ruleset
table inet lxd {
	chain pstrt.lxdbr0 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 10.59.20.0/24 ip daddr != 10.59.20.0/24 masquerade
		ip6 saddr fd42:ce9f:b336:915b::/64 ip6 daddr != fd42:ce9f:b336:915b::/64 masquerade
	}

	chain fwd.lxdbr0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdbr0" accept
		ip version 4 iifname "lxdbr0" accept
		ip6 version 6 oifname "lxdbr0" accept
		ip6 version 6 iifname "lxdbr0" accept
	}

	chain in.lxdbr0 {
		type filter hook input priority filter; policy accept;
		iifname "lxdbr0" tcp dport 53 accept
		iifname "lxdbr0" udp dport 53 accept
		iifname "lxdbr0" udp dport 67 accept
		iifname "lxdbr0" udp dport 547 accept
	}

	chain out.lxdbr0 {
		type filter hook output priority filter; policy accept;
		oifname "lxdbr0" tcp sport 53 accept
		oifname "lxdbr0" udp sport 53 accept
		oifname "lxdbr0" udp sport 67 accept
		oifname "lxdbr0" udp sport 547 accept
	}
}
❯ lxc network show lxdbr0           
config:
  ipv4.address: 10.59.20.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:ce9f:b336:915b::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/charmed-teal
- /1.0/profiles/default
managed: true
status: Created
locations:
- none
❯ lxc config show charmed-teal --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu xenial amd64 (20210630_07:42)
  image.os: Ubuntu
  image.release: xenial
  image.serial: "20210630_07:42"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 112018147ebcdf37b26f132fdc49fc32f7447576ea33e75a8d3b988e0d42646d
  volatile.eth0.host_name: veth56aab6e5
  volatile.eth0.hwaddr: 00:16:3e:b9:db:d8
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: d1e1e84a-ef38-4da8-90e7-249944e48a2e
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

I tried some suggested troubleshooting steps:

lxc network unset lxdbr0 raw.dnsmasq
sudo ufw allow in on lxdbr0
sudo ufw route allow in on lxdbr0

I also tried disabling all iptables rules/ufw, but I still am getting the same error.

Hmm, so you definitely have dnsmasq running properly as we see it in the process list as well as your container getting an IPv6 address.

There is no obvious issue with the firewalling either. Your other rules aren’t blocking/rejecting anything other than some of their own traffic.

The fact that LXD picked nftables for its firewall could explain the lack of DHCPv4 in older instances which rely on dhclient instead of systemd-networkd, so that could explain the issue with Ubuntu 16.04 but not with 18.04, 20.04 or Fedora 34.

@tomp any ideas?

Thank you so much for the response! I think I was mistaken, and that the issue is firewalld (I forgot to restart after I disabled the service). Disabling firewalld and restarting fixes the issue. I’m not quite sure what the issue is though looking at the rules.

This ended up fixing the issue for me for all but ubuntu 16.04:

firewall-cmd --add-interface=lxdbr0 --zone=trusted --permanent
firewall-cmd --reload

Do you have any suggestions for resolving the issue with older clients not using systemd-networkd?

Glad you got it working.

You don’t necessarily need to use systemd-networkd as the DHCP client, modern versions of dhclient work fine with nftables without the checksum fix support.

Example:

lxc launch images:ubuntu/focal c1
lxc exec c1 -- ip a flush dev eth0
lxc exec c1 -- dhclient -v
Internet Systems Consortium DHCP Client 4.4.1

lxc exec c1 -- dhclient eth0
lxc exec c1 -- ip a
42: eth0@if43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:5b:d1:bb brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.98.30.3/24 brd 10.98.30.255 scope global dynamic eth0
       valid_lft 3521sec preferred_lft 3521sec