LXC containers cannot ping outside world

Hello,

After a recent server reboot, my containers are unable to ping the outside world. When I ping, there is no response other than:

root@marmoset03-submit-05:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

I have referred to this thread, but not had any luck yet:
https://discuss.linuxcontainers.org/t/lxc-containers-cannot-ping-outside-world-network-is-unreachable/7583

Host OS: Centos 7
Guest OS: Ubuntu 18.04

ip a on the host machine:

[d24lau-adm@marmoset03 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 0c:c4:7a:ad:1f:b4 brd ff:ff:ff:ff:ff:ff
    inet 129.97.105.45/24 brd 129.97.105.255 scope global noprefixroute eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fead:1fb4/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 0c:c4:7a:ad:1f:b5 brd ff:ff:ff:ff:ff:ff
4: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 0c:c4:7a:ad:1f:b6 brd ff:ff:ff:ff:ff:ff
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 0c:c4:7a:ad:1f:b7 brd ff:ff:ff:ff:ff:ff
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:b2:56:39 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
7: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:b2:56:39 brd ff:ff:ff:ff:ff:ff
8: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:91:04:f3 brd ff:ff:ff:ff:ff:ff
    inet 10.129.221.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:a693:78c7:b0b9::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe62:b0f5/64 scope link
       valid_lft forever preferred_lft forever
10: veth62ba338f@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 6a:26:8b:4a:b5:ed brd ff:ff:ff:ff:ff:ff link-netnsid 0
20: vethede25ae9@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether da:b0:2f:ed:82:6a brd ff:ff:ff:ff:ff:ff link-netnsid 2
44: vethad343f7c@if43: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 1e:8f:f4:6d:18:0e brd ff:ff:ff:ff:ff:ff link-netnsid 1

ip r on the host machine:

[d24lau-adm@marmoset03 ~]$ ip r
default via 129.97.105.1 dev eno1 proto static metric 100
10.129.221.0/24 dev lxdbr0 proto kernel scope link src 10.129.221.1
129.97.105.0/24 dev eno1 proto kernel scope link src 129.97.105.45 metric 100
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

ip a on the guest:

root@marmoset03-submit-05:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
43: eth0@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:32:85:6d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.129.221.213/24 brd 10.129.221.255 scope global dynamic eth0
       valid_lft 2953sec preferred_lft 2953sec
    inet6 fd42:a693:78c7:b0b9:216:3eff:fe32:856d/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 3203sec preferred_lft 3203sec
    inet6 fe80::216:3eff:fe32:856d/64 scope link
       valid_lft forever preferred_lft forever

ip r on the guest:

root@marmoset03-submit-05:~# ip r
default via 10.129.221.1 dev eth0 proto dhcp src 10.129.221.213 metric 100
10.129.221.0/24 dev eth0 proto kernel scope link src 10.129.221.213
10.129.221.1 dev eth0 proto dhcp scope link src 10.129.221.213 metric 100

Any assistance would be greatly appreciated.

I did try adding the following to /etc/dnsmasq.conf:

listen-address=::1,127.0.0.1,129.97.105.45

Then I restarted dnsmasq with sudo systemctl restart dnsmasq and restarted the container, but that didn’t seem to make a difference.

Just in case it is helpful, here is the result of sudo netstat -tulnp | grep :53 on the host:

[d24lau-adm@marmoset03 ~]$ sudo netstat -tulnp | grep :53
tcp        0      0 10.129.221.1:53         0.0.0.0:*               LISTEN      13438/dnsmasq
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      4125/dnsmasq
tcp6       0      0 fd42:a693:78c7:b0b9::53 :::*                    LISTEN      13438/dnsmasq
tcp6       0      0 fe80::216:3eff:fe62::53 :::*                    LISTEN      13438/dnsmasq
udp        0      0 10.129.221.1:53         0.0.0.0:*                           13438/dnsmasq
udp        0      0 192.168.122.1:53        0.0.0.0:*                           4125/dnsmasq
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           3244/avahi-daemon:
udp6       0      0 fd42:a693:78c7:b0b9::53 :::*                                13438/dnsmasq
udp6       0      0 fe80::216:3eff:fe62::53 :::*                                13438/dnsmasq

And the result of lxc network show lxdbr0:

root@marmoset03-submit-05:~# [d24lau-adm@marmoset03 ~]$ lxc network show lxdbr0
config:
  ipv4.address: 10.129.221.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:a693:78c7:b0b9::1/64
  ipv6.nat: "true"
  volatile.bridge.hwaddr: 00:16:3e:91:04:f3
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/marmoset-build-ece-05
- /1.0/instances/marmoset-build-ece-06
- /1.0/instances/marmoset-build-mme-01
- /1.0/instances/marmoset03-submit-01
- /1.0/instances/marmoset03-submit-05
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

Here is lxc config show <container> --expanded

[d24lau-adm@marmoset03 lxd]$ lxc config show marmoset03-submit-05 --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20191205)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20191205"
  image.type: squashfs
  image.version: "18.04"
  security.privileged: "true"
  volatile.base_image: f75468c572cc50eca7f76391182e6fdaf58431f84c3d35a2c92e83814e701698
  volatile.eth0.host_name: veth010c66a1
  volatile.eth0.hwaddr: 00:16:3e:32:85:6d
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  forward_marmoset03_80_http:
    connect: tcp:10.129.221.213:80
    listen: tcp:129.97.105.45:80
    type: proxy
  forward_marmoset03_443_https:
    connect: tcp:10.129.221.213:443
    listen: tcp:129.97.105.45:443
    type: proxy
  forward_marmoset03_8080_tomcat:
    connect: tcp:10.129.221.213:8080
    listen: tcp:129.97.105.45:8080
    type: proxy
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Restarting lxd with systemctl reload snap.lxd.daemon seems to have fixed the issue.