Container loses network connectivity after host resumes from sleep

LXD
1

I have some LXD containers with a routed network setup. They are routed back to my ethernet nic on my laptop. This works fine. However, if I put my laptop to sleep and then later resume, the containers all lose outgoing network connectivity until I reboot them.

They are still on the network as they are running nginx or apache and I can access the websites they host. But if they try to talk to the outside like using git or ping, they cannot reach anything other than the host’s IP address (not even the gateway on the network).

Does anyone know what is going on here and how to resolve it? I can reboot the container which works fine, but it is annoying to do as sometimes I’m running stuff in the container and it handles the host’s sleep just fine except for the network issue. Thanks.

2

This kinda rings a bell, basically NetworkManager resetting a bunch of stuff on resume and dropping the routes?

@tomp does that ring a bell?

3

Can you show the output of lxc config show <instance> --expanded for one of the problem containers.

Also can you show the output of ip a, ip r and ip neigh show proxy before and after your laptop goes to sleep.

4

Sure, here is the config:

architecture: x86_64
config:
  environment.TZ: America/Chicago
  image.architecture: amd64
  image.description: Debian buster amd64 (20201006_11:18)
  image.os: Debian
  image.release: buster
  image.serial: "20201006_11:18"
  image.type: squashfs
  raw.idmap: |-
    uid 117 105
    gid 124 108
  user.network-config: |
    version: 2
    ethernets:
        eth0:
            addresses:
            - 10.20.0.62/32
            nameservers:
                addresses:
                - 8.8.8.8
                search: []
            routes:
            -   to: 0.0.0.0/0
                via: 169.254.0.1
  volatile.base_image: f87c0c968536c5c964cbc186a8b1da538612908efa43784c9fe3acb535bc74f3
  volatile.eth0.host_name: vetha06eef17
  volatile.eth0.hwaddr: 00:16:3e:9b:92:f4
  volatile.eth0.last_state.created: "false"
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":105},{"Isuid":true,"Isgid":false,"Hostid":117,"Nsid":105,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1000106,"Nsid":106,"Maprange":999999894},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":108},{"Isuid":false,"Isgid":true,"Hostid":124,"Nsid":108,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1000109,"Nsid":109,"Maprange":999999891}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":105},{"Isuid":true,"Isgid":false,"Hostid":117,"Nsid":105,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1000106,"Nsid":106,"Maprange":999999894},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":108},{"Isuid":false,"Isgid":true,"Hostid":124,"Nsid":108,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1000109,"Nsid":109,"Maprange":999999891}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":105},{"Isuid":true,"Isgid":false,"Hostid":117,"Nsid":105,"Maprange":1},{"Isuid":true,"Isgid":false,"Hostid":1000106,"Nsid":106,"Maprange":999999894},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":108},{"Isuid":false,"Isgid":true,"Hostid":124,"Nsid":108,"Maprange":1},{"Isuid":false,"Isgid":true,"Hostid":1000109,"Nsid":109,"Maprange":999999891}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: d8bb6c11-04d1-4346-a037-5e6516664210
devices:
  eth0:
    ipv4.address: 10.20.0.62
    nictype: routed
    parent: enx847beb4fd13d
    type: nic
  mysql-data:
    path: /var/lib/mysql/data/
    source: /containerdata/mf/mysql/data/
    type: disk
  mysql-log:
    path: /var/lib/mysql/log/
    source: /containerdata/mf/mysql/log/
    type: disk
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
- routed_10.20.0.62
stateful: false
description: ""

ip a before sleep:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.12.0.60/32 scope global wg0
       valid_lft forever preferred_lft forever
5: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:a3:4e:f7 brd ff:ff:ff:ff:ff:ff
    inet 10.25.0.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:5344:cddf:3735::1/64 scope global 
       valid_lft forever preferred_lft forever
17: vethc9d3c856@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:51:e8:08:66:c4 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 169.254.0.1/32 scope global vethc9d3c856
       valid_lft forever preferred_lft forever
    inet6 fe80::fc51:e8ff:fe08:66c4/64 scope link 
       valid_lft forever preferred_lft forever
23: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 44:1c:a8:e2:c6:45 brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.65/24 brd 10.20.0.255 scope global dynamic noprefixroute wlp2s0
       valid_lft 83683sec preferred_lft 83683sec
    inet6 fe80::87b2:fabc:1aab:9bf2/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
24: enx847beb4fd13d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 84:7b:eb:4f:d1:3d brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.60/24 brd 10.20.0.255 scope global dynamic noprefixroute enx847beb4fd13d
       valid_lft 72581sec preferred_lft 72581sec
    inet6 fe80::a7fe:72b5:cb02:8763/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
25: vetha06eef17@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:fe:5f:a5:98:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.0.1/32 scope global vetha06eef17
       valid_lft forever preferred_lft forever
    inet6 fe80::fcfe:5fff:fea5:980c/64 scope link 
       valid_lft forever preferred_lft forever

ip r before sleep:

default via 10.20.0.1 dev enx847beb4fd13d proto dhcp metric 100 
default via 10.20.0.1 dev wlp2s0 proto dhcp metric 600 
10.10.24.0/24 dev wg0 scope link 
10.12.0.0/24 dev wg0 scope link 
10.20.0.0/24 dev enx847beb4fd13d proto kernel scope link src 10.20.0.60 metric 100 
10.20.0.0/24 dev wlp2s0 proto kernel scope link src 10.20.0.65 metric 600 
10.20.0.61 dev vethc9d3c856 scope link 
10.20.0.62 dev vetha06eef17 scope link 
10.25.0.0/24 dev lxdbr0 proto kernel scope link src 10.25.0.1 linkdown 
192.168.0.0/24 dev wg0 scope link

ip neigh show proxy before sleep:

10.20.0.62 dev enx847beb4fd13d  proxy
169.254.0.1 dev vetha06eef17  proxy
169.254.0.1 dev vethc9d3c856  proxy

ip a after sleep:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.12.0.60/32 scope global wg0
       valid_lft forever preferred_lft forever
5: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:a3:4e:f7 brd ff:ff:ff:ff:ff:ff
    inet 10.25.0.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:5344:cddf:3735::1/64 scope global 
       valid_lft forever preferred_lft forever
17: vethc9d3c856@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:51:e8:08:66:c4 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 169.254.0.1/32 scope global vethc9d3c856
       valid_lft forever preferred_lft forever
    inet6 fe80::fc51:e8ff:fe08:66c4/64 scope link 
       valid_lft forever preferred_lft forever
25: vetha06eef17@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:fe:5f:a5:98:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.0.1/32 scope global vetha06eef17
       valid_lft forever preferred_lft forever
    inet6 fe80::fcfe:5fff:fea5:980c/64 scope link 
       valid_lft forever preferred_lft forever
26: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 44:1c:a8:e2:c6:45 brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.65/24 brd 10.20.0.255 scope global dynamic noprefixroute wlp2s0
       valid_lft 78382sec preferred_lft 78382sec
    inet6 fe80::87b2:fabc:1aab:9bf2/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
27: enx847beb4fd13d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 84:7b:eb:4f:d1:3d brd ff:ff:ff:ff:ff:ff
    inet 10.20.0.60/24 brd 10.20.0.255 scope global dynamic noprefixroute enx847beb4fd13d
       valid_lft 74772sec preferred_lft 74772sec
    inet6 fe80::a7fe:72b5:cb02:8763/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

ip r after sleep:

default via 10.20.0.1 dev enx847beb4fd13d proto dhcp metric 100 
default via 10.20.0.1 dev wlp2s0 proto dhcp metric 600 
10.10.24.0/24 dev wg0 scope link 
10.12.0.0/24 dev wg0 scope link 
10.20.0.0/24 dev enx847beb4fd13d proto kernel scope link src 10.20.0.60 metric 100 
10.20.0.0/24 dev wlp2s0 proto kernel scope link src 10.20.0.65 metric 600 
10.20.0.61 dev vethc9d3c856 scope link 
10.20.0.62 dev vetha06eef17 scope link 
10.25.0.0/24 dev lxdbr0 proto kernel scope link src 10.25.0.1 linkdown 
192.168.0.0/24 dev wg0 scope link

ip neigh show proxy after sleep:

169.254.0.1 dev vetha06eef17  proxy
169.254.0.1 dev vethc9d3c856  proxy

Thanks for the help.

5

So it looks like the parent interface enx847beb4fd13d was removed/taken down during the sleep process and in doing so it removed the neighbour proxy entries for your routed NICs.

When the interface was restored when you resumed your laptop they would no longer be there.

The veth interfaces didn’t go down/removed during sleep so their proxy entries remain.

Is that a USB ethernet adapter?

6

Yes, you are right. In this case the ethernet is on a docking port that is connected via a Thunderbolt cable. When it resumes from sleep it initially has no connection to the docking port so it drops the ethernet but it reestablishes the connection a couple of seconds after waking up.

Is there a command I could run to reestablish these proxy entries after it takes up from sleep? It wouldn’t be a huge deal to do that and would be a lot better than having to reboot the container. Thanks.

7

I was able to just add the proxy entry on the host after resuming from sleep and that seems to work:

ip neigh add proxy 10.20.0.62 dev enx847beb4fd13d

1 Like
8

Yes that is what LXD does during NIC start up.