I am seeing random occurrences where the IPv6 address of a container is somehow being unassigned/dropped. It hangs my active SSH connections into the container, so it is very obvious. It takes a good 5 mins or more for it to resolve itself at which point the hung connections go back to normal.
Any ideas what might cause this? Below are observations from the host machine and the container shortly after it happens.
Host Info:
OS: Ubuntu 18.04.5 LTS
LXD: 4.16
LXD network config for lxdbr0:
config:
dns.search: lxd,corp.terasci.com
ipv4.address: 10.11.12.1/24
ipv4.nat: "true"
ipv6.address: fd42:c8f3:56ae:8db::1/64
ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/build-armbian
managed: true
status: Created
locations:
- none
Container config:
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 21.04 amd64 (release) (20210616)
image.label: release
image.os: ubuntu
image.release: hirsute
image.serial: "20210616"
image.type: squashfs
image.version: "21.04"
security.syscalls.intercept.mount: "true"
security.syscalls.intercept.mount.allowed: devtmpfs
volatile.base_image: f1d9d2d7ea5d90691c4559f0bdb1b68598041f0c90678451695b5d7e8a98d327
volatile.eth0.host_name: vethfc8ff1ce
volatile.eth0.hwaddr: 00:16:3e:fa:f9:5e
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.power: RUNNING
volatile.uuid: df7b3c90-f517-41d6-82df-ef328bbe7b38
devices: {}
ephemeral: false
profiles:
- default
- loop
stateful: false
description: ""
Interface details for ‘lxdbr0’ on host:
anderson-ryzen9:~$ ip addr show dev lxdbr0
4: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:dc:42:47 brd ff:ff:ff:ff:ff:ff
inet 10.11.12.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:c8f3:56ae:8db::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fedc:4247/64 scope link
valid_lft forever preferred_lft forever
DNS lookup for container name ‘build-armbian.lxd’:
anderson-ryzen9:~$ dig -tAAAA build-armbian.lxd
; <<>> DiG 9.11.3-1ubuntu1.15-Ubuntu <<>> -tAAAA build-armbian.lxd
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28439
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;build-armbian.lxd. IN AAAA
;; ANSWER SECTION:
build-armbian.lxd. 0 IN AAAA fd42:c8f3:56ae:8db:216:3eff:fefa:f95e
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Aug 04 12:21:42 PDT 2021
;; MSG SIZE rcvd: 74
Interface details from within the container:
build-armbian:~$ ip addr
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:fa:f9:5e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.11.12.199/24 brd 10.11.12.255 scope global dynamic eth0
valid_lft 2848sec preferred_lft 2848sec
inet6 fe80::216:3eff:fefa:f95e/64 scope link
valid_lft forever preferred_lft forever
Ping, from the host, to the IPv6 address returned from DNS:
anderson-ryzen9:~$ ping6 build-armbian.lxd
PING build-armbian.lxd(fd42:c8f3:56ae:8db:216:3eff:fefa:f95e (fd42:c8f3:56ae:8db:216:3eff:fefa:f95e)) 56 data bytes
From anderson-ryzen9 (fd42:c8f3:56ae:8db::1) icmp_seq=1 Destination unreachable: Address
From anderson-ryzen9 (fd42:c8f3:56ae:8db::1) icmp_seq=196 Destination unreachable: Address unreachable
64 bytes from fd42:c8f3:56ae:8db:216:3eff:fefa:f95e (fd42:c8f3:56ae:8db:216:3eff:fefa:f95e): icmp_seq=198 ttl=64 time=1024 ms
64 bytes from fd42:c8f3:56ae:8db:216:3eff:fefa:f95e (fd42:c8f3:56ae:8db:216:3eff:fefa:f95e): icmp_seq=197 ttl=64 time=2048 ms
64 bytes from fd42:c8f3:56ae:8db:216:3eff:fefa:f95e (fd42:c8f3:56ae:8db:216:3eff:fefa:f95e): icmp_seq=199 ttl=64 time=0.027 ms
64 bytes from fd42:c8f3:56ae:8db:216:3eff:fefa:f95e (fd42:c8f3:56ae:8db:216:3eff:fefa:f95e): icmp_seq=200 ttl=64 time=0.056 ms
So, at this point the connection has returned according to the ping, below are the interface details again after the connections are back.
Interface details for ‘lxdbr0’ on host:
anderson-ryzen9:~$ ip addr show dev lxdbr0
4: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:dc:42:47 brd ff:ff:ff:ff:ff:ff
inet 10.11.12.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:c8f3:56ae:8db:216:3eff:fedc:4247/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 2679sec preferred_lft 2679sec
inet6 fd42:c8f3:56ae:8db::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fedc:4247/64 scope link
valid_lft forever preferred_lft forever
Interface details from within the container:
build-armbian:~$ ip addr
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:fa:f9:5e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.11.12.199/24 brd 10.11.12.255 scope global dynamic eth0
valid_lft 2716sec preferred_lft 2716sec
inet6 fd42:c8f3:56ae:8db:216:3eff:fefa:f95e/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 3594sec preferred_lft 3594sec
inet6 fe80::216:3eff:fefa:f95e/64 scope link
valid_lft forever preferred_lft forever
No indication of any problems on ‘dmesg’ output on the host.
Not sure where to look next?