Ipv6 route gets invalidated but not renewed

I’m using a hetzner cloud instance with a /64 ipv6 subnet. I want to assign the containers a dedicated ipv6 address.

Basically the assignment works. But I got problems with loosing ipv4/ipv6 network connectivity after some time. I solved at least the ipv4 connectivity loss by setting:

ipv4.dhcp.expiry: infinite
ipv6.dhcp.expiry: infinite

But ipv6 still looses connectivity after some time.

I did some digging at recognized that ipv6 route expires and is not renewed:

root@proud-goldfish:~# ip -6 route
2a01:4f8:xxxx:xxxx::/120 dev eth0 proto ra metric 100 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::185f:9bff:xxxx:xxxx dev eth0 proto ra metric 100 expires 707sec mtu 1500 pref medium

After some time the route is removed but not renewed:

root@proud-goldfish:~# ip -6 route
2a01:4f8:xxxx:xxxx:::/120 dev eth0 proto ra metric 100 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
root@proud-goldfish:~# journalctl | grep eth0
May 19 07:09:45 proud-goldfish systemd-networkd[73]: eth0: IPv6 successfully enabled
May 19 07:09:45 proud-goldfish systemd-networkd[73]: eth0: DHCPv4 address 10.254.210.85/24 via 10.254.210.1
May 19 07:09:46 proud-goldfish systemd-networkd[73]: eth0: Gained IPv6LL
May 19 07:09:47 proud-goldfish systemd-networkd[73]: eth0: DHCPv6 address 2a01:4f8:xxxx:xxxx::c0/128 timeout preferred -1 valid -1
asbachb@ubuntu-8gb-nbg1-1:~$ sudo lxc network show lxdbr0
config:
  ipv4.address: 10.254.210.1/24
  ipv4.dhcp.expiry: infinite
  ipv4.nat: "true"
  ipv6.address: 2a01:4f8:xxxx:xxxx::1/120
  ipv6.dhcp.expiry: infinite
  ipv6.dhcp.stateful: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/mutual-hen
- /1.0/instances/nixos-acme
- /1.0/instances/nixos-mail
- /1.0/instances/proud-goldfish
managed: true
status: Created
locations:
- none
asbachb@ubuntu-8gb-nbg1-1:~$ sudo lxc version
Client version: 4.1
Server version: 4.1
asbachb@ubuntu-8gb-nbg1-1:~$ sudo lxc info
config:
  core.https_address: "8899"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 8899:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
...
    -----END CERTIFICATE-----
  certificate_fingerprint: ...
  driver: lxc
  driver_version: 4.0.2
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.4.0-29-generic
  lxc_features:
    cgroup2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    seccomp_notify: "true"
  os_name: Ubuntu
  os_version: "20.04"
  project: default
  server: lxd
  server_clustered: false
  server_name: ubuntu-8gb-nbg1-1
  server_pid: 997
  server_version: "4.1"
  storage: dir
  storage_version: "1"

My first thought would be to check your firewall settings on the LXD host to make sure that IPv6 router advertisements and DHCPv6 isn’t getting blocked.

I guess that was the problem which I solved with the infinite configuration. But it seems not to solve the problem with disappearing route

After your comment I change default incoming and routed to allow and set ipv{4,6}.dhcp.expiry to 5m:

asbachb@ubuntu-8gb-nbg1-1:~$ sudo lxc network show lxdbr0
[sudo] password for asbachb: 
config:
  ipv4.address: 10.254.210.1/24
  ipv4.dhcp.expiry: 5m
  ipv4.nat: "true"
  ipv6.address: 2a01:4f8:xxxx:xxxx::1/120
  ipv6.dhcp.expiry: 5m
  ipv6.dhcp.stateful: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/mutual-hen
- /1.0/instances/nixos-acme
- /1.0/instances/nixos-mail
- /1.0/instances/proud-goldfish
managed: true
status: Created
locations:
- none
abachb@ubuntu-8gb-nbg1-1:~$ sudo ufw status verbose
Status: active
Logging: on (high)
Default: allow (incoming), allow (outgoing), allow (routed)
New profiles: skip
...

But route still disappears after expiry:

root@mutual-hen:~# ip -6 route
2a01:4f8:xxxx:xxxx::/120 dev eth0 proto ra metric 100 expires 1774sec pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::b0b4:8bff:xxxx:xxxx dev eth0 proto ra metric 100 expires 1774sec mtu 1500 pref medium
root@mutual-hen:~# ip -6 route
fe80::/64 dev eth0 proto kernel metric 256 pref medium

Just to mention. Ther’re some additional ufw rules:

asbachb@ubuntu-8gb-nbg1-1:~$ sudo cat /etc/ufw/before.rules 
#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
#   ufw-before-input
#   ufw-before-output
#   ufw-before-forward
#

*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-F
# SMTP
-A PREROUTING -p tcp -i eth0 --dport 25 -j DNAT --to-destination 10.254.210.183:25
-A PREROUTING -p tcp -i eth0 --dport 587 -j DNAT --to-destination 10.254.210.183:587
# IMAP
-A PREROUTING -p tcp -i eth0 --dport 143 -j DNAT --to-destination 10.254.210.183:143
-A PREROUTING -p tcp -i eth0 --dport 993 -j DNAT --to-destination 10.254.210.183:993
COMMIT

# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines


# allow all on loopback
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-output -o lo -j ACCEPT

# quickly process packets for which we already have a connection
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# drop INVALID packets (logs these in loglevel medium and higher)
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP

# ok icmp codes for INPUT
-A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT

# ok icmp code for FORWARD
-A ufw-before-forward -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-forward -p icmp --icmp-type echo-request -j ACCEPT

# allow dhcp client to work
-A ufw-before-input -p udp --sport 67 --dport 68 -j ACCEPT

#
# ufw-not-local
#
-A ufw-before-input -j ufw-not-local

# if LOCAL, RETURN
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN

# if MULTICAST, RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN

# if BROADCAST, RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN

# all other non-local packets are dropped
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP

# allow MULTICAST mDNS for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT

# allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above
# is uncommented)
-A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT

# don't delete the 'COMMIT' line or these rules won't be processed
COMMIT

Are you able to disable the firewall and retry to check it is causing the issue?

I disabled ufw and did a reboot.

asbachb@ubuntu-8gb-nbg1-1:~$ sudo ufw disable
Firewall stopped and disabled on system startup
asbachb@ubuntu-8gb-nbg1-1:~$ sudo reboot

But still the same issue:

root@mutual-hen:~# ip -6 route
2a01:4f8:xxxx:xxxx::/120 dev eth0 proto ra metric 100 expires 6sec pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::c810:f5ff:xxxx:xxxx dev eth0 proto ra metric 100 expires 6sec mtu 1500 pref medium

root@mutual-hen:~# ip -6 route
fe80::/64 dev eth0 proto kernel metric 256 pref medium

If you set ipv6.dhcp.stateful=false does it still occur out of interest?

Can you also run this on the host side on lxdbr0 and inside the container (on eth0) so we can see if there are any router announcements:

sudo tcpdump -l -nn -i lxdbr0 icmp6 and 'ip6[40] = 134'

With ipv6.dhcp.staeful = false. The container don’t get a route at all:

oot@mutual-hen:~# ip -6 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
    inet6 fe80::216:3eff:xxxx:xxxx/64 scope link 
       valid_lft forever preferred_lft forever
root@mutual-hen:~# ip -6 route
fe80::/64 dev eth0 proto kernel metric 256 pref medium

Interesting, so something is blocking either your router solicitation request (or your container isn’t sending one) or the router advertisements.

What container OS/version are you running and what network configuration files do you have in there?

The host is running stock Hetzner Ubuntu 20.04. The guest image is images:ubuntu/focal.

AFAIK networking is done via cloud-init:

asbachb@ubuntu-8gb-nbg1-1:~$ cat /etc/netplan/50-cloud-init.yaml 
# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    version: 2
    ethernets:
        eth0:
            addresses:
            - 2a01:4f8:xxxx:xxxx::1/64
            dhcp4: true
            gateway6: fe80::1
            match:
                macaddress: 96:00:00:xx:xx:xx
            set-name: eth0

Is that the container’s netplan config?

I guess it is. At least there’s only the file I pasted in that folder

asbachb@ubuntu-8gb-nbg1-1:~$ ls -l /etc/netplan/
total 4
-rw-r--r-- 1 root root 572 May  1 19:04 50-cloud-init.yaml

ubuntu-8gb-nbg1-1 is your host not your container. What is the network config inside your container?

I guess that might be the problem:

root@mutual-hen:~# cat /etc/netplan/10-lxc.yaml 
network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      dhcp-identifier: mac

OK, that looks ok to me.

Have you got the output of tcpdump yet?

Also, when the route disappears can you run netplan apply and see if it re-appears.

With netplan apply the route is back and ipv6 connectivity too.

I also can see the request on host:

asbachb@ubuntu-8gb-nbg1-1:~$ sudo tcpdump -l -nn -i lxdbr0 icmp6 and 'ip6[40] = 134' -v
tcpdump: listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:36:10.711396 IP6 (class 0xc0, flowlabel 0x55a2e, hlim 255, next-header ICMPv6 (58) payload length: 88) fe80::c810:xxxx:xxxx:c700 > fe80::216:xxxx:xxxx:694e: [icmp6 sum ok] ICMP6, router advertisement, length 88
	hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
	  prefix info option (3), length 32 (4): 2a01:4f8:xxxx:xxxx::/120, Flags [onlink], valid time 1800s, pref. time 1800s
	  mtu option (5), length 8 (1):  1500
	  source link-address option (1), length 8 (1): 56:f7:92:xx:xx:xx
	  rdnss option (25), length 24 (3):  lifetime 1800s, addr: fe80::c810:f5ff:xxxx:xxxx

OK so now leave tcpdump running and wait until the route drops off again and advise if you still see those periodic route advertisements every few minutes.

There is no traffic. When route expires.

asbachb@ubuntu-8gb-nbg1-1:~$ sudo tcpdump -l -nn -i lxdbr0 icmp6 and 'ip6[40] = 134' -v
tcpdump: listening on lxdbr0, link-type EN10MB (Ethernet), capture size 262144 bytes
root@proud-goldfish:~# tcpdump -l -nn -i eth0 icmp6 and 'ip6[40] = 134' -v
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

I only see traffic on host when restarting the container.

OK thanks, how long are you leaving the capture running (as by default the advertisements only happen every 8 minutes or so from my observations).

I guess about 20 minutes.