Enabling security.ipv6_filtering causes the host to stop responding to DHCPv6 renew requests

Incus
1

I’m not sure if this is expected behavior or a bug. Can someone help me?

If I enable security.ipv6_filtering, the host only responds to the initial DHCPv6 Solicit request. Once the lease expires, all subsequent DHCPv6 Renew messages sent from the virtual machine to the host receive no response. Below is the packet capture result:

tcpdump -i incusbr0 -n -vvv port 546 or port 547
tcpdump: listening on incusbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

08:59:53.496935 IP6 (flowlabel 0x7fea9, hlim 1, next-header UDP (17) payload length: 104) fe80::1266:6aff:fe51:3f79.546 > ff02::1:2.547: [bad udp cksum 0xb930 -> 0x61d4!] dhcp6 solicit (xid=3cc8dd (rapid-commit) (IA_NA IAID:2492595448 T1:0 T2:0) (IA_PD IAID:2492595448 T1:0 T2:0) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_82 opt_103 opt_144) (client-ID enterprise 43793 a670def4bb01e237) (elapsed-time 0))

08:59:53.497132 IP6 (class 0xc0, flowlabel 0x73593, hlim 64, next-header UDP (17) payload length: 157) fe80::1266:6aff:fe2c:800b.547 > fe80::1266:6aff:fe51:3f79.546: [bad udp cksum 0xb47e -> 0xbd84!] dhcp6 reply (xid=3cc8dd (client-ID enterprise 43793 a670def4bb01e237) (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (rapid-commit) (IA_NA IAID:2492595448 T1:100 T2:175 (IA_ADDR 2607:5300:60:8401::4 pltime:200 vltime:200)) (status-code Success) (preference 255) (DNS-server 2607:5300:60:8401::1) (Client-FQDN))

09:01:32.495593 IP6 (flowlabel 0x7fea9, hlim 1, next-header UDP (17) payload length: 128) fe80::1266:6aff:fe51:3f79.546 > ff02::1:2.547: [bad udp cksum 0xb948 -> 0x2e25!] dhcp6 renew (xid=53f5e4 (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2492595448 T1:0 T2:0 (IA_ADDR 2607:5300:60:8401::4 pltime:0 vltime:0)) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_103 opt_144) (client-ID enterprise 43793 a670def4bb01e237) (elapsed-time 0))
09:01:41.540976 IP6 (flowlabel 0x7fea9, hlim 1, next-header UDP (17) payload length: 128) fe80::1266:6aff:fe51:3f79.546 > ff02::1:2.547: [bad udp cksum 0xb948 -> 0x2a9d!] dhcp6 renew (xid=53f5e4 (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2492595448 T1:0 T2:0 (IA_ADDR 2607:5300:60:8401::4 pltime:0 vltime:0)) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_103 opt_144) (client-ID enterprise 43793 a670def4bb01e237) (elapsed-time 904))
09:01:59.219353 IP6 (flowlabel 0x7fea9, hlim 1, next-header UDP (17) payload length: 128) fe80::1266:6aff:fe51:3f79.546 > ff02::1:2.547: [bad udp cksum 0xb948 -> 0x23b5!] dhcp6 renew (xid=53f5e4 (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2492595448 T1:0 T2:0 (IA_ADDR 2607:5300:60:8401::4 pltime:0 vltime:0)) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_103 opt_144) (client-ID enterprise 43793 a670def4bb01e237) (elapsed-time 2672))

If I disable security.ipv6_filtering , the host returns to normal and immediately responds to DHCPv6 Reply requests.

09:20:02.495918 IP6 (flowlabel 0x7fea9, hlim 1, next-header UDP (17) payload length: 128) fe80::1266:6aff:fe51:3f79.546 > ff02::1:2.547: [bad udp cksum 0xb948 -> 0xa9cd!] dhcp6 renew (xid=bd79d2 (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2492595448 T1:0 T2:0 (IA_ADDR 2607:5300:60:8401::4 pltime:0 vltime:0)) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_103 opt_144) (client-ID enterprise 43793 a670def4bb01e237) (elapsed-time 0))
09:20:02.496171 IP6 (class 0xc0, flowlabel 0x73593, hlim 64, next-header UDP (17) payload length: 135) fe80::1266:6aff:fe2c:800b.547 > fe80::1266:6aff:fe51:3f79.546: [bad udp cksum 0xb468 -> 0xc4b8!] dhcp6 reply (xid=bd79d2 (client-ID enterprise 43793 a670def4bb01e237) (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2492595448 T1:93 T2:168 (IA_ADDR 2607:5300:60:8401::4 pltime:200 vltime:200)) (DNS-server 2607:5300:60:8401::1) (Client-FQDN))
09:21:42.495917 IP6 (flowlabel 0x7fea9, hlim 1, next-header UDP (17) payload length: 128) fe80::1266:6aff:fe51:3f79.546 > ff02::1:2.547: [bad udp cksum 0xb948 -> 0x5f1e!] dhcp6 renew (xid=4fc4ef (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2492595448 T1:0 T2:0 (IA_ADDR 2607:5300:60:8401::4 pltime:0 vltime:0)) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_103 opt_144) (client-ID enterprise 43793 a670def4bb01e237) (elapsed-time 0))
09:21:42.496206 IP6 (class 0xc0, flowlabel 0x73593, hlim 64, next-header UDP (17) payload length: 135) fe80::1266:6aff:fe2c:800b.547 > fe80::1266:6aff:fe51:3f79.546: [bad udp cksum 0xb468 -> 0x7a09!] dhcp6 reply (xid=4fc4ef (client-ID enterprise 43793 a670def4bb01e237) (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2492595448 T1:93 T2:168 (IA_ADDR 2607:5300:60:8401::4 pltime:200 vltime:200)) (DNS-server 2607:5300:60:8401::1) (Client-FQDN))

incusbr0 config:

project: default
name: incusbr0
description: ''
type: bridge
config:
  ipv4.address: 10.38.58.1/24
  ipv4.nat: 'true'
  ipv6.address: 2607:5300:60:8401::1/64
  ipv6.dhcp: 'true'
  ipv6.dhcp.expiry: '200'
  ipv6.dhcp.ranges: 2607:5300:60:8401::2-2607:5300:60:8401::10
  ipv6.dhcp.stateful: 'true'
  ipv6.nat: 'false'

profile config:

name: default
description: Default Incus profile
devices:
  eth0:
    network: incusbr0
    security.ipv4_filtering: 'true'
    security.ipv6_filtering: 'true'
    security.mac_filtering: 'true'
    type: nic

guest vm debian13 cloud image with systemd-networkd

2

Can you show incus info | grep firewall:?

Looking at the logic and your traffic dump, we seem to allow any traffic on port 547 with the address being the multicast address for DHCPv6, we don’t have special handling for request vs renewal so I’m not sure what’s causing the filtering to make this fail.

Though I looked at the modern nftables logic.

3

Thank you for your quick response.

incus info | grep firewall:
firewall: nftables

4

Can you show nft list ruleset when things are broken?

5 
nft list ruleset
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
        chain DOCKER {
                iifname "docker0" counter packets 0 bytes 0 return
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                fib daddr type local counter packets 70311 bytes 2683172 jump DOCKER
        }

        chain OUTPUT {
                type nat hook output priority dstnat; policy accept;
                ip daddr != 127.0.0.0/8 fib daddr type local counter packets 1 bytes 278 jump DOCKER
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr 172.17.0.0/16 oifname != "docker0" counter packets 0 bytes 0 masquerade
        }
}
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
        chain DOCKER {
                iifname != "docker0" oifname "docker0" counter packets 0 bytes 0 drop
        }

        chain DOCKER-FORWARD {
                counter packets 114554 bytes 143865418 jump DOCKER-CT
                counter packets 114554 bytes 143865418 jump DOCKER-ISOLATION-STAGE-1
                counter packets 114554 bytes 143865418 jump DOCKER-BRIDGE
                iifname "docker0" counter packets 0 bytes 0 accept
        }

        chain DOCKER-BRIDGE {
                oifname "docker0" counter packets 0 bytes 0 jump DOCKER
        }

        chain DOCKER-CT {
                oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
        }

        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "docker0" counter packets 0 bytes 0 drop
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                counter packets 114554 bytes 143865418 jump DOCKER-USER
                counter packets 114554 bytes 143865418 jump DOCKER-FORWARD
        }

        chain DOCKER-USER {
        }
}
# Warning: table ip6 nat is managed by iptables-nft, do not touch!
table ip6 nat {
        chain DOCKER {
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                fib daddr type local counter packets 6319 bytes 554162 jump DOCKER
        }

        chain OUTPUT {
                type nat hook output priority dstnat; policy accept;
                ip6 daddr != ::1 fib daddr type local counter packets 0 bytes 0 jump DOCKER
        }
}
table ip6 filter {
        chain DOCKER {
        }

        chain DOCKER-FORWARD {
                counter packets 15273 bytes 26286715 jump DOCKER-CT
                counter packets 15273 bytes 26286715 jump DOCKER-ISOLATION-STAGE-1
                counter packets 15273 bytes 26286715 jump DOCKER-BRIDGE
        }

        chain DOCKER-BRIDGE {
        }

        chain DOCKER-CT {
        }

        chain DOCKER-ISOLATION-STAGE-1 {
        }

        chain DOCKER-ISOLATION-STAGE-2 {
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                counter packets 15273 bytes 26286715 jump DOCKER-USER
                counter packets 15273 bytes 26286715 jump DOCKER-FORWARD
        }

        chain DOCKER-USER {
        }
}
table inet incus {
        chain pstrt.incusbr0 {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr 10.38.58.0/24 ip daddr != 10.38.58.0/24 masquerade
        }

        chain fwd.incusbr0 {
                type filter hook forward priority filter; policy accept;
                ip version 4 oifname "incusbr0" accept
                ip version 4 iifname "incusbr0" accept
                ip6 version 6 oifname "incusbr0" accept
                ip6 version 6 iifname "incusbr0" accept
        }

        chain in.incusbr0 {
                type filter hook input priority filter; policy accept;
                iifname "incusbr0" tcp dport 53 accept
                iifname "incusbr0" udp dport 53 accept
                iifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
                iifname "incusbr0" udp dport 67 accept
                iifname "incusbr0" ip protocol udp udp checksum set 0
                iifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
                iifname "incusbr0" udp dport 547 accept
        }

        chain out.incusbr0 {
                type filter hook output priority filter; policy accept;
                oifname "incusbr0" tcp sport 53 accept
                oifname "incusbr0" udp sport 53 accept
                oifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
                oifname "incusbr0" udp sport 67 accept
                oifname "incusbr0" ip protocol udp udp checksum set 0
                oifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
                oifname "incusbr0" udp sport 547 accept
        }
}
table bridge incus {
        chain in.test.eth0 {
                type filter hook input priority filter; policy accept;
                iifname "veth5ad86f59" ether saddr != 10:66:6a:a2:95:3c drop
                iifname "veth5ad86f59" arp saddr ether != 10:66:6a:a2:95:3c drop
                iifname "veth5ad86f59" icmpv6 type nd-neighbor-advert @nh,528,48 != 0x10666aa2953c drop
                iifname "veth5ad86f59" ip saddr 0.0.0.0 ip daddr 255.255.255.255 udp dport 67 accept
                iifname "veth5ad86f59" arp saddr ip != 10.38.58.3 drop
                iifname "veth5ad86f59" ip saddr != 10.38.58.3 drop
                iifname "veth5ad86f59" ip6 saddr fe80::/10 ip6 daddr ff02::1:2 udp dport 547 accept
                iifname "veth5ad86f59" ip6 saddr fe80::/10 ip6 daddr ff02::2 icmpv6 type nd-router-solicit accept
                iifname "veth5ad86f59" icmpv6 type nd-router-advert drop
                iifname "veth5ad86f59" icmpv6 type nd-neighbor-advert @nh,384,128 != 0x26075300006084010000000000000003 drop
                iifname "veth5ad86f59" ip6 saddr != 2607:5300:60:8401::3 drop
                iifname "veth5ad86f59" ether type != { ip, arp, ip6 } drop
        }

        chain fwd.test.eth0 {
                type filter hook forward priority filter; policy accept;
                iifname "veth5ad86f59" ether saddr != 10:66:6a:a2:95:3c drop
                iifname "veth5ad86f59" arp saddr ether != 10:66:6a:a2:95:3c drop
                iifname "veth5ad86f59" icmpv6 type nd-neighbor-advert @nh,528,48 != 0x10666aa2953c drop
                iifname "veth5ad86f59" arp saddr ip != 10.38.58.3 drop
                iifname "veth5ad86f59" ip saddr != 10.38.58.3 drop
                iifname "veth5ad86f59" icmpv6 type nd-router-advert drop
                iifname "veth5ad86f59" icmpv6 type nd-neighbor-advert @nh,384,128 != 0x26075300006084010000000000000003 drop
                iifname "veth5ad86f59" ip6 saddr != 2607:5300:60:8401::3 drop
                iifname "veth5ad86f59" ether type != { ip, arp, ip6 } drop
        }
}
6

That should allow DHCPv6 of any type through…

Can you try the tcpdump again but this time making sure to pass the -e flag?
tcpdump -eni eth0 would be my go-to in this situation. That will let us check on the MAC addresses used.

7

The MAC address of host incusbr0 is 10:66:6a:2c:80:0b.

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 0c:c4:7a:6c:02:6a brd ff:ff:ff:ff:ff:ff
    altname enp4s0f0
    altname enx0cc47a6c026a
    inet 158.69.55.210/24 metric 100 brd 158.69.55.255 scope global dynamic eno1
       valid_lft 52073sec preferred_lft 52073sec
    inet6 2607:5300:60:84d2::1/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fe6c:26a/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c4:7a:6c:02:6b brd ff:ff:ff:ff:ff:ff
    altname enp4s0f1
    altname enx0cc47a6c026b
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 32:35:92:0d:3a:63 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: incusbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 10:66:6a:2c:80:0b brd ff:ff:ff:ff:ff:ff
    inet 10.38.58.1/24 brd 10.38.58.255 scope global incusbr0
       valid_lft forever preferred_lft forever
    inet6 2607:5300:60:8401::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::1266:6aff:fe2c:800b/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
20: veth5064a09c@vethe812b5bf: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 10:66:6a:51:3f:79 brd ff:ff:ff:ff:ff:ff
21: vethe812b5bf@veth5064a09c: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master incusbr0 state LOWERLAYERDOWN group default qlen 1000
    link/ether c2:44:1e:44:d8:96 brd ff:ff:ff:ff:ff:ff
31: veth7c265499@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP group default qlen 1000
    link/ether b2:75:97:0c:f3:45 brd ff:ff:ff:ff:ff:ff link-netnsid 0

The MAC address of the virtual machine’s eth0 is 10:66:6a:a2:95:3c.

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
30: eth0@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 10:66:6a:a2:95:3c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.38.58.3/24 metric 1024 brd 10.38.58.255 scope global dynamic eth0
       valid_lft 3185sec preferred_lft 3185sec
    inet6 fe80::1266:6aff:fea2:953c/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

tcpdump

tcpdump -i incusbr0 -n -e -vvv port 546 or port 547

tcpdump: listening on incusbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

11:47:45.558737 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 152: (flowlabel 0x4b274, hlim 1, next-header UDP (17) payload length: 98) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [bad udp cksum 0x0f3f -> 0x35aa!] dhcp6 solicit (xid=7d92be (rapid-commit) (IA_NA IAID:2186565111 T1:0 T2:0) (IA_PD IAID:2186565111 T1:0 T2:0) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_82 opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 0))

11:47:45.558953 10:66:6a:2c:80:0b > 10:66:6a:a2:95:3c, ethertype IPv6 (0x86dd), length 205: (class 0xc0, flowlabel 0x268e0, hlim 64, next-header UDP (17) payload length: 151) fe80::1266:6aff:fe2c:800b.547 > fe80::1266:6aff:fea2:953c.546: [bad udp cksum 0x0a8d -> 0xd81c!] dhcp6 reply (xid=7d92be (client-ID enterprise 43793 5d59b28d559f75d3) (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (rapid-commit) (IA_NA IAID:2186565111 T1:100 T2:175 (IA_ADDR 2607:5300:60:8401::3 pltime:200 vltime:200)) (status-code Success) (preference 255) (DNS-server 2607:5300:60:8401::1) (Client-FQDN))

11:49:19.556045 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 176: (flowlabel 0x4b274, hlim 1, next-header UDP (17) payload length: 122) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [bad udp cksum 0x0f57 -> 0xbc3c!] dhcp6 renew (xid=344ca6 (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2186565111 T1:0 T2:0 (IA_ADDR 2607:5300:60:8401::3 pltime:0 vltime:0)) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 0))

11:49:29.509332 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 176: (flowlabel 0x4b274, hlim 1, next-header UDP (17) payload length: 122) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [bad udp cksum 0x0f57 -> 0xb859!] dhcp6 renew (xid=344ca6 (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2186565111 T1:0 T2:0 (IA_ADDR 2607:5300:60:8401::3 pltime:0 vltime:0)) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 995))

11:49:48.663608 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 176: (flowlabel 0x4b274, hlim 1, next-header UDP (17) payload length: 122) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [bad udp cksum 0x0f57 -> 0xb0de!] dhcp6 renew (xid=344ca6 (server-ID hwaddr/time type 1 time 811588800 0cc47a6c026a) (IA_NA IAID:2186565111 T1:0 T2:0 (IA_ADDR 2607:5300:60:8401::3 pltime:0 vltime:0)) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 2910))
8

The thing which has bitten me with DHCP (v4) in virtual environments in the past is UDP checksum offloading - and I do notice udp checksum errors in your tcpdump. However, I don’t see why this would be different between initial lease and renew.

Inside the VM, you could try:

ethtool -K eth0 gso off gro off tso off
9

This single rule is likely not sufficient: The DHCPv6 server is not replying from the multicast address on port 547 but from its link-local address (fe80::1266:6aff:fe2c:800b.547 in the capture), but the referenced rule has only allowed for outbound daddr ff02::1:2 udp dport 547 - therefore it’s not allowing the server’s response based on established state. I think an additional rule similar to saddr fe80::/10 udp sport 547 udp dport 546 is needed.

10

Thank you for your advice. I will try it later. :slightly_smiling_face:

11

I tried adding the nftables rules you provided, but to no avail. :slightly_frowning_face:

12

I was only able to get everything back to normal after deleting this nftables rule: iifname "vetha02f7a43" ip6 saddr != 2607:5300:60:8401::3 drop , but it also defeated the purpose of IPv6 filtering. :slightly_frowning_face:

13

The question then is, why did the previous rule

iifname "..." ip6 saddr fe80::/10 ip6 daddr ff02::1:2 udp dport 547 accept

not permit the packet? It’s conceivable that a bad UDP checksum causes this rule not to match. You could try changing it to:

iifname "..." ip6 saddr fe80::/10 ip6 daddr ff02::1:2 accept

to see if that helps narrow down the problem.

14

The default rules are added by security.ipv6_filtering:

nft -a list chain bridge incus in.test.eth0

table bridge incus {
        chain in.test.eth0 { # handle 126
                type filter hook input priority filter; policy accept;
                iifname "vethe5690360" ether saddr != 10:66:6a:a2:95:3c drop # handle 128
                iifname "vethe5690360" arp saddr ether != 10:66:6a:a2:95:3c drop # handle 129
                iifname "vethe5690360" icmpv6 type nd-neighbor-advert @nh,528,48 != 0x10666aa2953c drop # handle 130
                iifname "vethe5690360" ip saddr 0.0.0.0 ip daddr 255.255.255.255 udp dport 67 accept # handle 131
                iifname "vethe5690360" arp saddr ip != 10.38.58.3 drop # handle 132
                iifname "vethe5690360" ip saddr != 10.38.58.3 drop # handle 133
                iifname "vethe5690360" ip6 saddr fe80::/10 ip6 daddr ff02::1:2 udp dport 547 accept # handle 134
                iifname "vethe5690360" ip6 saddr fe80::/10 ip6 daddr ff02::2 icmpv6 type nd-router-solicit accept # handle 135
                iifname "vethe5690360" icmpv6 type nd-router-advert drop # handle 136
                iifname "vethe5690360" icmpv6 type nd-neighbor-advert @nh,384,128 != 0x26075300006084010000000000000003 drop # handle 137
                iifname "vethe5690360" ip6 saddr != 2607:5300:60:8401::3 drop # handle 138
                iifname "vethe5690360" ether type != { ip, arp, ip6 } drop # handle 140
        }
}

delete and add new rules:

nft delete rule bridge incus in.test.eth0 handle 134
nft insert rule bridge incus in.test.eth0 position 135 iifname "vethe5690360" ip6 saddr fe80::/10 ip6 
daddr ff02::1:2 accept

current rules:

nft -a list chain bridge incus in.test.eth0
table bridge incus {
        chain in.test.eth0 { # handle 126
                type filter hook input priority filter; policy accept;
                iifname "vethe5690360" ether saddr != 10:66:6a:a2:95:3c drop # handle 128
                iifname "vethe5690360" arp saddr ether != 10:66:6a:a2:95:3c drop # handle 129
                iifname "vethe5690360" icmpv6 type nd-neighbor-advert @nh,528,48 != 0x10666aa2953c drop # handle 130
                iifname "vethe5690360" ip saddr 0.0.0.0 ip daddr 255.255.255.255 udp dport 67 accept # handle 131
                iifname "vethe5690360" arp saddr ip != 10.38.58.3 drop # handle 132
                iifname "vethe5690360" ip saddr != 10.38.58.3 drop # handle 133
                iifname "vethe5690360" ip6 saddr fe80::/10 ip6 daddr ff02::1:2 accept # handle 152
                iifname "vethe5690360" ip6 saddr fe80::/10 ip6 daddr ff02::2 icmpv6 type nd-router-solicit accept # handle 135
                iifname "vethe5690360" icmpv6 type nd-router-advert drop # handle 136
                iifname "vethe5690360" icmpv6 type nd-neighbor-advert @nh,384,128 != 0x26075300006084010000000000000003 drop # handle 137
                iifname "vethe5690360" ip6 saddr != 2607:5300:60:8401::3 drop # handle 138
                iifname "vethe5690360" ether type != { ip, arp, ip6 } drop # handle 140
        }
}

By the way, I have already disabled UDP Checksum Offloading.

tcpdump:

tcpdump -i incusbr0 -n -e -vvv port 546 or port 547
tcpdump: listening on incusbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
19:42:27.227200 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 152: (flowlabel 0x5ef3f, hlim 1, next-header UDP (17) payload length: 98) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=5b6010 (rapid-commit) (IA_NA IAID:2186565111 T1:0 T2:0) (IA_PD IAID:2186565111 T1:0 T2:0) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_82 opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 65535))
19:44:24.230161 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 152: (flowlabel 0x5ef3f, hlim 1, next-header UDP (17) payload length: 98) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=5b6010 (rapid-commit) (IA_NA IAID:2186565111 T1:0 T2:0) (IA_PD IAID:2186565111 T1:0 T2:0) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_82 opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 65535))
19:46:13.360440 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 152: (flowlabel 0x5ef3f, hlim 1, next-header UDP (17) payload length: 98) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=5b6010 (rapid-commit) (IA_NA IAID:2186565111 T1:0 T2:0) (IA_PD IAID:2186565111 T1:0 T2:0) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_82 opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 65535))
19:48:03.760747 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 152: (flowlabel 0x5ef3f, hlim 1, next-header UDP (17) payload length: 98) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=5b6010 (rapid-commit) (IA_NA IAID:2186565111 T1:0 T2:0) (IA_PD IAID:2186565111 T1:0 T2:0) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_82 opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 65535))
19:50:01.207355 10:66:6a:a2:95:3c > 33:33:00:01:00:02, ethertype IPv6 (0x86dd), length 152: (flowlabel 0x5ef3f, hlim 1, next-header UDP (17) payload length: 98) fe80::1266:6aff:fea2:953c.546 > ff02::1:2.547: [udp sum ok] dhcp6 solicit (xid=5b6010 (rapid-commit) (IA_NA IAID:2186565111 T1:0 T2:0) (IA_PD IAID:2186565111 T1:0 T2:0) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_82 opt_103 opt_144) (client-ID enterprise 43793 5d59b28d559f75d3) (elapsed-time 65535))

Additionally, I specifically used another server today with a fresh installation of Debian 12. I installed Incus on this system and was able to reproduce the issue. Therefore, I believe this is a bug in Incus.

This is utterly torturous. I’ve already wasted several days on this issue and I’m completely exhausted. As an alternative, I could change the DHCPv6 lease time to a very large value, like 10 years, to prevent the virtual machines from losing their IPv6 addresses. Alternatively, I could use ndppd on the host machine to proxy the entire /64 IPv6 subnet and then use SLAAC to assign addresses to the VMs.

15

I can fix my issue temporarily by adding this rule:

nft insert rule bridge incus in.test.eth0 position 343 iifname "veth003b6128" ip6 saddr fe80::/10 accept

Current rule:

nft -a list chain bridge incus in.test.eth0
table bridge incus {
        chain in.test.eth0 { # handle 339
                type filter hook input priority filter; policy accept;
                iifname "veth003b6128" ether saddr != 10:66:6a:a2:95:3c drop # handle 341
                iifname "veth003b6128" arp saddr ether != 10:66:6a:a2:95:3c drop # handle 342
                iifname "veth003b6128" ip6 saddr fe80::/10 accept # handle 365
                iifname "veth003b6128" icmpv6 type nd-neighbor-advert @nh,528,48 != 0x10666aa2953c drop # handle 343
                iifname "veth003b6128" ip saddr 0.0.0.0 ip daddr 255.255.255.255 udp dport 67 accept # handle 344
                iifname "veth003b6128" arp saddr ip != 10.38.58.3 drop # handle 345
                iifname "veth003b6128" ip saddr != 10.38.58.3 drop # handle 346
                iifname "veth003b6128" ip6 saddr fe80::/10 ip6 daddr ff02::1:2 udp dport 547 accept # handle 347
                iifname "veth003b6128" ip6 saddr fe80::/10 ip6 daddr ff02::2 icmpv6 type nd-router-solicit accept # handle 348
                iifname "veth003b6128" icmpv6 type nd-router-advert drop # handle 349
                iifname "veth003b6128" icmpv6 type nd-neighbor-advert @nh,384,128 != 0x26075300006084010000000000000003 drop # handle 350
                iifname "veth003b6128" ip6 saddr != 2607:5300:60:8401::3 drop # handle 351
                iifname "veth003b6128" ether type != { ip, arp, ip6 } drop # handle 353
        }
}

But every time a VM is created or restarted, Incus resets the bridge incus chain and wipes out my custom rules.

My questions:

  • How can I make this nftables rule persistent across restarts?
  • Is there a way to apply it globally to all VMs instead of adding it per-interface?

Or can Incus fix this bug? It looks like it’s just a simple firewall rule configuration error.

16

Thanks, I’ll try to look at the Incus logic later today or tomorrow.

17

Just an update to say that I didn’t forget about this.

I actually spent some time last night looking at this. I can reproduce it here easily enough and the workaround certainly works, but it’s puzzling why the existing rule doesn’t match. The workaround also isn’t safe so not something we can just include and move on.

18

I might be on the wrong track here, but my understanding of why the original rule logic is not sufficient and why the workaround “fixes” it (but is too permissive to be suitable as proper fix, as implied by Stéphane):

The existing nft rules are applied on the bridge interface, and on inbound direction. However, “inbound” for a bridge interface is kind of ambiguous/deceptive, because traffic could be “inbound” from both the attached client and the server. The current Incus-generated rule of ip6 saddr fe80::/10 ip6 daddr ff02::1:2 udp dport 547 accept allows the client-originating DHCPv6 solicitation, from its link-local address to the DHCPv6 multicast address. A response would be allowed implicitly (because: stateful firewall), if it would originate from ff02::1:2 – which in reality it does not, because the DHCPv6 server is replying from its own link-local address and not from the multicast address (see traffic dump). Therefore, the existing rule alone will not suffice as there is no matching rule allowing the DHCPv6 server’s response.

xenon’s temp fix with an additional rule of ip6 saddr fe80::/10 accept “fixes” this issue because it simply allows any traffic from link-local addresses, including above response from the DHCPv6 server to the client. But the rule is way to broad, of course, creating a security issue. In my previous post, I had suggested a specific/narrow rule, only matching on the DHCPv6 server’s reply from fe80::/10 and the respective DHCPv6 server and client ports, but @xenon mentioned it did not work, could you please elaborate?

Regards

19

Yeah, I went down that path too, what doesn’t make sense is why does the initial DHCP request and response work fine but the subsequent one for renewal doesn’t.

Both of them use the exact same source and destination addresses in both directions, same port too and even same message size, MAC address and everything else.