Troubleshooting: incus network bridge with dhcp for ipv4 enabled on rocky9 not assigning an ipv4 to rocky9 system container

Incus
,
1

hello!
I’m currently trying to setup a network bridge with dhcp for only ipv4. I’m running incus on a rocky9 host. however, when I spin up a new system container using as base image rocky9 (note the default profile points to the default bridge, incusbr0) it does not get an ip from the range i have specified in the network bridge.

i’m out of ideas of what to check next, i’d be happy to receive pointers to further guide me in my troubleshooting or maybe some resources to read :slight_smile:

note: i have disabled nftables, iptables and the firewalld

image

image
image402×546 53.7 KB

bridge config

image
image692×229 28 KB

troubleshooting inside the system container

checking the dhclient command

image
image1873×39 38.1 KB

listening to the sockets connections

image
image1907×77 58.9 KB

troubleshooting on the host itself
checking the dnsmasq command

image
image1914×81 90.7 KB

listening to the socket connections

image
image1493×79 88.4 KB

tcp dump, note only seeing Discover requests, no answers from the dhcp server via the dnsmasq

image
image1361×292 84.4 KB

2

Still smells like firewalling, can you show:

  • iptables -L -n -v
  • nft list ruleset
3 
[root@REDACTED ~]# iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

For the nft table though it returned a ~6000 lines output, i’ve filtered down and emptied some tables for readability and not polluting.

What would you suggest I look into or make sure to set in general? the ports for dnsmasq ?

[root@REDACTED]# nft list ruleset
table inet ILO-FILTER {
   set ILON-<IDENTIFIER-1> {
     type ipv4_addr
     flags interval
     elements = { list of IPs }
  }
  ...
  set ILON-<IDENTIFIER-N> {
    type ipv4_addr
    flags interval
    elements = { list of IPs }
  }
  chain ILO-FILTER-INPUT {}
  chain ILO-FILTER-OUTPUT {}
  chain ILO-FILTER-FORWARD {}
  chain ILO-FILTER-INPUT-TCP {}
  chain ILO-FILTER-INPUT-UDP {}
  chain ILO-FILTER-INPUT-ICMP {}
  chain ILO-FILTER-INPUT-OTHER {}
  chain ILO-FILTER-INPUT-ENFORCE {}
  chain ILO-FILTER-OUTPUT-TCP {}
  chain ILO-FILTER-OUTPUT-UDP {}
  chain ILO-FILTER-OUTPUT-ICMP {}
  chain ILO-FILTER-OUTPUT-OTHER {}
  chain ILO-FILTER-OUTPUT-ENFORCE {}
  chain ILO-FILTER-FORWARD-TCP-IN {}
  chain ILO-FILTER-FORWARD-UDP-IN {}
  chain ILO-FILTER-FORWARD-ICMP-IN {}
  chain ILO-FILTER-FORWARD-OTHER-IN {}
  chain ILO-FILTER-FORWARD-TCP-OUT {}
  chain ILO-FILTER-FORWARD-UDP-OUT {}
  chain ILO-FILTER-FORWARD-ICMP-OUT {}
  chain ILO-FILTER-FORWARD-OTHER-OUT {}
  chain ILO-FILTER-INPUT-DROP-NOLOG {}
  chain ILO-FILTER-OUTPUT-DROP-NOLOG {}
  chain ILO-FILTER-INPUT-USR {}
  chain ILO-FILTER-FORWARD-USR {}
  chain ILO-FILTER-OUTPUT-USR {}
  chain ILO-FILTER-INPUT-CONNTRACK {}
  chain ILO-FILTER-CONNTRACK {}
  chain ILO-FILTER-ACCEPT {}
  chain ILO-FILTER-SYSTEM {}
  chain ILO-FILTER-DROP {}
  chain ILO-FILTER-BOUNDARY-DROP {}
  chain ILO-REJECT {}
  chain ILO-FILTER-INPUT-REJECT {}
  chain ILO-FILTER-INPUT-BOUNDARY-REJECT {}
  chain ILO-FILTER-FQDN-DROP {}
  chain ILO-FILTER-ILLUMINATE {}
  chain ILO-FILTER-BOUNDARY-ILLUM {}
  chain ILO-RAW-PREROUTING {}
  chain ILO-RAW-OUTPUT {}
  chain ILO-RAW-PREROUTING-TCP {}
  chain ILO-RAW-PREROUTING-UDP {}
  chain ILO-RAW-PREROUTING-ICMP {}
  chain ILO-RAW-PREROUTING-OTHER {}
  chain ILO-RAW-OUTPUT-TCP {}
  chain ILO-RAW-OUTPUT-UDP {}
  chain ILO-RAW-OUTPUT-ICMP {}
  chain ILO-RAW-OUTPUT-OTHER {}
  chain ILO-NAT-PREROUTING {}
  chain ILO-NAT-POSTROUTING {}
  chain ILO-NAT-INPUT {}
  chain ILO-NAT-OUTPUT {}
  chain ILO-NAT-PREROUTING-USR {}
  chain ILO-NAT-POSTROUTING-USR {}
  chain ILO-NAT-INPUT-USR {}
  chain ILO-NAT-OUTPUT-USR {}
  chain ILO-NAT-ACCEPT {}
  chain ILO-MANGLE-PREROUTING {}
  chain ILO-MANGLE-POSTROUTING {}
  chain ILO-MANGLE-INPUT {}
  chain ILO-MANGLE-OUTPUT {}
  chain ILO-MANGLE-FORWARD {}
  chain ILO-MANGLE-MARK-NATT-IN {}
  chain ILO-MANGLE-PREROUTING-USR {}
  chain ILO-MANGLE-INPUT-USR {}
  chain ILO-MANGLE-FORWARD-USR {}
  chain ILO-MANGLE-OUTPUT-USR {}
  chain ILO-MANGLE-POSTROUTING-USR {}
  chain ILO-MANGLE-ACCEPT {}
  chain ILO-MANGLE-DROP {}  
}
4

Putting my thoughts:
so when a container is started, it runs

root         119  0.0  0.0   5580  4204 ?        S    02:13   0:00 /usr/sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /run/NetworkManager/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03-eth0.lease -cf /var/lib/Netwo

at (port 68)

# ss -laputen
Netid                       State                        Recv-Q                       Send-Q                                             Local Address:Port                                             Peer Address:Port                      Process
udp                         UNCONN                       0                            0                                                        0.0.0.0:68                                                    0.0.0.0:*                          users:(("dhclient",pid=119,fd=7)) ino:806005671 sk:3002 cgroup:/system.slice/NetworkManager.service <->

listening on the host

# tcpdump -i incusbr0 udp port 68
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on incusbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
22:17:09.321482 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:83:46:52 (oui Unknown), length 300

and on the host, it seems the dnsmasq is correctly listening to the bridge interface but on the port 67 which is not the port the dhclient is sending requests to as shown above? port 68

]# ss -laputen | grep dnsmasq
udp   UNCONN    0      0                10.46.30.1:53                    0.0.0.0:*     users:(("dnsmasq",pid=3098641,fd=6)) ino:788017155 sk:1 cgroup:/user.slice/user-3185488.slice/session-12351.scope <->                                                            
udp   UNCONN    0      0          0.0.0.0%incusbr0:67                    0.0.0.0:*     users:(("dnsmasq",pid=3098641,fd=4)) ino:788013056 sk:2 cgroup:/user.slice/user-3185488.slice/session-12351.scope <->                                                            
tcp   LISTEN    0      32               10.46.30.1:53                    0.0.0.0:*     users:(("dnsmasq",pid=3098641,fd=7)) ino:788017156 sk:1007 cgroup:/user.slice/user-3185488.slice/session-12351.scope <->

lastly, the following chains might be relevant?

	chain ILO-FILTER-OUTPUT-UDP {
		ip protocol udp udp sport 68 udp dport 67 counter packets 0 bytes 0 goto ILO-FILTER-ACCEPT
		...
		ip protocol udp udp dport 53 counter packets 36 bytes 2602 goto ILO-FILTER-ACCEPT
        ...
		goto ILO-FILTER-OUTPUT-OTHER
	}

	chain ILO-FILTER-OUTPUT {
		type filter hook output priority filter - 1; policy accept;
		oif != "lo" ct state untracked accept
		oif "lo" accept
		ct state established,related goto ILO-FILTER-CONNTRACK
		...
		meta l4proto udp goto **ILO-FILTER-OUTPUT-UDP**
		...
		goto ILO-FILTER-OUTPUT-OTHER
	}

the ipv4 address (10.46.30.1) of my network bridge incusbr0 is in the sets in the chains below

	chain ILO-FILTER-INPUT-OTHER {
		ip saddr @REDACTED counter packets 24 bytes 4253 goto ILO-FILTER-ACCEPT
		ip protocol igmp counter packets 1 bytes 36 goto ILO-FILTER-ACCEPT
		goto ILO-FILTER-INPUT-ENFORCE
	}

	chain ILO-FILTER-OUTPUT-OTHER {
		ip daddr @REDACTED counter packets 46 bytes 9191 goto ILO-FILTER-ACCEPT
		ip protocol igmp ip daddr REDACTED counter packets 0 bytes 0 goto ILO-FILTER-ACCEPT
		goto ILO-FILTER-OUTPUT-ENFORCE
	}

does this paint a better picture?

5

What’s creating all those nftables rules on your system?

6

Illumio is responsible for creating those nftables. I’m trying to make this setup work despite Illumio. Seems I will have to make some changes to the illumio for this machine.
If this works, that would be great doc for future readers.