Unable to start container or create network on Debian 11

LXD
1

OS: Debian 11. LXD was working fine until certain point, though I’m not sure exactly when. Now when I try to start a container I get

Bash 5.1 :) lxc start ubcon
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
Error: Failed preparing container for start: Failed to start device "eth0": Parent device "lxdbr0" doesn't exist
Try `lxc info --show-log ubcon` for more info

But log is empty:

bash 5.1 :) lxc info --show-log ubcon
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
Name: ubcon
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2021/05/20 05:13 UTC
Last Used: 2021/07/20 14:07 UTC

Log:

And lxdbr0 appears in network list

bash 5.1 :) lxc network list
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
|  NAME  |   TYPE   | MANAGED |      IPV4       |           IPV6            | DESCRIPTION | USED BY |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| lxdbr0 | bridge   | YES     | 10.208.106.1/24 | fd42:22de:f638:b119::1/64 |             | 2       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| virbr0 | bridge   | NO      |                 |                           |             | 0       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+
| wlp1s0 | physical | NO      |                 |                           |             | 0       |
+--------+----------+---------+-----------------+---------------------------+-------------+---------+

But doesn’t show up in system as an interface

bash 5.1 :) ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: wlp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7c:2a:31:17:88:dd brd ff:ff:ff:ff:ff:ff
    inet 192.168.223.83/24 brd 192.168.223.255 scope global dynamic noprefixroute wlp1s0
       valid_lft 2618sec preferred_lft 2618sec
    inet6 2607:fb90:3127:317f:946d:a9ed:408c:a310/64 scope global dynamic noprefixroute 
       valid_lft 3311sec preferred_lft 3311sec
    inet6 fe80::bace:b177:dbed:4d24/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:14:a7:f2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

Upon trying to create a new bridge, I encounter

bash 5.1 :) lxc network create lxdbr1
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
Error: Failed to setup firewall: Failed adding ICMP, DHCP and DNS access rules for network "lxdbr1" (inet): Failed apply nftables config: Failed to run: nft 
table inet lxd {
chain in.lxdbr1 {
	type filter hook input priority 0; policy accept;

	iifname "lxdbr1" tcp dport 53 accept
	iifname "lxdbr1" udp dport 53 accept
	iifname "lxdbr1" icmp type {3, 11, 12} accept
	iifname "lxdbr1" udp dport 67 accept
	
	iifname "lxdbr1" icmpv6 type {1, 2, 3, 4, 133, 135, 136, 143} accept
	iifname "lxdbr1" udp dport 547 accept
}

chain out.lxdbr1 {
	type filter hook output priority 0; policy accept;

	oifname "lxdbr1" tcp sport 53 accept
	oifname "lxdbr1" udp sport 53 accept
	oifname "lxdbr1" icmp type {3, 11, 12} accept
	oifname "lxdbr1" udp sport 67 accept
	
	oifname "lxdbr1" icmpv6 type {1, 2, 3, 4, 128, 134, 135, 136, 143}  accept
	oifname "lxdbr1" udp sport 547 accept
}
}
: Error: Could not process rule: Operation not supported

                            ^^^^^^^^^^^
Error: Could not process rule: No such file or directory

                            ^^^^^^^^^^^
Error: Could not process rule: Operation not supported

                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: Operation not supported

                            ^^^^^^^^^^^
Error: Could not process rule: No such file or directory

                            ^^^^^^^^^^^
Error: Could not process rule: Operation not supported

                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

And upon examining nftables, lxd rules are already there

bash 5.1 :) sudo nft list ruleset
table ip filter {
	chain LIBVIRT_INP {
		iifname "virbr0" meta l4proto udp udp dport 53 counter packets 0 bytes 0 accept
		iifname "virbr0" meta l4proto tcp tcp dport 53 counter packets 0 bytes 0 accept
		iifname "virbr0" meta l4proto udp udp dport 67 counter packets 0 bytes 0 accept
		iifname "virbr0" meta l4proto tcp tcp dport 67 counter packets 0 bytes 0 accept
	}

	chain INPUT {
		type filter hook input priority filter; policy accept;
		counter packets 794 bytes 98210 jump LIBVIRT_INP
	}

	chain LIBVIRT_OUT {
		oifname "virbr0" meta l4proto udp udp dport 53 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp tcp dport 53 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto udp udp dport 68 counter packets 0 bytes 0 accept
		oifname "virbr0" meta l4proto tcp tcp dport 68 counter packets 0 bytes 0 accept
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
		counter packets 865 bytes 66477 jump LIBVIRT_OUT
	}

	chain LIBVIRT_FWO {
		iifname "virbr0" ip saddr 192.168.122.0/24 counter packets 0 bytes 0 accept
		iifname "virbr0" counter packets 0 bytes 0 reject
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		counter packets 0 bytes 0 jump LIBVIRT_FWX
		counter packets 0 bytes 0 jump LIBVIRT_FWI
		counter packets 0 bytes 0 jump LIBVIRT_FWO
	}

	chain LIBVIRT_FWI {
		oifname "virbr0" ip daddr 192.168.122.0/24 ct state related,established counter packets 0 bytes 0 accept
		oifname "virbr0" counter packets 0 bytes 0 reject
	}

	chain LIBVIRT_FWX {
		iifname "virbr0" oifname "virbr0" counter packets 0 bytes 0 accept
	}
}
table ip nat {
	chain LIBVIRT_PRT {
		ip saddr 192.168.122.0/24 ip daddr 224.0.0.0/24 counter packets 6 bytes 413 return
		ip saddr 192.168.122.0/24 ip daddr 255.255.255.255 counter packets 0 bytes 0 return
		meta l4proto tcp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 
		meta l4proto udp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade to :1024-65535 
		ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 counter packets 0 bytes 0 masquerade 
	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		counter packets 395 bytes 28317 jump LIBVIRT_PRT
	}
}
table ip mangle {
	chain LIBVIRT_PRT {
	}

	chain POSTROUTING {
		type filter hook postrouting priority mangle; policy accept;
		counter packets 952 bytes 75272 jump LIBVIRT_PRT
	}
}
table ip6 filter {
	chain LIBVIRT_INP {
	}

	chain INPUT {
		type filter hook input priority filter; policy accept;
		counter packets 11448 bytes 10059142 jump LIBVIRT_INP
	}

	chain LIBVIRT_OUT {
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
		counter packets 10852 bytes 2251399 jump LIBVIRT_OUT
	}

	chain LIBVIRT_FWO {
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		counter packets 0 bytes 0 jump LIBVIRT_FWX
		counter packets 0 bytes 0 jump LIBVIRT_FWI
		counter packets 0 bytes 0 jump LIBVIRT_FWO
	}

	chain LIBVIRT_FWI {
	}

	chain LIBVIRT_FWX {
	}
}
table ip6 nat {
	chain LIBVIRT_PRT {
	}

	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		counter packets 60 bytes 13967 jump LIBVIRT_PRT
	}
}
table ip6 mangle {
	chain LIBVIRT_PRT {
	}

	chain POSTROUTING {
		type filter hook postrouting priority mangle; policy accept;
		counter packets 10881 bytes 2254933 jump LIBVIRT_PRT
	}
}
table inet lxd {
	chain pstrt.lxdbr0 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 10.208.106.0/24 ip daddr != 10.208.106.0/24 masquerade
		ip6 saddr fd42:22de:f638:b119::/64 ip6 daddr != fd42:22de:f638:b119::/64 masquerade
	}

	chain fwd.lxdbr0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdbr0" accept
		ip version 4 iifname "lxdbr0" accept
		ip6 version 6 oifname "lxdbr0" accept
		ip6 version 6 iifname "lxdbr0" accept
	}

	chain pstrt.lxdbr1 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 10.53.48.0/24 ip daddr != 10.53.48.0/24 masquerade
		ip6 saddr fd42:ff9:37f6:fcde::/64 ip6 daddr != fd42:ff9:37f6:fcde::/64 masquerade
	}

	chain fwd.lxdbr1 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdbr1" accept
		ip version 4 iifname "lxdbr1" accept
		ip6 version 6 oifname "lxdbr1" accept
		ip6 version 6 iifname "lxdbr1" accept
	}
}

I thought it could be due to something I messed up when I rebuilt the kernel, but comparing my kernel to that around 5/20 (when container was created) and 7/20 (when it was last used), I don’t see any netfilter option being unchecked.

2

The command lxc warning list may show you an error that LXD is having starting lxdbr0 interface and firewall.

Yes if you’re running a different kernel config without the required nftables support that could well be the issue.

Can you try switching back to the old one to confirm that is the issue before trying anything else.

3

I tried the older kernels and am still unable to start any container or create any network. lxc warning list produces “could not find the CGroup hugetlb controller” and “could not find the CGroup network priority controller.” The latter is odd because CGroup network controller module is definitely built in the kernel. Suspecting an issue with CGroup2, I added systemd.unified_cgroup_hierarchy=0 to the kernel boot command line (even though I did not have to do this for lxc to work in the past), but I still can’t start any network or container, although ostensibly now the reason is because Error: Failed generating auto config: Failed to automatically find an unused IPv6 subnet, manual configuration required. And if I skip IPv6 in network creation, I’m back at

Error: Failed to create network "lxdbr0" in project "default": Failed to setup firewall: Failed adding ICMP, DHCP and DNS access rules for network "lxdbr0" (inet): Failed apply nftables config: Failed to run: nft 
table inet lxd {
chain in.lxdbr0 {
	type filter hook input priority 0; policy accept;

	iifname "lxdbr0" tcp dport 53 accept
	iifname "lxdbr0" udp dport 53 accept
	iifname "lxdbr0" icmp type {3, 11, 12} accept
	iifname "lxdbr0" udp dport 67 accept
	
}

chain out.lxdbr0 {
	type filter hook output priority 0; policy accept;

	oifname "lxdbr0" tcp sport 53 accept
	oifname "lxdbr0" udp sport 53 accept
	oifname "lxdbr0" icmp type {3, 11, 12} accept
	oifname "lxdbr0" udp sport 67 accept
	
}
}
: Error: Could not process rule: Operation not supported

                            ^^^^^^^^^^^
Error: Could not process rule: No such file or directory

                            ^^^^^^^^^^^
Error: Could not process rule: Operation not supported

                            ^^^^^^^^^^^
Error: Could not process rule: No such file or directory

                            ^^^^^^^^^^^
Error: Could not process rule: No such file or directory

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4

I thought I would try this in a fresh Debian 11 VM to see if it works, and it seems fine on a standard fresh install.

I used a LXD VM for testing, here are my steps.

lxc launch images:debian/11 v1 --vm
lxc shell v1

Inside the VM:

uname -a
Linux vtest 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 GNU/Linux

apt install snapd -y
snap install lxd
2021-08-31T08:48:16Z INFO Waiting for automatic snapd restart...
Warning: /snap/bin was not found in your $PATH. If you've not restarted your session since you
         installed snapd, try doing that. Please see https://forum.snapcraft.io/t/9469 for more
         details.

lxd 4.17 from Canonical✓ installed

Restart session to get PATH set correctly.

exit
lxc shell v1

Initialise LXD and check firewall driver detected if nftables:

lxd init --minimal
lxc info | grep firewall:
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
  firewall: nftables

Install nftables and check LXD’s ruleset:

apt install nftables -y
nft list ruleset
table inet lxd {
	chain pstrt.lxdbr0 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 10.133.131.0/24 ip daddr != 10.133.131.0/24 masquerade
		ip6 saddr fd42:1fe7:82f1:f94b::/64 ip6 daddr != fd42:1fe7:82f1:f94b::/64 masquerade
	}

	chain fwd.lxdbr0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdbr0" accept
		ip version 4 iifname "lxdbr0" accept
		ip6 version 6 oifname "lxdbr0" accept
		ip6 version 6 iifname "lxdbr0" accept
	}

	chain in.lxdbr0 {
		type filter hook input priority filter; policy accept;
		iifname "lxdbr0" tcp dport 53 accept
		iifname "lxdbr0" udp dport 53 accept
		iifname "lxdbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		iifname "lxdbr0" udp dport 67 accept
		iifname "lxdbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
		iifname "lxdbr0" udp dport 547 accept
	}

	chain out.lxdbr0 {
		type filter hook output priority filter; policy accept;
		oifname "lxdbr0" tcp sport 53 accept
		oifname "lxdbr0" udp sport 53 accept
		oifname "lxdbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		oifname "lxdbr0" udp sport 67 accept
		oifname "lxdbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
		oifname "lxdbr0" udp sport 547 accept
	}
}

lxc warning ls
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+--------------------------------------+------------------------------------------------------+--------+----------+-------+---------+------------------------------+
|                 UUID                 |                         TYPE                         | STATUS | SEVERITY | COUNT | PROJECT |          LAST SEEN           |
+--------------------------------------+------------------------------------------------------+--------+----------+-------+---------+------------------------------+
| 28c38208-f48e-4f32-a844-4fc278197da2 | Couldn't find the CGroup network priority controller | NEW    | LOW      | 1     |         | Aug 31, 2021 at 8:48am (UTC) |
+--------------------------------------+------------------------------------------------------+--------+----------+-------+---------+------------------------------+

Check launching container works:

lxc launch images:debian/11 c1
lxc ls c1
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4          |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| c1   | RUNNING | 10.133.131.197 (eth0) | fd42:1fe7:82f1:f94b:216:3eff:feb5:43e0 (eth0) | CONTAINER | 0         |
+------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
5

Do you have IPv6 disabled in your kernel?

I wonder if its being tripped up by the PR that @stgraber added recently to allow inbound ICMP to the firewall:

It is allowing IPv6 ICMP, but perhaps if you have that disabled in your kernel then its not allowed to be referenced in the firewall rules.

6

You do seem to have IPv6 enabled, and yet it looks like both the " Failed to automatically find an unused IPv6 subnet" and the nftables error could be IPv6 related.

Can you try running lxc network set lxdbr0 ipv6.address=none or lxc network set lxdbr0 ipv6.firewall=false to see if that fixes it, then at least we know its an IPv6 issue.

7

Thanks! Since it’s a fresh install I lost lxdbr0 and was unable to create one. I’ve tried the following couple combinations:

bash 5.1 :) lxc network create lxdbr1 ipv6.address=none ipv4.firewall=false
Network lxdbr1 created

bash 5.1 :) lxc network create lxdbr2 ipv4.address=false ipv6.firewall=false
Error: Failed generating auto config: Failed to automatically find an unused IPv6 subnet, manual configuration required

bash 5.1 :) lxc network create lxdbr3 ipv6.address=none
Error: Failed to setup firewall: Failed adding ICMP, DHCP and DNS access rules for network "lxdbr3" (inet): Failed apply nftables config: Failed to run: nft 
table inet lxd {
chain in.lxdbr3 {
	type filter hook input priority 0; policy accept;

	iifname "lxdbr3" tcp dport 53 accept
	iifname "lxdbr3" udp dport 53 accept
	iifname "lxdbr3" icmp type {3, 11, 12} accept
	iifname "lxdbr3" udp dport 67 accept
	
}

chain out.lxdbr3 {
	type filter hook output priority 0; policy accept;

	oifname "lxdbr3" tcp sport 53 accept
	oifname "lxdbr3" udp sport 53 accept
	oifname "lxdbr3" icmp type {3, 11, 12} accept
	oifname "lxdbr3" udp sport 67 accept
	
}
}
: Error: Could not process rule: Operation not supported

                            ^^^^^^^^^^^
Error: Could not process rule: No such file or directory

                            ^^^^^^^^^^^
Error: Could not process rule: Operation not supported

                            ^^^^^^^^^^^
Error: Could not process rule: No such file or directory

                            ^^^^^^^^^^^
Error: Could not process rule: No such file or directory

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

So disabling both ipv6 and firewall rules for ipv4 leads to the successful creation of the bridge, but it fails at either adding ipv6 or having ipv4 firewall rules. I should mention that at this point I’ve enabled all the netfilter kernel modules possible including (some are built-in and some load by default, although most of them do not automatically load at boot and I haven’t tried the exhaustive method yet).

8

Are you building your own kernel? Gave you tried switching back to the normal one and does that fix it?

9

I am. There’s no normal kernel, since it’s the ChromiumOS kernel for Chromebook hardware. When I switched to what worked before, for some reason it stopped working. My netfilter config is identical to that of Debian Bullseye shipped kernel (which checks everything but 1 box), though it could very well be that certain required netfilter module wasn’t loaded. I can try building everything into the kernel, though that won’t isolate the problem. I thought it could be because I was using a mobile phone hotspot, but on my home network the issue persists, and I have not been successful setting up an lxd network using an open VPN device like tun0. Just using ipv4 without adding firewall rules works for my purpose on this particular laptop, though.

10

I think I know what the problem is.

Can you try running:

sudo sudo nft add table inet test

Also, what version of LXD is this?

Also can you reload LXD and look for entries in the log mentioning firewall.

I think your kernel version is too old (requires >= 5.2) and doesn’t support nftable’s inet table type.
LXD should check for this and try and use iptables in this case, but perhaps there are no xtables compatible tools either and its falling back to partial nftables support.

See lxd/drivers_nftables.go at master · lxc/lxd · GitHub

11

The Chromium OS kernel version is based on Linux 5.4 so it should be recent enough. It turned out to be a kernel configuration, after all. After some tinkering, I finally got it to work by default behavior, and compared the modules between the successful kernel and the prior one. The offending omission is that of module nf_tables_set, with which loaded the network function behaves as intended.

Thank you @tomp for your help! I wish there were documentations on the kernel configuration requirements for lxd, at least for network stuff. I had similar problems with getting the network to work for libvirt as well building up from the stock ChromeOS kernel for the hardware, which is rather lean, although in that case it was immediately obvious from the error message what was missing (missing module for a specific netfilter target).

1 Like