Runnning LXD in WSL2 Kali

hypeit · July 11, 2022, 3:38pm

Hi all,

I was trying to run lxd containters in wsl2 distro(Kali linux), I will start with some steps I have taken so far:
1. Installed wsl2 Kali image from MS Store with the latest kernel.
2. Converted it to WSL2 and installed distrod from https://github.com/nullpo-head/wsl-distrod
3. Installed snapd and lxd in stable channel. (Maybe it would be worth poining that I have not done any change to my WSL network configuration)
4. Tried to run
lxd init --minimal
(I knew the default options an I was fine with it) but got following error message:

table inet lxd {
chain pstrt.lxdbr0 {
        type nat hook postrouting priority 100; policy accept;
        ip saddr 10.90.183.0/24 ip daddr != 10.90.183.0/24 masquerade
}
}
: Error: Could not process rule: No such file or directory

  ^^^^^^^^^^^^
Error: Could not process rule: No such file or directory

   ^^^^^^^^^^^^

I ended up with specifying my current anapter eth0 as default network, but it does not allow the containers I create to connect to the internet.
This error seems weird, due to the fact that I did not install nftables in any form or any other firewall utility.
What I tried was to create network of type bridge with ipv4 firewall disabled

lxc network createlxdbr1 ipv6.address=none ipv4.firewall=false

as well as to install nftables and add the chain manually

nft add chain inet lxd pstrt.lxdbr0 { type net hook postrouting priority 100 \; }

The same setup works for Ubuntu 20.04 in WSL2 with distrod installed, what am I missing here?

tomp · July 11, 2022, 6:40pm

Sounds like the kernel is missing some nftables or net filter functionality.

hypeit · July 11, 2022, 7:30pm

the lxd check output:

/snap/lxd/23315/bin/lxc-checkconfig: 72: lxc-start: not found
LXC version
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/cpu
/sys/fs/cgroup/cpuacct
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/devices
/sys/fs/cgroup/freezer
/sys/fs/cgroup/net_cls
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/net_prio
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/pids
/sys/fs/cgroup/rdma
/sys/fs/cgroup/systemd

Cgroup v2 mount points:
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /snap/lxd/23315/bin/lxc-checkconfig```

Is there any other way to check it?

tomp · July 12, 2022, 8:03am

What do you see for lsmod | grep nft I see on Ubuntu 22.04:

sudo lsmod | grep nft
nft_reject_inet        16384  4
nf_reject_ipv4         16384  1 nft_reject_inet
nf_reject_ipv6         20480  1 nft_reject_inet
nft_reject             16384  1 nft_reject_inet
nft_ct                 20480  2
nft_masq               16384  4
nft_chain_nat          16384  2
nf_tables             241664  499 nft_ct,nft_reject_inet,nft_masq,nft_chain_nat,nft_reject
nf_nat                 49152  5 ip6table_nat,openvswitch,nft_masq,nft_chain_nat,iptable_nat
nf_conntrack          167936  6 nf_nat,nft_ct,nfnetlink_cttimeout,openvswitch,nft_masq,nf_conncount

hypeit · July 12, 2022, 8:43am

I am afraid, it will be an empty list, since WSL already has all available modules compiled in, is recompiling the only solution here?

hypeit · July 12, 2022, 10:35am

I was wondering if the logic cited here should not behave as if it was xtables?

github.com

lxc/lxd/blob/master/lxd/firewall/firewall_load.go

package firewall

import (
	"github.com/lxc/lxd/lxd/firewall/drivers"
	"github.com/lxc/lxd/shared/logger"
)

// New returns an appropriate firewall implementation.
// Uses xtables if nftables isn't compatible or isn't in use already, otherwise uses nftables.
func New() Firewall {
	nftables := drivers.Nftables{}
	xtables := drivers.Xtables{}

	nftablesInUse, nftablesCompatErr := nftables.Compat()
	if nftablesCompatErr != nil {
		logger.Debugf(`Firewall detected "nftables" incompatibility: %v`, nftablesCompatErr)
	} else if nftablesInUse {
		// If nftables is compatible and already in use, then we prefer to use the nftables driver
		// irrespective of whether xtables is in use or not.
		return nftables

This file has been truncated. show original

Most likely I am missing something but from the crrent WSL2 kernel I see that xtables modules are loaded during compilation

github.com

microsoft/WSL2-Linux-Kernel/blob/linux-msft-wsl-5.10.y/Microsoft/config-wsl

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 5.10.102.1 Kernel Configuration
#
CONFIG_CC_VERSION_TEXT="x86_64-msft-linux-gcc (GCC) 9.3.0"
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=90300
CONFIG_LD_VERSION=234000000
CONFIG_CLANG_VERSION=0
CONFIG_LLD_VERSION=0
CONFIG_CC_HAS_ASM_GOTO=y
CONFIG_CC_HAS_ASM_INLINE=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_TABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32

This file has been truncated. show original

tomp · July 12, 2022, 10:37am

What does lxc info | grep firewall: show?

tomp · July 12, 2022, 10:39am

Also are you using ipv6 if not have you tried turning off ipv6 on the bridge, using lxc network set lxdbr0 ipv6.address=none? That might cure it.

hypeit · July 12, 2022, 10:55am

Yes, and it did not change anything, unfortunately

tomp · July 12, 2022, 11:00am

And this one?

hypeit · July 12, 2022, 11:32am

nftables

tomp · July 12, 2022, 12:05pm

Try running sudo nft flush ruleset then adding a single iptables rule using iptables -A INPUT and then reload LXD using sudo systemctl reload snap.lxd.daemon and then re-run that command and see what the firewall driver is.

hypeit · July 12, 2022, 6:11pm

Unfortunately that did not help too

21:49:41 ❯ sudo nft flush ruleset
20:07:13 ❯ iptables -A INPUT
iptables v1.8.8 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)
20:07:23 ❯ sudo iptables -A INPUT
iptables v1.8.8 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain INPUT
20:07:30 ❯  sudo systemctl reload snap.lxd.daemon
snap.lxd.daemon.service is not active, cannot reload.
20:07:42 ❯  sudo systemctl start snap.lxd.daemon
20:07:51 ❯  sudo systemctl reload snap.lxd.daemon
20:07:57 ❯ lxc info | grep firewall:
  firewall: nftables

Unfortunately iptables seems to not even accept custom rules

└─# iptables -L
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

iptables-legacy -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
┌──(root㉿DESKTOP-LGTDTIN)-[~]
└─# nft list tables

┌──(root㉿DESKTOP-LGTDTIN)-[~]
└─# iptables -A INPUT  -p tcp --dport 22 -j ACCEPT
Warning: Extension tcp revision 0 not supported, missing kernel module?
iptables v1.8.8 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain INPUT

hypeit · July 12, 2022, 7:07pm

Later I tried

iptables -A INPUT -i eth0 -j ACCEPT
iptables v1.8.8 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain INPUT

but it definitely confirms that something is missing here

tomp · July 13, 2022, 7:14am

It looks like your iptables command is just a wrapper to nftables, so not surprising its experiencing the same issues.

Suggest you remove nftables and iptables (nf_tables) version and then install iptables legacy (I don’t know how this is done in Kali).

Then reboot and see if LXD picks up support for iptables legacy and uses that instead.

hypeit · September 8, 2022, 6:34am

Thank you very much for your help, I ended up with:

Rebuilding my kernel using the config uploaded here Kernel config for my custom WSL kernel 5.10.60.1 · GitHub
which I slightly modified but, did not twiddle anything with the NFTables support
removing nft (sudo apt remove nftables)
restarted machine with new kernel

In the menatime I came up with a simplier solution(update-alternatives). This could point the system to use iptables legacy independent on the packages that are present