LXD 5.0.1 on Ubuntu 22.04 no longer creates NAT rules

LXD
1

We are moving from Ubuntu 20.04 with LXD 5.2 to Ubuntu 22.04 with LXD 5.0.1. All hosts are reprovisioned, i.e. are clean installs. The install is very close to identical with our 20.04 installer, but something seems to cause an issue with LXD NAT rule generation and I cannot find what is causing it.

The short of it is that the relevant rules in the nat table of iptables do not get generated: no masquerade, no proxy directive forwards where nat is set to true.

Here are the (lack of) insights I have gathered so far:


$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
time="2022-09-27T10:47:27Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
time="2022-09-27T10:47:27Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine

$ lxc warning list
+--------------------------------------+------------------------------------------------------+--------+----------+-------+---------+-------------------------------+
|                 UUID                 |                         TYPE                         | STATUS | SEVERITY | COUNT | PROJECT |           LAST SEEN           |
+--------------------------------------+------------------------------------------------------+--------+----------+-------+---------+-------------------------------+
| 537b255b-4de2-44b8-8174-f80cb1f0caa4 | Couldn't find the CGroup network priority controller | NEW    | LOW      | 3     |         | Sep 27, 2022 at 10:47am (UTC) |
+--------------------------------------+------------------------------------------------------+--------+----------+-------+---------+-------------------------------+
| e12e920c-ff06-4491-be62-3b18384e91bb | Instance type not operational                        | NEW    | LOW      | 3     |         | Sep 27, 2022 at 10:47am (UTC) |
+--------------------------------------+------------------------------------------------------+--------+----------+-------+---------+-------------------------------+

$ lxd.check-kernel 
/snap/lxd/23541/bin/lxc-checkconfig: 72: lxc-start: not found
LXC version 
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-5.15.0-48-generic
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled

Cgroup v1 mount points: 


Cgroup v2 mount points: 
/sys/fs/cgroup

Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup ns_cgroup: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /snap/lxd/23541/bin/lxc-checkconfig

$ lxc network show lxdbr0 
config:
  dns.search: i.molnix.com
  ipv4.address: 10.10.20.1/24
  ipv4.nat: "true"
  ipv4.nat.order: after
  ipv6.address: fd42:5842:ca6d:3ec2::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/test
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

$ lxc config device add t1:test tcp83.136.254.6port88 proxy listen=tcp:83.136.254.6:88 connect=tcp:10.10.20.5:88 nat=true

$ lxc monitor --type=logging | grep Success
  message: Success for operation

Any idea where to look further?

2

lxc info | grep firewall: would be a good start to see what firewall driver LXD is using.

Then showing output of lxc network list and sudo iptables-save and sudo nft list ruleset should show where things are being configured.

3

Also the contents of /var/snap/lxd/common/lxd/logs/lxd.log would be useful

4

OK, I think you nailed it on this one.

22.04 host that is not working

$ lxc info | grep firewall
- network_firewall_filtering
- firewall_driver
  firewall: nftables

$ sudo nft list ruleset
<long list including all the missing rules>

$ sudo iptables-save -t nat
# Generated by iptables-save v1.8.7 on Tue Sep 27 17:05:10 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [229:17587]
-A POSTROUTING -s 10.10.20.5/32 ! -d 10.0.0.0/8 -j SNAT --to-source 83.136.254.6
COMMIT
# Completed on Tue Sep 27 17:05:10 2022

20.04 instance that is working

$ lxc info | grep firewall
- network_firewall_filtering
- firewall_driver
  firewall: xtables

$ sudo nft list ruleset
sudo: nft: command not found

$ sudo iptables-save -t nat
<long list including all the expected rules>

Common

lxd.log is actually already above; just two lines logged.

both hosts have the same networks just IP differs:

$  lxc network list
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
|  NAME  |   TYPE   | MANAGED |     IPV4      |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
| ens3   | physical | NO      |               |                           |             | 0       |         |
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
| ens4   | physical | NO      |               |                           |             | 0       |         |
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
| lxdbr0 | bridge   | YES     | 10.10.20.1/24 | fd42:5842:ca6d:3ec2::1/64 |             | 2       | CREATED |
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
1 Like
5

I remember we stumbled on the nftables being incorrectly selected once before. I will try to find the case and link here.

6

Found it: LXD stopped generating firewall rules after switch to core20 - #12 by johanehnberg

That case was a bug in LXD around 4.15 as I understand it. This time may be different.

7

This boils down to a number of things:

  1. Ubuntu 22.04 uses nftables as default firewall
  2. iptables - but not iptables-save - transparently works as before
  3. ufw seems to create rules in both iptables as well as nftables, and while it is supposed to be able to use both transparently, that does not apply to before.rules etc. ufw default rules have changed and iptables only accesses one table in nftables
  4. LXD tries to autoselect the right firewall in an ambiguous situation

I may opt to migrate everything to nftables and/or switch out UFW.

8

You could try this to force ufw to use only classic iptables rather than the partially implemented iptables nftables shim:

Recommend rebooting afterwards to ensure all rulesets are cleanly recreated.

9

Since we are still pre-rollout, I want the new defaults to work, i.e. build the orchestration for nftables and not rely on the shim.

A key observation I made is that while iptables mostly smoothly translates basic filter rules back and forth, rules relevant to LXD completely fail to show up with the usual commands. For example, iptables-save -t nat no longer represents all nat rules. This is due to LXD putting them in its own table (table inet lxd) rather than the system default table (table ip nat). This was not documented in release notes, so I completely missed it when designing the updates for our installer.

And as a correction to my comment above, ufw does not generate rules for legacy iptables in parallel. I was simply confused by the fact that the rules reside in separate tables, of which iptables only shows the one with ufw-generated rules.

Now I just have to figure out why the same proxy device declarations no longer work on 22.04 with nftables.

10

My understanding is that there isn’t a system default table for native nftables. But rather those tables are used by the iptables nftables shim.

11

This might be useful if you’re using ufw too

12

Indeed, with ‘system default’ I meant those implied by iptables-nft.

The firewall is already configured with the additional ufw rules:

sudo ufw allow in on lxdbr0
sudo ufw route allow in on lxdbr0

But the proxy device rule’s traffic is still being blocked.

$ sudo tcpdump -i ens3 -nn -s0 -v port 80
tcpdump: listening on ens3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:50:14.559054 IP (tos 0x10, ttl 52, id 11758, offset 0, flags [DF], proto TCP (6), length 60)
    91.190.196.54.33434 > 83.136.254.6.80: Flags [S], cksum 0xdc8d (correct), seq 3856753587, win 64240, options [mss 1460,sackOK,TS val 3459933251 ecr 0,nop,wscale 7], length 0

[83363.638934] [UFW BLOCK] IN=ens3 OUT=lxdbr0 MAC=a6:51:e6:3d:1f:f7:9a:7e:00:6e:f9:36:08:00 SRC=91.190.196.54 DST=10.10.20.5 LEN=60 TOS=0x10 PREC=0x00 TTL=51 ID=53051 DF PROTO=TCP SPT=59186 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
13

You’ll most likely also need:

sudo ufw allow out on lxdbr0
14

Default output is ACCEPT. I also tried adding the rule but as expected it had no effect.

15

Maybe this is the one dropping it:

-A ufw-skip-to-policy-forward -j DROP

16

Indeed the new set of base ufw rules in the documentation could be augmented with:

sudo ufw route allow out on lxdbr0

This adds the FORWARD rule needed for proxy nat devices.

2 Likes
17

Would you be able to submit a pull request?

18

Case closed on my part. I will be happy to contribute a PR adding this to the docs.

The FORWARD rule was a bit sneaky since pings were working. This is due to ICMP having a special forward allow rule in the default ufw setup.

Thank you for diagnosing!

1 Like