LXD 3.21 firewalld issue

seven · February 26, 2020, 4:24pm

Upon fresh install, creating fresh container wth image opensuse/15.1/amd64. then installing firewalld on the container, starting it some times fail and always has this issue.

● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-02-26 16:23:31 UTC; 11s ago
     Docs: man:firewalld(1)
 Main PID: 67 (firewalld)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/firewalld.service
           └─67 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid

Feb 26 16:23:30 ct1 systemd[1]: Starting firewalld - dynamic firewall daemon...
Feb 26 16:23:31 ct1 systemd[1]: Started firewalld - dynamic firewall daemon.
Feb 26 16:23:31 ct1 firewalld[67]: WARNING: modinfo command is missing, not able to detect conntrack helpers.
Feb 26 16:23:31 ct1 firewalld[67]: ERROR: Failed to read file "/proc/sys/net/netfilter/nf_conntrack_helper": [Errno 2] No such file or directory: '/proc/sys/net/netfilter/nf_conntrack_helper'
Feb 26 16:23:31 ct1 firewalld[67]: WARNING: Failed to get and parse nf_conntrack_helper setting```
any suggestion, thanks.

tomp · February 26, 2020, 4:49pm

You probably need to ensure the correct kernel modules for conntrack are loaded before the container starts, as they cannot be loaded by firewalld inside the container.

You can use the instance setting linux.kernel_modules to specify that LXD should load these modules before container starts.

See https://linuxcontainers.org/lxd/docs/master/instances#properties

linux.kernel_modules	string	-	yes	container	Comma separated list of kernel modules to load before starting the instance

seven · February 26, 2020, 5:06pm

Hi, sorry am getting confused, in my case i use a profile for my containers, what modification and where it has to be done?

tomp · February 26, 2020, 5:54pm

So you should be able to do:

lxc profile set <profile name> linux.kernel_modules nf_conntrack

Where nf_conntrack is the name of the kernel module you need.

seven · February 26, 2020, 6:36pm

i have just done that, unfortunately the problem remains the same.
UPDATE:
ls /etc/modules-load.d/
is empty.
UPDATE:2:
cat /proc/modules | grep 'nf_conntrack'
nf_conntrack 155648 8 ipt_MASQUERADE
but sadly the issue persist.

tomp · February 27, 2020, 9:43am

Can you show output of: lxc show config <container> --expanded to confirm the kernel modules key has been applied to the container.

and lsmod on the host.

I am going off of this guide here https://github.com/firewalld/firewalld/issues/353 which suggests that nf_conntrack module is what is needed.

seven · February 27, 2020, 12:00pm

lsmod

nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  1 iptable_nat
nf_nat                 32768  3 nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4
nf_conntrack          155648  8 nf_conntrack_ipv6,nf_conntrack_ipv4,ipt_MASQUERADE,nf_nat_ipv6,nf_nat_masquerade_ipv4,xt_conn```
lxc config show ct1 --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Opensuse 15.1 amd64 (20200227_04:20)
  image.os: Opensuse
  image.release: "15.1"
  image.serial: "20200227_04:20"
  image.type: squashfs
  limits.cpu: "1"
  limits.memory: 2048MB
  volatile.base_image: 93a7ad2932bb806441d34a3f29b799e129228ad617c361970572e953794f0567
  volatile.eth0.host_name: veth776d7bfc
  volatile.eth0.hwaddr: 00:16:3e:9c:10:1d
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":400000000,"Nsid":0,"Maprange":500000001},{"Isuid":false,"Isgid":true,"Hostid":400000000,"Nsid":0,"Maprange":500000001}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":400000000,"Nsid":0,"Maprange":500000001},{"Isuid":false,"Isgid":true,"Hostid":400000000,"Nsid":0,"Maprange":500000001}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":400000000,"Nsid":0,"Maprange":500000001},{"Isuid":false,"Isgid":true,"Hostid":400000000,"Nsid":0,"Maprange":500000001}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- local
stateful: false
description: ""

tomp · February 27, 2020, 1:15pm

I cant see the linux.kernel_modules setting in your container’s config, suggesting its not been added to the same profile as the container.

Try this instead:

lxc config set <container> linux.kernel_modules = nf_conntrack

Then restart container.

seven · February 27, 2020, 1:27pm

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Opensuse 15.1 amd64 (20200227_04:20)
  image.os: Opensuse
  image.release: "15.1"
  image.serial: "20200227_04:20"
  image.type: squashfs
  limits.cpu: "1"
  limits.memory: 2048MB
  linux.kernel_modules: nf_conntrack
  volatile.base_image: 93a7ad2932bb806441d34a3f29b799e129228ad617c361970572e953794f0567
  volatile.eth0.host_name: veth776d7bfc
  volatile.eth0.hwaddr: 00:16:3e:9c:10:1d
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":400000000,"Nsid":0,"Maprange":500000001},{"Isuid":false,"Isgid":true,"Hostid":400000000,"Nsid":0,"Maprange":500000001}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":400000000,"Nsid":0,"Maprange":500000001},{"Isuid":false,"Isgid":true,"Hostid":400000000,"Nsid":0,"Maprange":500000001}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":400000000,"Nsid":0,"Maprange":500000001},{"Isuid":false,"Isgid":true,"Hostid":400000000,"Nsid":0,"Maprange":500000001}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- local
stateful: false
description: ""```

tomp · February 27, 2020, 3:11pm

Thats better

seven · February 27, 2020, 3:18pm

well, problem remains the same, is this a possible bug with the image itself or lxd somehow?

tomp · February 27, 2020, 3:34pm

@seven ok, so I’ve confirmed that /proc/sys/net/netfilter/nf_conntrack_helper isn’t visible inside the container. Although you’re now reliably loading the nf_conntrack module so that’s good.

Doing a google search for that error shows that it maybe something to do with firewalld trying to decide whether to allow netfilter helpers or not. This is why it is trying to access that file. But as the file is not being passed into the container it fails to start.

You may be able to work around this issue by setting: AutomaticHelpers=no

stgraber · February 27, 2020, 8:27pm

It’s also quite likely that this particular kernel feature just isn’t properly namespaced and so it was a deliberate decision not to expose it inside child network namespaces.

kerphi · April 15, 2020, 11:11am

@seven what is your problem?
Is it the firewalld start log error ?

Feb 26 16:23:31 ct1 firewalld[67]: ERROR: Failed to read file "/proc/sys/net/netfilter/nf_conntrack_helper": [Errno 2] No such file or directory: '/proc/sys/net/netfilter/nf_conntrack_helper'

Or the fact LXD containers are not able to access internet after the firewalld startup?

I don’t know if the firewalld log error is related to the not working LXD containers but I observed this:

I installed basic LXD 3.0.3
I created a basic container named vm1 lxc launch images:debian/buster vm1
I executed apt update from vm1 lxc exec vm1 apt update and it was ok
I installed firewalld on the host: apt install firewalld (without any configuration)
I repeated step 3 and the apt update in the vm1 was not ok anymore
I stopped firewalld: systemctl stop firewalld
I repeated step 3 and the apt update in the vm1 was still not ok
I restarted lxd daemon: systemctl stop lxd
I repeated step 3 and the apt update in the vm1 was then OK
I then restarted firewalld: systemctl start firewalld
I repeated step 3 and the apt update in the vm1 was again not ok
I then restarted lxd daemon: systemctl restart lxd (notice firewalld is still running)
I then repeated step 3 and the apt update in the vm1 was finally OK !

Moral of the story, i suspect firewalld is overwriting iptables rules of LXD. I don’t know if i’m really right but if I am, i don’t know how to do to have a persistent good starting sequence: LXD should be started after firewalld and never before …
Any ideas?

seven · April 16, 2020, 10:56pm

@kerphi The issue is both actually.
whenever i restart the firewalld on the host, i have to restart containers 1 by 1.
and the systemctl log message too, why do we have to see it in the first place.