LXC container doesn't start

,

Hi guys.I’m trying to run a container using the command


sudo lxc-start mycontainer

but the container failes to start and the command give me this message error:


lxc-start: mycontainer: lxccontainer.c: wait_on_daemonized_start: 877 Received container state "ABORTING" instead of "RUNNING"

lxc-start: mycontainer: tools/lxc_start.c: main: 306 The container failed to start

lxc-start: mycontainer: tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode

lxc-start: mycontainer: tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

So, I run the container in foreground mode as specified by the above message and it gives me more information about the error:


lxc-start: mycontainer: network.c: netdev_configure_server_veth: 711 No such file or directory - Failed to attach "vethC1QWRX" to bridge "lxcbr0", bridge interface doesn't exist

lxc-start: mycontainer: network.c: lxc_create_network_priv: 3427 No such file or directory - Failed to create network device

lxc-start: mycontainer: start.c: lxc_spawn: 1843 Failed to create the network

lxc-start: mycontainer: start.c: __lxc_start: 2074 Failed to spawn container "mycontainer"

lxc-start: mycontainer: tools/lxc_start.c: main: 306 The container failed to start

lxc-start: mycontainer: tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

I tried to create the lxcbr0 (which is missing) by using thi commands:


sudo ip link add name lxcbr0 type bridge

sudo ip addr add 10.0.3.1/24 dev lxcbr0

sudo ip link set lxcbr0 up

and configure the /etc/lxc/lxc-usernet file as follow


# USERNAME TYPE BRIDGE COUNT

spena veth lxcbr0 2

Unfortunately nothing as changed. Do you have an idea ? Can anyone help me ?

Below I write my system configuration:

  • Virtual machine with ubuntu 22.04

  • Linux kernel 5.15.52-dfl

  • /etc/default/lxc-net file


# This file is auto-generated by lxc.postinst if it does not

# exist. Customizations will not be overridden.

# Leave USE_LXC_BRIDGE as "true" if you want to use lxcbr0 for your

# containers. Set to "false" if you'll use virbr0 or another existing

# bridge, or mavlan to your host's NIC.

USE_LXC_BRIDGE="true"

# If you change the LXC_BRIDGE to something other than lxcbr0, then

# you will also need to update your /etc/lxc/default.conf as well as the

# configuration (/var/lib/lxc/<container>/config) for any containers

# already created using the default config to reflect the new bridge

# name.

# If you have the dnsmasq daemon installed, you'll also have to update

# /etc/dnsmasq.d/lxc and restart the system wide dnsmasq daemon.

LXC_BRIDGE="lxcbr0"

LXC_ADDR="10.0.3.1"

LXC_NETMASK="255.255.255.0"

LXC_NETWORK="10.0.3.0/24"

LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"

LXC_DHCP_MAX="253"

# Uncomment the next line if you'd like to use a conf-file for the lxcbr0

# dnsmasq. For instance, you can use 'dhcp-host=mail1,10.0.3.100' to have

# container 'mail1' always get ip address 10.0.3.100.

#LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf

# Uncomment the next line if you want lxcbr0's dnsmasq to resolve the .lxc

# domain. You can then add "server=/lxc/10.0.3.1' (or your actual $LXC_ADDR)

# to your system dnsmasq configuration file (normally /etc/dnsmasq.conf,

# or /etc/NetworkManager/dnsmasq.d/lxc.conf on systems that use NetworkManager).

# Once these changes are made, restart the lxc-net and network-manager services.

# 'container1.lxc' will then resolve on your host.

#LXC_DOMAIN="lxc"

  • /etc/default/lxc file

# LXC_AUTO - whether or not to start containers at boot

LXC_AUTO="true"

# BOOTGROUPS - What groups should start on bootup?

# Comma separated list of groups.

# Leading comma, trailing comma or embedded double

# comma indicates when the NULL group should be run.

# Example (default): boot the onboot group first then the NULL group

BOOTGROUPS="onboot,"

# SHUTDOWNDELAY - Wait time for a container to shut down.

# Container shutdown can result in lengthy system

# shutdown times. Even 5 seconds per container can be

# too long.

SHUTDOWNDELAY=5

# OPTIONS can be used for anything else.

# If you want to boot everything then

# options can be "-a" or "-a -A".

OPTIONS=

# STOPOPTS are stop options. The can be used for anything else to stop.

# If you want to kill containers fast, use -k

STOPOPTS="-a -A -s"

USE_LXC_BRIDGE="true" # overridden in lxc-net

[ ! -f /etc/default/lxc-net ] || . /etc/default/lxc-net

  • lxc-checkconfig

LXC version 5.0.0

Kernel configuration not found at /proc/config.gz; searching...

Kernel configuration found at /boot/config-5.15.52-dfl

--- Namespaces ---

Namespaces: enabled

Utsname namespace: enabled

Ipc namespace: enabled

Pid namespace: enabled

User namespace: enabled

Network namespace: enabled

--- Control groups ---

Cgroups: enabled

Cgroup namespace: enabled

Cgroup v1 mount points:

/sys/fs/cgroup/systemd

/sys/fs/cgroup/net_cls,net_prio

/sys/fs/cgroup/cpu,cpuacct

/sys/fs/cgroup/devices

/sys/fs/cgroup/rdma

/sys/fs/cgroup/pids

/sys/fs/cgroup/memory

/sys/fs/cgroup/cpuset

/sys/fs/cgroup/perf_event

/sys/fs/cgroup/freezer

/sys/fs/cgroup/blkio

/sys/fs/cgroup/misc

/sys/fs/cgroup/hugetlb

Cgroup v2 mount points:

/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled

Cgroup device: enabled

Cgroup sched: enabled

Cgroup cpu account: enabled

Cgroup memory controller: enabled

Cgroup cpuset: enabled

--- Misc ---

Veth pair device: enabled, loaded

Macvlan: enabled, not loaded

Vlan: enabled, not loaded

Bridges: enabled, loaded

Advanced netfilter: enabled, loaded

CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded

CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded

CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded

CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded

FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---

checkpoint restore: enabled

CONFIG_FHANDLE: enabled

CONFIG_EVENTFD: enabled

CONFIG_EPOLL: enabled

CONFIG_UNIX_DIAG: enabled

CONFIG_INET_DIAG: enabled

CONFIG_PACKET_DIAG: enabled

CONFIG_NETLINK_DIAG: enabled

File capabilities:

Note : Before booting a new kernel, you can check its configuration

usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

So ip link show lxcbr0 shows you the bridge existing but you’re still getting the exact same error telling you the bridge doesn’t exist?

At the beginnig yes, but a few minutes ago (after shutdown the virtual machine), I tried to create again the lxcbr0 and it worked. But I noticed that after rebooting the lxbr0 is missing, how can I make the lxbr0 permanent ?

Normally the USE_LXC_BRIDGE in /etc/default/lxc-net does that, having it set to true should cause the lxc-net systemd unit to run the network on boot.

You may want to look at systemctl status lxc-net to see what’s going on.

The is the output of systemctl status lxc-net

× lxc-net.service - LXC network bridge setup
     Loaded: loaded (/lib/systemd/system/lxc-net.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-08-08 13:14:48 CEST; 4h 20min ago
       Docs: man:lxc
    Process: 891 ExecStart=/usr/lib/x86_64-linux-gnu/lxc/lxc-net start (code=exited, status=1/FAILURE)
   Main PID: 891 (code=exited, status=1/FAILURE)
ago 08 13:14:46 spena-VirtualBox systemd[1]: Starting LXC network bridge setup...
ago 08 13:14:48 spena-VirtualBox lxc-net[1070]: Error: Could not process rule: No such file or directory
ago 08 13:14:48 spena-VirtualBox lxc-net[1070]: ;
ago 08 13:14:48 spena-VirtualBox lxc-net[1070]:                                                                          ^^^^^^^
ago 08 13:14:48 spena-VirtualBox lxc-net[891]: Failed to setup lxc-net.
ago 08 13:14:48 spena-VirtualBox systemd[1]: lxc-net.service: Main process exited, code=exited, status=1/FAILURE
ago 08 13:14:48 spena-VirtualBox systemd[1]: lxc-net.service: Failed with result 'exit-code'.
ago 08 13:14:48 spena-VirtualBox systemd[1]: Failed to start LXC network bridge setup.

Can you try sh -x /usr/lib/x86_64-linux-gnu/lxc/lxc-net start that may provide some useful context

Sorry for the late reply. This is the output

+ distrosysconfdir=/etc/default
+ varrun=/run/lxc
+ varlib=/var/lib
+ USE_LXC_BRIDGE=true
+ LXC_BRIDGE=lxcbr0
+ LXC_BRIDGE_MAC=00:16:3e:00:00:00
+ LXC_ADDR=10.0.3.1
+ LXC_NETMASK=255.255.255.0
+ LXC_NETWORK=10.0.3.0/24
+ LXC_DHCP_RANGE=10.0.3.2,10.0.3.254
+ LXC_DHCP_MAX=253
+ LXC_DHCP_CONFILE=
+ LXC_DHCP_PING=true
+ LXC_DOMAIN=
+ LXC_USE_NFT=true
+ LXC_IPV6_ADDR=
+ LXC_IPV6_MASK=
+ LXC_IPV6_NETWORK=
+ LXC_IPV6_NAT=false
+ [ ! -f /etc/default/lxc ]
+ . /etc/default/lxc
+ LXC_AUTO=true
+ BOOTGROUPS=onboot,
+ SHUTDOWNDELAY=5
+ OPTIONS=
+ STOPOPTS=-a -A -s
+ USE_LXC_BRIDGE=true
+ [ ! -f /etc/default/lxc-net ]
+ . /etc/default/lxc-net
+ USE_LXC_BRIDGE=true
+ LXC_BRIDGE=lxcbr0
+ LXC_ADDR=10.0.3.1
+ LXC_NETMASK=255.255.255.0
+ LXC_NETWORK=10.0.3.0/24
+ LXC_DHCP_RANGE=10.0.3.2,10.0.3.254
+ LXC_DHCP_MAX=253
+ command -v nft
+ NFT=/usr/sbin/nft
+ use_nft
+ [ -n /usr/sbin/nft ]
+ nft list ruleset
+ use_iptables_lock=-w
+ iptables -w -L -n
+ use_iptables_lock=
+ start
+ [ ! -f /etc/default/lxc-net ]
+ [ xtrue = xtrue ]
+ [ ! -f /run/lxc/network_up ]
+ [ -d /sys/class/net/lxcbr0 ]
+ FAILED=1
+ trap cleanup EXIT HUP INT TERM
+ set -e
+ [ ! -d /sys/class/net/lxcbr0 ]
+ ip link add dev lxcbr0 type bridge
RTNETLINK answers: Operation not permitted
+ cleanup
+ set +e
+ [ 1 = 1 ]
+ echo Failed to setup lxc-net.
Failed to setup lxc-net.
+ stop force
+ [ xtrue = xtrue ]
+ [ -f /run/lxc/network_up ]
+ [ force = force ]
+ [ -d /sys/class/net/lxcbr0 ]
+ rm -f /run/lxc/network_up
+ exit 1

Sorry, you need to run this as root.

Don’t worry, this is the output

+ distrosysconfdir=/etc/default
+ varrun=/run/lxc
+ varlib=/var/lib
+ USE_LXC_BRIDGE=true
+ LXC_BRIDGE=lxcbr0
+ LXC_BRIDGE_MAC=00:16:3e:00:00:00
+ LXC_ADDR=10.0.3.1
+ LXC_NETMASK=255.255.255.0
+ LXC_NETWORK=10.0.3.0/24
+ LXC_DHCP_RANGE=10.0.3.2,10.0.3.254
+ LXC_DHCP_MAX=253
+ LXC_DHCP_CONFILE=
+ LXC_DHCP_PING=true
+ LXC_DOMAIN=
+ LXC_USE_NFT=true
+ LXC_IPV6_ADDR=
+ LXC_IPV6_MASK=
+ LXC_IPV6_NETWORK=
+ LXC_IPV6_NAT=false
+ [ ! -f /etc/default/lxc ]
+ . /etc/default/lxc
+ LXC_AUTO=true
+ BOOTGROUPS=onboot,
+ SHUTDOWNDELAY=5
+ OPTIONS=
+ STOPOPTS=-a -A -s
+ USE_LXC_BRIDGE=true
+ [ ! -f /etc/default/lxc-net ]
+ . /etc/default/lxc-net
+ USE_LXC_BRIDGE=true
+ LXC_BRIDGE=lxcbr0
+ LXC_ADDR=10.0.3.1
+ LXC_NETMASK=255.255.255.0
+ LXC_NETWORK=10.0.3.0/24
+ LXC_DHCP_RANGE=10.0.3.2,10.0.3.254
+ LXC_DHCP_MAX=253
+ command -v nft
+ NFT=/usr/sbin/nft
+ use_nft
+ [ -n /usr/sbin/nft ]
+ nft list ruleset
+ [ true = true ]
+ start
+ [ ! -f /etc/default/lxc-net ]
+ [ xtrue = xtrue ]
+ [ ! -f /run/lxc/network_up ]
+ [ -d /sys/class/net/lxcbr0 ]
+ stop force
+ [ xtrue = xtrue ]
+ [ -f /run/lxc/network_up ]
+ [ force = force ]
+ [ -d /sys/class/net/lxcbr0 ]
+ _ifdown
+ ip addr flush dev lxcbr0
+ ip link set dev lxcbr0 down
+ use_nft
+ [ -n /usr/sbin/nft ]
+ nft list ruleset
+ [ true = true ]
+ stop_nftables
+ NFT_RULESET=add table inet lxc;
delete table inet lxc;
add table ip lxc;
delete table ip lxc;
+ [ false = true ]
+ nft add table inet lxc;
delete table inet lxc;
add table ip lxc;
delete table ip lxc;
+ cat /run/lxc/dnsmasq.pid
+ pid=
+ rm -f /run/lxc/dnsmasq.pid
+ ls /sys/class/net/lxcbr0/brif/*
+ ip link delete lxcbr0
+ rm -f /run/lxc/network_up
+ FAILED=1
+ trap cleanup EXIT HUP INT TERM
+ set -e
+ [ ! -d /sys/class/net/lxcbr0 ]
+ ip link add dev lxcbr0 type bridge
+ echo 1
+ echo 0
+ [ ! -d /run/lxc ]
+ _ifup
+ _netmask2cidr 255.255.255.0
+ local x=0
+ set -- 0^^^128^192^224^240^248^252^254^ 24 0
+ x=
+ echo 24
+ MASK=24
+ CIDR_ADDR=10.0.3.1/24
+ ip addr add 10.0.3.1/24 broadcast + dev lxcbr0
+ ip link set dev lxcbr0 address 00:16:3e:00:00:00
+ ip link set dev lxcbr0 up
+ use_nft
+ [ -n /usr/sbin/nft ]
+ nft list ruleset
+ [ true = true ]
+ start_nftables
+ start_ipv6
+ LXC_IPV6_ARG=
+ [ -n  ]
+ NFT_RULESET=
+ [ -n  ]
+ NFT_RULESET=;
add table inet lxc;
flush table inet lxc;
add chain inet lxc input { type filter hook input priority 0; };
add rule inet lxc input iifname lxcbr0 udp dport { 53, 67 } accept;
add rule inet lxc input iifname lxcbr0 tcp dport { 53, 67 } accept;
add chain inet lxc forward { type filter hook forward priority 0; };
add rule inet lxc forward iifname lxcbr0 accept;
add rule inet lxc forward oifname lxcbr0 accept;
add table ip lxc;
flush table ip lxc;
add chain ip lxc postrouting { type nat hook postrouting priority 100; };
add rule ip lxc postrouting ip saddr 10.0.3.0/24 ip daddr != 10.0.3.0/24 counter masquerade
+ nft ;
add table inet lxc;
flush table inet lxc;
add chain inet lxc input { type filter hook input priority 0; };
add rule inet lxc input iifname lxcbr0 udp dport { 53, 67 } accept;
add rule inet lxc input iifname lxcbr0 tcp dport { 53, 67 } accept;
add chain inet lxc forward { type filter hook forward priority 0; };
add rule inet lxc forward iifname lxcbr0 accept;
add rule inet lxc forward oifname lxcbr0 accept;
add table ip lxc;
flush table ip lxc;
add chain ip lxc postrouting { type nat hook postrouting priority 100; };
add rule ip lxc postrouting ip saddr 10.0.3.0/24 ip daddr != 10.0.3.0/24 counter masquerade
Error: Could not process rule: No such file or directory
;

                                                                         ^^^^^^^
+ cleanup
+ set +e
+ [ 1 = 1 ]
+ echo Failed to setup lxc-net.
Failed to setup lxc-net.
+ stop force
+ [ xtrue = xtrue ]
+ [ -f /run/lxc/network_up ]
+ [ force = force ]
+ [ -d /sys/class/net/lxcbr0 ]
+ _ifdown
+ ip addr flush dev lxcbr0
+ ip link set dev lxcbr0 down
+ use_nft
+ [ -n /usr/sbin/nft ]
+ nft list ruleset
+ [ true = true ]
+ stop_nftables
+ NFT_RULESET=add table inet lxc;
delete table inet lxc;
add table ip lxc;
delete table ip lxc;
+ [ false = true ]
+ nft add table inet lxc;
delete table inet lxc;
add table ip lxc;
delete table ip lxc;
+ cat /run/lxc/dnsmasq.pid
+ pid=
+ rm -f /run/lxc/dnsmasq.pid
+ ls /sys/class/net/lxcbr0/brif/*
+ ip link delete lxcbr0
+ rm -f /run/lxc/network_up
+ exit 1

Can you test with an official kernel?

The error above suggests that your current kernel may be lacking some netfilter features.

Unfortunately, no. I need to use my current kernel for other purposes. What kind of netfilter features are missing ?

Try running lxc-checkconfig as an initial check to see what may be missing.

Because all the rules are pushed together in one shot by nft, it’s kind hard to tell what bit exactly is missing in your kernel.

I’ve already run it and from the output (which I posted above) nothing seems to be missing.

root@v1:~# systemctl status lxc-net
● lxc-net.service - LXC network bridge setup
     Loaded: loaded (/lib/systemd/system/lxc-net.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2024-08-12 15:46:25 UTC; 4s ago
       Docs: man:lxc
    Process: 322 ExecStart=/usr/lib/x86_64-linux-gnu/lxc/lxc-net start (code=exited, status=0/SUCCESS)
   Main PID: 322 (code=exited, status=0/SUCCESS)
      Tasks: 1 (limit: 1103)
     Memory: 4.0M
        CPU: 22ms
     CGroup: /system.slice/lxc-net.service
             └─420 dnsmasq --conf-file=/dev/null -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address 10.0.3.1 --d>

Aug 12 15:46:25 v1 systemd[1]: Starting LXC network bridge setup...
Aug 12 15:46:25 v1 dnsmasq[420]: started, version 2.90 cachesize 150
Aug 12 15:46:25 v1 dnsmasq[420]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset no-nftset auth cryptohas>
Aug 12 15:46:25 v1 dnsmasq-dhcp[420]: DHCP, IP range 10.0.3.2 -- 10.0.3.254, lease time 1h
Aug 12 15:46:25 v1 dnsmasq-dhcp[420]: DHCP, sockets bound exclusively to interface lxcbr0
Aug 12 15:46:25 v1 dnsmasq[420]: reading /etc/resolv.conf
Aug 12 15:46:25 v1 dnsmasq[420]: using nameserver 127.0.0.53#53
Aug 12 15:46:25 v1 dnsmasq[420]: read /etc/hosts - 7 names
Aug 12 15:46:25 v1 systemd[1]: Finished LXC network bridge setup.
root@v1:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy
root@v1:~# uname -a
Linux v1 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@v1:~# nft list ruleset
table inet lxc {
	chain input {
		type filter hook input priority filter; policy accept;
		iifname "lxcbr0" udp dport { 53, 67 } accept
		iifname "lxcbr0" tcp dport { 53, 67 } accept
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		iifname "lxcbr0" accept
		oifname "lxcbr0" accept
	}
}
table ip lxc {
	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 10.0.3.0/24 ip daddr != 10.0.3.0/24 counter packets 1 bytes 40 masquerade
	}
}
root@v1:~# 

So everything works properly on the official Ubuntu kernel. You’re going to have to track this down with the vendor of your custom kernel.

Thank you for the help