[Solved] Can not start more than 10 - 14 Containers

LXC
1

Hello,

I have a strange problem… somtimes I can just start 10 containers other times it is one or two more. The maximum I reached today, is 14.

I start one container after the other. There is no error message. But when I run “lxc-ls --fancy” I see that the last containers have no IP address.

NAME           STATE   AUTOSTART GROUPS IPV4            IPV6 UNPRIVILEGED      
container      RUNNING 0         -      -               -    false

At first I thought it was a network problem. But it seems like the container starts only partially.
I can enter the container with “lxc-mirror” but get the following errors when executing commands.

root@container:~# dhclient
Failed to connect to bus: No such file or directory

root@container:~# shutdown -r now
Failed to connect to bus: Datei oder Verzeichnis nicht gefunden
Failed to talk to init daemon.

Wenn stopping the container with “lxc-stop” it takes about a minute to stop it.

The container’s syslog files last enties are from the last successfull shutdown.
The host’s syslog shows the following for every container, not just the broken one…

Dec 20 19:32:58 homesrv systemd-udevd[9710]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 20 19:32:58 homesrv systemd-udevd[9710]: Could not generate persistent MAC address for vethX0NR3L: No such file or directory
Dec 20 19:32:58 homesrv systemd-udevd[9711]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-subsystem-net-devices-vethX0NR3L.device.wants: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-subsystem-net-devices-vethX0NR3L.device.wants: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-subsystem-net-devices-vethX0NR3L.device.requires: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-subsystem-net-devices-vethX0NR3L.device.requires: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-subsystem-net-devices-vethX0NR3L.device.d: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-subsystem-net-devices-vethX0NR3L.device.d: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-devices-virtual-net-vethX0NR3L.device.wants: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-devices-virtual-net-vethX0NR3L.device.wants: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-devices-virtual-net-vethX0NR3L.device.requires: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-devices-virtual-net-vethX0NR3L.device.requires: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-devices-virtual-net-vethX0NR3L.device.d: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-devices-virtual-net-vethX0NR3L.device.d: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-subsystem-net-devices-veth3YE0FR.device.wants: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-subsystem-net-devices-veth3YE0FR.device.wants: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-subsystem-net-devices-veth3YE0FR.device.requires: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-subsystem-net-devices-veth3YE0FR.device.requires: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-subsystem-net-devices-veth3YE0FR.device.d: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-subsystem-net-devices-veth3YE0FR.device.d: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-devices-virtual-net-veth3YE0FR.device.wants: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-devices-virtual-net-veth3YE0FR.device.wants: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-devices-virtual-net-veth3YE0FR.device.requires: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-devices-virtual-net-veth3YE0FR.device.requires: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.config/systemd/user/sys-devices-virtual-net-veth3YE0FR.device.d: Permission denied
Dec 20 19:32:58 homesrv systemd[7891]: Failed to canonicalize path /home/therebel/.local/share/systemd/user/sys-devices-virtual-net-veth3YE0FR.device.d: Permission denied
Dec 20 19:33:00 homesrv avahi-daemon[1100]: Joining mDNS multicast group on interface veth3YE0FR.IPv6 with address fe80::fc3e:79ff:fee6:697a.
Dec 20 19:33:00 homesrv avahi-daemon[1100]: New relevant interface veth3YE0FR.IPv6 for mDNS.
Dec 20 19:33:00 homesrv avahi-daemon[1100]: Registering new address record for fe80::fc3e:79ff:fee6:697a on veth3YE0FR.*.

Host System: Ubuntu 18.04 Server
LXC Version: 3.0.3-0ubuntu1~18.04.1
Container: Ubuntu bionic
Filesystem: btrfs

Container Config

# Common configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf

# Container specific configuration
lxc.rootfs.path = /var/lib/lxc/container/rootfs
lxc.mount.fstab = /var/lib/lxc/container/fstab
lxc.uts.name = container
lxc.arch = amd64

# Network configuration
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.hwaddr = 00:16:3e:22:07:46
2

Looks like a permissions issue, did you execute the lxc commands as root and now those files are owned by root?

Seems like the files should be owned by the user or lxd as they are in a home directory.

/home/therebel/.
3

You are right. Some folders in the homedirectory were owned by root. I changed that to “therebel”.
That does fix the syslog messages, but the problem still exists.
A container start now looks like that in syslog:

Dec 21 08:10:27 homesrv systemd-udevd[13603]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 21 08:10:27 homesrv systemd-udevd[13603]: Could not generate persistent MAC address for veth0NO68C: No such file or directory
Dec 21 08:10:27 homesrv systemd-udevd[13604]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 21 08:10:27 homesrv kernel: [63101.114577] br0: port 3(vethGL2YSN) entered blocking state
Dec 21 08:10:27 homesrv kernel: [63101.114579] br0: port 3(vethGL2YSN) entered disabled state
Dec 21 08:10:27 homesrv kernel: [63101.114634] device vethGL2YSN entered promiscuous mode
Dec 21 08:10:27 homesrv kernel: [63101.114765] IPv6: ADDRCONF(NETDEV_UP): vethGL2YSN: link is not ready
Dec 21 08:10:29 homesrv avahi-daemon[1100]: Joining mDNS multicast group on interface vethGL2YSN.IPv6 with address fe80::fca7:5cff:fec7:581b.
Dec 21 08:10:29 homesrv avahi-daemon[1100]: New relevant interface vethGL2YSN.IPv6 for mDNS.
Dec 21 08:10:29 homesrv avahi-daemon[1100]: Registering new address record for fe80::fca7:5cff:fec7:581b on vethGL2YSN.*.
4

Not sure if this is applicable to LXC but you might want to try to change some kernel settings to up some limits https://github.com/lxc/lxd/blob/master/doc/production-setup.md

1 Like
5

Thank you very much… that soved the problem

/etc/security/limits.conf

*               soft    nofile          1048576
*               hard    nofile          1048576
root            soft    nofile          1048576
root            hard    nofile          1048576
*               soft    memlock         unlimited
*               hard    memlock         unlimited

/etc/sysctl.conf

fs.inotify.max_queued_events=1048576
fs.inotify.max_user_instances=104576
fs.inotify.max_user_watches=104576
vm.max_map_count=262144
kernel.dmesg_restrict=1
net.ipv4.neigh.default.gc_thresh3=8192
net.ipv6.neigh.default.gc_thresh3=8192
kernel.keys.maxkeys=2000

REBOOT

Now 30 containers running smoothly

6

I wonder which limit did you hit with so few containers.

Do you run something specific in those containers that may use up the kernel resources.
Do you think it could be network related, filesystem related (inotify) or other kernel related?

7


:slight_smile:

8

It’s just a homeserver with minimal load. This are the containers/applications running…

collabora
dovecot
dyndns
elasticsearch
grafana
homegear
influxdb
mosquitto
mysql
nextcloud
owncloud
pihole
postfix
rspamd
openhab
talk
tvheadend

When reading “filesystem related (inotify)” I remember a message I got sometimes when opening logfiles.
It was like: Can not use inotify. To much open files.

9

Ah yeah, so you’d have run out of inotify handles, I suspect some of those applications consume a lot of them somehow, on top of systemd already liking to consume a whole bunch.

Normally you could run about 50 containers before hitting the default inotify limit and having to bump it to some higher value.

The values in our production setup documentation should be plenty for hundreds of containers.

10

I have this problem when trying to launch more than 10 LXC unprivileged containers on Debian 10 (buster):

network - network.c:lxc_create_network_unpriv_exec:2178 - lxc-user-nic failed to configure requested network: cmd/lxc_user_nic.c: 1296: main: Quota reached

I’ve been looking into lxd/doc/production-setup.md but didn’t found what limit can I tweak to pass from 10 to 20 veth NICs for example.

11

That quota comes from the /etc/lxc/lxc-usernet configuration

12

Thank you. I was affected by same as reported in issue #245
I think file lxd/doc/production-setup.md should include a mention to /etc/lxc/lxc-usernet limits.