Centos/7 containers don't get an IPv4 address

x-yuri · June 8, 2021, 4:35am

Hi,

I tried the following images: images:centos/7, images:centos/8, ubuntu:20.04. The last two seems to be doing fine. The first one doesn’t get an IPv4 address, and doesn’t stop unless I force (-f) it.

When starting an centos/7 container in /var/log/lxd/c1/console.log I see:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

The host OS is Arch Linux, lxd-4.15.

tomp · June 8, 2021, 8:14am

Hi,

I just tried this on Ubuntu 20.04 host:

lxc launch images:centos/7 c1
Creating c1
Starting c1                                 

lxc ls
+------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |       IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| c1   | RUNNING | 10.98.30.5 (eth0) | fd42:f402:8623:5b6b:216:3eff:fe82:6e3f (eth0) | CONTAINER | 0         |
+------+---------+-------------------+-----------------------------------------------+-----------+-----------+


lxc stop c1
lxc ls
+------+---------+------+------+-----------+-----------+
| NAME |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+------+---------+------+------+-----------+-----------+
| c1   | STOPPED |      |      | CONTAINER | 0         |
+------+---------+------+------+-----------+-----------+

So it seems to be working OK generally, at least on an Ubuntu host.

Can you try clearing the centos 7 image cache you have downloaded (lxc image ls then lxc image dete <hash>) and then launching a fresh one to see if its fixed.

Also can you show the one hash of the one you’re using if its still happening.

CC @monstermunchkin

x-yuri · June 8, 2021, 12:32pm

Actually I reinstalled lxd today. It’s unlikely it has something to do with the image:

$ lxc image list
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE          |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
|       | 0be7dc7195b8 | no     | Centos 7 amd64 (20210608_07:08)             | x86_64       | CONTAINER | 83.44MB  | Jun 8, 2021 at 12:25pm (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+

$ lxc image delete 0be7dc7195b8
$ lxc stop -f c1
$ lxc delete c1
$ sudo rm -r /var/log/lxd/c1

$ lxc launch images:centos/7 c1
Creating c1
Starting c1

$ lxc list
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |        IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c1   | RUNNING |                    | fd42:6c7f:db14:1561:216:3eff:fec4:7b10 (eth0) | CONTAINER | 0         |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+

$ sudo cat /var/log/lxd/c1/console.log
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

$ lxc image list
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE          |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
|       | 0be7dc7195b8 | no     | Centos 7 amd64 (20210608_07:08)             | x86_64       | CONTAINER | 83.44MB  | Jun 8, 2021 at 12:25pm (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+

tomp · June 8, 2021, 12:36pm

Can you run dhclient manually inside the container and does that allocate an address?

Basically we need to work out if its an issue with networking (i.e something preventing DHCP requests reaching the host’s DHCP server) or something inside the container that is preventing network config from running?

tomp · June 8, 2021, 12:37pm

Also, if you setup IP addressing manually inside the container, do things work then?

tomp · June 8, 2021, 12:37pm

Does systemctl show any failed services?

x-yuri · June 8, 2021, 1:00pm

$ lxc list
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |        IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c1   | RUNNING |                    | fd42:6c7f:db14:1561:216:3eff:fec4:7b10 (eth0) | CONTAINER | 0         |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+

$ lxc exec c1 dhclient

$ lxc list
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| c1   | RUNNING | 10.110.99.112 (eth0) | fd42:6c7f:db14:1561:216:3eff:fec4:7b10 (eth0) | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+

$ lxc exec c1 systemctl
Failed to get D-Bus connection: No such file or directory

And the container still won’t stop w/o -f.

tomp · June 8, 2021, 1:07pm

Seems like an issue to do with cgroups (likely because the Arch host runs more modern versions of packages than many other distros).

@brauner @stgraber have you seen anything like this before?

Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

brauner · June 8, 2021, 1:11pm

My bet is that Archlinux makes use of cgroup v2 only and systemd on CentOS 7 is tool old too know how to deal with such a system and expects the named systemd cgroup v1 controller which isn’t available.

brauner · June 8, 2021, 1:14pm

Can you show the output of findmnt on your host, please?

x-yuri · June 8, 2021, 3:14pm

Well, from what I can see systemd defaults to v2 only since v243. But Arch Linux was overriding the default till recently (v248).

$ findmnt
TARGET                                                                                             SOURCE                 FSTYPE          OPTIONS
/                                                                                                  /dev/nvme0n1p5         ext4            rw,relatime
├─/proc                                                                                            proc                   proc            rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc                                                                       systemd-1              autofs          rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13652
│   └─/proc/sys/fs/binfmt_misc                                                                     binfmt_misc            binfmt_misc     rw,nosuid,nodev,noexec,relatime
├─/sys                                                                                             sys                    sysfs           rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars                                                                      efivarfs               efivarfs        rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security                                                                           securityfs             securityfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup                                                                                 cgroup2                cgroup2         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/pstore                                                                                 pstore                 pstore          rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf                                                                                    none                   bpf             rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug                                                                              debugfs                debugfs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing                                                                            tracefs                tracefs         rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/config                                                                             configfs               configfs        rw,nosuid,nodev,noexec,relatime
│ └─/sys/fs/fuse/connections                                                                       fusectl                fusectl         rw,nosuid,nodev,noexec,relatime
├─/dev                                                                                             dev                    devtmpfs        rw,nosuid,relatime,size=7869688k,nr_inodes=1967422,mode=755,inode64
│ ├─/dev/shm                                                                                       tmpfs                  tmpfs           rw,nosuid,nodev,inode64
│ ├─/dev/pts                                                                                       devpts                 devpts          rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/hugepages                                                                                 hugetlbfs              hugetlbfs       rw,relatime,pagesize=2M
│ └─/dev/mqueue                                                                                    mqueue                 mqueue          rw,nosuid,nodev,noexec,relatime
├─/run                                                                                             run                    tmpfs           rw,nosuid,nodev,relatime,mode=755,inode64
│ ├─/run/user/1000                                                                                 tmpfs                  tmpfs           rw,nosuid,nodev,relatime,size=1582052k,nr_inodes=395513,mode=700,uid=1000,gid=1000,inode64
│ │ └─/run/user/1000/gvfs                                                                          gvfsd-fuse             fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000
│ ├─/run/docker/netns/1ed82675d267                                                                 nsfs[net:[4026533974]] nsfs            rw
│ ├─/run/docker/netns/c91da7a9b800                                                                 nsfs[net:[4026533911]] nsfs            rw
│ └─/run/docker/netns/5525f938a6a9                                                                 nsfs[net:[4026534041]] nsfs            rw
├─/tmp                                                                                             tmpfs                  tmpfs           rw,nosuid,nodev,nr_inodes=409600,inode64
├─/boot                                                                                            /dev/nvme0n1p1         vfat            rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro
├─/var/lib/lxcfs                                                                                   lxcfs                  fuse.lxcfs      rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
├─/var/lib/lxd/shmounts                                                                            tmpfs                  tmpfs           rw,relatime,size=100k,mode=711,inode64
├─/var/lib/lxd/devlxd                                                                              tmpfs                  tmpfs           rw,relatime,size=100k,mode=755,inode64
├─/var/lib/docker/overlay2/9e5e6df44bd4f93874dec778299b30ed4f0ad0a3dabfa13702472dc76273137d/merged overlay                overlay         rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QQ2SXOS6GBCHJVPP6ZZOOKAHNT:/var/lib/docker/overlay2/l/T5RDZDILUQAIBJMR22BC7WODWV:/var/lib/docker/overlay2/l/PEES24FSR44WHLT4OI7FASVSQ5:/var/lib/docker/overlay2/l/NQEM573733UF7WI33O37JPTVU5:/var/lib/docker/overlay2/l/BOSQJSIXJG3M3COQNJMKN4XWEF:/var/lib/docker/overlay2/l/UAMEIUGQW2ESOVUNEVQZ4AMXBP:/var/lib/docker/overlay2/l/VYC2RUVJDM2ZBP4CJR7YPEAALD:/var/lib/docker/overlay2/l/GLJBYEF766TSNHGISIYUXUADMX:/var/lib/docker/overlay2/l/ANW2NWIM5GK4AX2TNPW7CACHQ7,upperdir=/var/lib/docker/overlay2/9e5e6df44bd4f93874dec778299b30ed4f0ad0a3dabfa13702472dc76273137d/diff,workdir=/var/lib/docker/overlay2/9e5e6df44bd4f93874dec778299b30ed4f0ad0a3dabfa13702472dc76273137d/work,index=off
├─/var/lib/docker/overlay2/03ed081ae6c312f1bc476702fb490927d4bb737eee1bb4c39f55f67c47e63519/merged overlay                overlay         rw,relatime,lowerdir=/var/lib/docker/overlay2/l/PJJKNRDZTB74HLWQDQX5O5CGOO:/var/lib/docker/overlay2/l/6IEB6FYYNNQJQN2QW3SE2YCHW6:/var/lib/docker/overlay2/l/T5EKLVCDIUEMZW3RXAD3BQ4Z6T:/var/lib/docker/overlay2/l/6SYKH6MHAQ32XRIVS35ZC477P5:/var/lib/docker/overlay2/l/G7J4SJHZNPFGD5YQQCMCLWFLDU:/var/lib/docker/overlay2/l/HXLQBXZEXXF7ZJFQV2GVYOZO4D:/var/lib/docker/overlay2/l/ANW2NWIM5GK4AX2TNPW7CACHQ7,upperdir=/var/lib/docker/overlay2/03ed081ae6c312f1bc476702fb490927d4bb737eee1bb4c39f55f67c47e63519/diff,workdir=/var/lib/docker/overlay2/03ed081ae6c312f1bc476702fb490927d4bb737eee1bb4c39f55f67c47e63519/work,index=off
└─/var/lib/docker/overlay2/7b843cde1f990218628be3801ed22f5c5791f0209140c002bf78feeedd826626/merged overlay                overlay         rw,relatime,lowerdir=/var/lib/docker/overlay2/l/7AP3XPMXPBSVQEO4CQL4JENLJW:/var/lib/docker/overlay2/l/IYRMWMKWIQTZNFE7UZNL552QF6:/var/lib/docker/overlay2/l/SUW7B557DKDD65NIWJATDW6OOO:/var/lib/docker/overlay2/l/WUPG2ZHVT5KXKJBFIWBFWYX5A3:/var/lib/docker/overlay2/l/WIIGTKJO5MGSZTGZX27GZOJLBM:/var/lib/docker/overlay2/l/WLTMVNTJL6KZWXQNMVQIFYK3DY:/var/lib/docker/overlay2/l/YE5A5INTBRESQZHIX22KBT4GZW:/var/lib/docker/overlay2/l/A4NPUJROEZBPZGYXPBRCBX36UM,upperdir=/var/lib/docker/overlay2/7b843cde1f990218628be3801ed22f5c5791f0209140c002bf78feeedd826626/diff,workdir=/var/lib/docker/overlay2/7b843cde1f990218628be3801ed22f5c5791f0209140c002bf78feeedd826626/work,index=off

x-yuri · June 9, 2021, 5:49am

Indeed after switching to hybrid mode (by adding kernel parameter systemd.unified_cgroup_hierarchy=0) it all works. The container obtains an IP:

$ lxc list
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| c1   | RUNNING | 10.110.99.112 (eth0) | fd42:6c7f:db14:1561:216:3eff:fec4:7b10 (eth0) | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+

And stops w/o -f.

To switch to legacy mode (for what it’s worth) you additionally need systemd.legacy_systemd_cgroup_controller=1, but lxd won’t start then.