No Network in CentOS 7 Container

The CentOS 7 container from images:centos/7/amd64 doesn’t get a network connection.

The host is Gentoo (using OpenRC for init, and dhcpcd as network manager).

Kernel: 5.4.97
LXD: 4.0.4-r5 (latest available via portage)

I also tried running archlinux/current/amd64, alpine/3.13/amd64 and centos/8/amd64: All of them just worked.

Here is what I did to check the connection, both of CentOS 7 (c7) and CentOS 8 (c8):

# lxc exec c7 ping d-mn.org
ping: d-mn.org: Name or service not known
# lxc exec c7 ping 185.26.156.93
connect: Network is unreachable
# lxc exec c8 ping d-mn.org
PING d-mn.org (185.26.156.93) 56(84) bytes of data.
64 bytes from gehrels.uberspace.de (185.26.156.93): icmp_seq=1 ttl=56 time=25.9 ms
	…
# lxc exec c7 ip route
# there’s no output by this command
# lxc exec c8 ip route
default via 10.204.58.1 dev eth0 proto dhcp metric 100
10.204.58.0/24 dev eth0 proto kernel scope link src 10.204.58.8 metric 100

Initially launching c7 using lxc launch images:centos/7/amd64 c7 went without any errors.
But the container could not be stopped without -f: lxc stop -f c7.

The container did not get an IPV4 address.

# lxc ls

+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |        IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c7   | RUNNING |                    | fd42:c3a4:d8ce:f043:216:3eff:fe6c:846e (eth0) | CONTAINER | 0         |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c8   | RUNNING | 10.204.58.8 (eth0) | fd42:c3a4:d8ce:f043:216:3eff:fecd:ba31 (eth0) | CONTAINER | 0         |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
# lxc info c7 --show-log

Name: c7
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/13 11:11 UTC
Status: Running
Type: container
Profiles: default
Pid: 13934
Ips:
  eth0:	inet6	fd42:c3a4:d8ce:f043:216:3eff:fe6c:846e	veth737a0cf6
  eth0:	inet6	fe80::216:3eff:fe6c:846e	veth737a0cf6
  lo:	inet	127.0.0.1
  lo:	inet6	::1
Resources:
  Processes: 1
  Network usage:
    eth0:
      Bytes received: 65.35kB
      Bytes sent: 966B
      Packets received: 248
      Packets sent: 11
    lo:
      Bytes received: 2.12kB
      Bytes sent: 2.12kB
      Packets received: 26
      Packets sent: 26

Log:
# lxc config show c7

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Centos 7 amd64 (20210213_07:08)
  image.os: Centos
  image.release: "7"
  image.serial: "20210213_07:08"
  image.type: squashfs
  volatile.base_image: 8b8c8b9b93a740aea97c90163f10e4d0dedc4f56ad085651e1a9d6afb5c36b0b
  volatile.eth0.host_name: veth737a0cf6
  volatile.eth0.hwaddr: 00:16:3e:6c:84:6e
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

There is only one profile (default).

# lxc profile show default

config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/instances/c7
- /1.0/instances/c8

I hope I included all necessary information to help figure this out. I am grateful for any little hint. Thank you!

Hmm, maybe run systemctl --failed in the container?

Hi, thanks!

# systemctl --failed
Failed to get D-Bus connection: No such file or directory

Further investigating what’s up with D-Bus:

# dbus-monitor
Failed to open connection to session bus: Unable to autolaunch a dbus-daemon without a $DISPLAY for X11

This doesn’t make much sense to me.

Also, do you have an idea why the container can’t be stopped without using --force? Running lxc info --show-log c7 as suggested when cancelling lxc stop doesn’t show anything either.

Can you show lxc console --show-log c7?

Ooh, this looks promising!

# lxc start c7
# lxc console --show-log c7

Console log:

Failed to insert module 'autofs4'
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Cannot determine cgroup we are running in: No such file or directory
Failed to allocate manager object: No such file or directory
[!!!!!!] Failed to allocate manager object, freezing.

If you don’t already know by now what causes the problem I’ll double check the Gentoo Wiki on LXD/LXC tomorrow (again), maybe I misunderstood one of the instructions on cgroups.

I made sure I followed this to enable cgroupsv2 (I already had this done, so nothing changed):
https://wiki.gentoo.org/wiki/LXD#Running_systemd_based_containers_on_OpenRC_hosts

This made me think:

It is recommended to use cgroupsv2 as most containers support it and OCI runtimes crun and runc also expect cgroupsv2 to be present.

How can I check if a container supports cgroupsv2? Could this be an issue?


I also checked this part from the LXC entry:
https://wiki.gentoo.org/wiki/LXC#OpenRC_configuration_pre-check

This basically says to do the opposite, making sure that cgroups v1 are used. Dos it apply to LXD as well?