Centos8 containers unable to automatically get ipv4 addresses after update

jsnjack · June 6, 2021, 6:26pm

Hi,

Since Friday (looks like related to Centos update), containers based on centos-8 images are not able to automatically get ipv4 address.

To reproduce:

lxc launch images:centos/8 fresh

lxc list
+-------+---------+---------------------+------+-----------+-----------+
| NAME  |  STATE  |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
+-------+---------+---------------------+------+-----------+-----------+
| fresh | RUNNING |                     |      | CONTAINER | 0         |

Logs:

-- Logs begin at Sun 2021-06-06 17:57:10 UTC, end at Sun 2021-06-06 17:57:21 UTC. --
Jun 06 17:57:10 fresh systemd-journald[425]: Journal started
Jun 06 17:57:10 fresh systemd-journald[425]: Runtime journal (/run/log/journal/bb2535490cf648c485782e5ff586b7e2) is 8.0M, max 1.5G, 1.5G free.
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: Permission denied
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' to 'kernel/core_pattern', ignoring: Permission denied
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '16' to 'kernel/sysrq', ignoring: Permission denied
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '1' to 'kernel/core_uses_pid', ignoring: Permission denied
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '1' to 'kernel/kptr_restrict', ignoring: Permission denied
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write 'fq_codel' to 'net/core/default_qdisc', ignoring: No such file or directory
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '1' to 'fs/protected_hardlinks', ignoring: Permission denied
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '1' to 'fs/protected_symlinks', ignoring: Permission denied
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '81920' to 'net/core/optmem_max', ignoring: No such file or directory
Jun 06 17:57:10 fresh systemd-sysctl[426]: Couldn't write '4194304' to 'kernel/pid_max', ignoring: Permission denied
Jun 06 17:57:10 fresh systemd[1]: Starting Flush Journal to Persistent Storage...
Jun 06 17:57:10 fresh systemd-journald[425]: Runtime journal (/run/log/journal/bb2535490cf648c485782e5ff586b7e2) is 8.0M, max 1.5G, 1.5G free.
Jun 06 17:57:10 fresh systemd[1]: Started Flush Journal to Persistent Storage.
Jun 06 17:57:10 fresh systemd[1]: Starting Create Volatile Files and Directories...
Jun 06 17:57:10 fresh systemd[1]: Started Rebuild Journal Catalog.
Jun 06 17:57:10 fresh systemd[1]: Started Create Volatile Files and Directories.
Jun 06 17:57:10 fresh systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Jun 06 17:57:10 fresh systemd[1]: Started Update UTMP about System Boot/Shutdown.
Jun 06 17:57:10 fresh systemd[1]: Started udev Coldplug all Devices.
Jun 06 17:57:10 fresh systemd[1]: Started Rebuild Dynamic Linker Cache.
Jun 06 17:57:10 fresh systemd[1]: Started Rebuild Hardware Database.
Jun 06 17:57:10 fresh systemd[1]: Starting Update is Completed...
Jun 06 17:57:10 fresh systemd[1]: Starting udev Kernel Device Manager...
Jun 06 17:57:10 fresh systemd[1]: Started Update is Completed.
Jun 06 17:57:10 fresh systemd[1]: Started udev Kernel Device Manager.
Jun 06 17:57:10 fresh systemd[1]: Reached target System Initialization.
Jun 06 17:57:10 fresh systemd[1]: Started dnf makecache --timer.
Jun 06 17:57:10 fresh systemd[1]: Listening on D-Bus System Message Bus Socket.
Jun 06 17:57:10 fresh systemd[1]: Reached target Sockets.
Jun 06 17:57:10 fresh systemd[1]: Reached target Basic System.
Jun 06 17:57:10 fresh systemd-udevd[442]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 06 17:57:10 fresh systemd[1]: Starting Turn off network device...
Jun 06 17:57:10 fresh systemd-udevd[443]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jun 06 17:57:10 fresh systemd[1]: Starting Login Service...
Jun 06 17:57:10 fresh systemd[1]: Started Daily Cleanup of Temporary Directories.
Jun 06 17:57:10 fresh systemd[1]: Reached target Timers.
Jun 06 17:57:10 fresh systemd[1]: Started D-Bus System Message Bus.
Jun 06 17:57:10 fresh systemd[1]: Started Turn off network device.
Jun 06 17:57:10 fresh systemd[1]: Starting Network Manager...
Jun 06 17:57:10 fresh systemd-logind[447]: New seat seat0.
Jun 06 17:57:10 fresh systemd[1]: Started Login Service.
Jun 06 17:57:10 fresh NetworkManager[451]: <info>  [1623002230.9606] NetworkManager (version 1.30.0-7.el8) is starting... (for the first time)
Jun 06 17:57:10 fresh NetworkManager[451]: <info>  [1623002230.9607] Read config: /etc/NetworkManager/NetworkManager.conf
Jun 06 17:57:10 fresh systemd[1]: Started Network Manager.
Jun 06 17:57:10 fresh NetworkManager[451]: <info>  [1623002230.9623] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
Jun 06 17:57:10 fresh systemd[1]: Reached target Network.
Jun 06 17:57:10 fresh systemd[1]: Starting Permit User Sessions...
Jun 06 17:57:10 fresh systemd[1]: Starting Network Manager Wait Online...
Jun 06 17:57:10 fresh NetworkManager[451]: <info>  [1623002230.9641] manager[0x55ef14d870a0]: monitoring kernel firmware directory '/lib/firmware'.
Jun 06 17:57:10 fresh dbus-daemon[448]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.2' (uid=0 pid=451 comm="/usr/sbin/Ne
tworkManager --no-daemon " label="kernel")
Jun 06 17:57:10 fresh systemd[1]: Starting Hostname Service...
Jun 06 17:57:10 fresh systemd[1]: Started Permit User Sessions.
Jun 06 17:57:10 fresh systemd[1]: Started Command Scheduler.
Jun 06 17:57:10 fresh systemd[1]: Started Console Getty.
Jun 06 17:57:10 fresh systemd[1]: Reached target Login Prompts.
Jun 06 17:57:10 fresh crond[461]: (CRON) STARTUP (1.5.2)
Jun 06 17:57:10 fresh crond[461]: (CRON) INFO (Syslog will be used instead of sendmail.)
Jun 06 17:57:10 fresh crond[461]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 42% if used.)
Jun 06 17:57:10 fresh crond[461]: (CRON) INFO (running with inotify support)
Jun 06 17:57:11 fresh dbus-daemon[448]: [system] Successfully activated service 'org.freedesktop.hostname1'
Jun 06 17:57:11 fresh systemd[1]: Started Hostname Service.
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0018] hostname: hostname: using hostnamed
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0022] dns-mgr[0x55ef14d7b130]: init: dns=default,systemd-resolved rc-manager=symlink
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0029] rfkill0: found Wi-Fi radio killswitch (at /sys/devices/platform/dell-laptop/rfkill/rfkill0) (platform driver dell-laptop)
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0057] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0058] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0059] manager: Networking is enabled by state file
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0060] dhcp-init: Using DHCP client 'internal'
Jun 06 17:57:11 fresh dbus-daemon[448]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.2' (uid=0 pid=451 comm="/usr
/sbin/NetworkManager --no-daemon " label="kernel")
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0075] settings: Loaded settings plugin: ifcfg-rh ("/usr/lib64/NetworkManager/1.30.0-7.el8/libnm-settings-plugin-ifcfg-rh.so")
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0075] settings: Loaded settings plugin: keyfile (internal)
Jun 06 17:57:11 fresh NetworkManager[451]: <warn>  [1623002231.0078] ifcfg-rh:     invalid MTU ''
Jun 06 17:57:11 fresh systemd[1]: Starting Network Manager Script Dispatcher Service...
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0093] device (lo): carrier: link connected
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0095] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0103] manager: (eth0): new Veth device (/org/freedesktop/NetworkManager/Devices/2)
Jun 06 17:57:11 fresh NetworkManager[451]: <info>  [1623002231.0125] manager: startup complete
Jun 06 17:57:11 fresh systemd[1]: Started Network Manager Wait Online.
Jun 06 17:57:11 fresh systemd[1]: Reached target Network is Online.
Jun 06 17:57:11 fresh dbus-daemon[448]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Jun 06 17:57:11 fresh systemd[1]: Starting System Logging Service...
Jun 06 17:57:11 fresh systemd[1]: Started Network Manager Script Dispatcher Service.
Jun 06 17:57:11 fresh rsyslogd[466]: [origin software="rsyslogd" swVersion="8.1911.0-7.el8" x-pid="466" x-info="https://www.rsyslog.com"] start
Jun 06 17:57:11 fresh rsyslogd[466]: imjournal: No statefile exists, /var/lib/rsyslog/imjournal.state will be created (ignore if this is first run): No such file or directory [v8.1911.0-7.e
l8 try https://www.rsyslog.com/e/2040 ]
Jun 06 17:57:11 fresh systemd[1]: Started System Logging Service.
Jun 06 17:57:11 fresh systemd[1]: Reached target Multi-User System.
Jun 06 17:57:11 fresh systemd[1]: Reached target Graphical Interface.
Jun 06 17:57:11 fresh systemd[1]: Starting Update UTMP about System Runlevel Changes...
Jun 06 17:57:11 fresh rsyslogd[466]: imjournal: journal files changed, reloading...  [v8.1911.0-7.el8 try https://www.rsyslog.com/e/0 ]
Jun 06 17:57:11 fresh systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Jun 06 17:57:11 fresh systemd[1]: Started Update UTMP about System Runlevel Changes.
Jun 06 17:57:11 fresh systemd[1]: Startup finished in 841ms.
Jun 06 17:57:21 fresh systemd[1]: NetworkManager-dispatcher.service: Succeeded.

Fedora 34

On my laptop (fedora 34) I’ve noticed that the container can get ipv4 if I bring up the interface manually:

[root@fresh ~]# nmcli
eth0: unmanaged
	"eth0"
	ethernet (veth), 00:16:3E:36:2E:D3, sw, mtu 1500

lo: unmanaged
	"lo"
	loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

[root@fresh ~]# nmcli connection up System\ eth0

$ lxc list
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
+-------+---------+---------------------+------+-----------+-----------+
| NAME  |  STATE  |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
+-------+---------+---------------------+------+-----------+-----------+
| fresh | RUNNING | 10.214.36.42 (eth0) |      | CONTAINER | 0         |

Centos 8

On the remote server, we have Centos 8 as host OS. When I create a new container, it is not able to get ipv4 address. If I try to bring up the connection manually, I have the following error:

[root@fresh ~]# nmcli connection up System\ eth0
Error: Connection activation failed: No suitable device found for this connection (device lo not available because device is strictly unmanaged).

The container has this version of NetworkManager:

[root@fresh ~]# dnf list --installed | grep NetworkManager
NetworkManager.x86_64               1:1.30.0-7.el8                    @baseos   
NetworkManager-libnm.x86_64         1:1.30.0-7.el8                    @baseos

However, If I downgrade NetworkManager to something older (1.26.0-12) it works!

Network details:

[client@lxd-node2 ~]$lxc network list
+----------------+----------+---------+----------------+------+-------------+---------+
|      NAME      |   TYPE   | MANAGED |      IPV4      | IPV6 | DESCRIPTION | USED BY |
+----------------+----------+---------+----------------+------+-------------+---------+
| enp5s0f3u2u3c2 | physical | NO      |                |      |             | 0       |
+----------------+----------+---------+----------------+------+-------------+---------+
| enp195s0       | physical | NO      |                |      |             | 0       |
+----------------+----------+---------+----------------+------+-------------+---------+
| lxdbr0         | bridge   | YES     | 192.168.2.1/24 | none | LXD network | 2       |
+----------------+----------+---------+----------------+------+-------------+---------+

[client@lxd-node2 ~]$lxc network show lxdbr0 
config:
  ipv4.address: 192.168.2.1/24
  ipv4.nat: "true"
  ipv6.address: none
description: LXD network
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/fresh
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

stgraber · June 7, 2021, 1:41am

According to our daily tests, this issue no longer affects the latest build of those images.

jsnjack · June 7, 2021, 11:59am

Thanks @stgraber ! I have just tested in on Fedora 34 host and it works fine.

However, the other host, that runs Centos 8 still has this problem:

[root@fresh ~]# nmcli c up System\ eth0
Error: Connection activation failed: No suitable device found for this connection (device lo not available because device is strictly unmanaged).

stgraber · June 7, 2021, 1:17pm

Is systemd-udev installed and running in that centos8 container?
I believe that’s what was needed to get nmcli to behave.

jsnjack · June 7, 2021, 1:45pm

It is installed and running:

[root@fresh ~]# ps uax | grep systemd-udev
root         432  0.0  0.0 104096  8344 ?        Ss   12:32   0:00 /usr/lib/systemd/systemd-udevd
root         569  0.0  0.0 229088  1156 pts/0    S+   13:43   0:00 grep --color=auto systemd-udev
[root@fresh ~]# dnf list --installed | grep systemd-udev
systemd-udev.x86_64                 239-45.el8                        @BaseOS

The container is created from the default centos8 image:

lxc launch images:centos/8 new

stgraber · June 7, 2021, 1:50pm

Can you show ls -lh /sys/class/net in the container?

I wonder if the difference between your setup and ours is that we have a kernel where network device ownership is now correct, letting udev do its job properly.

jsnjack · June 7, 2021, 3:13pm

This is the output:

[root@fresh ~]# ls -lh /sys/class/net
total 0
lrwxrwxrwx 1 nobody nobody 0 Jun  7 15:12 eth0 -> ../../devices/virtual/net/eth0
lrwxrwxrwx 1 nobody nobody 0 Jun  7 15:12 lo -> ../../devices/virtual/net/lo

stgraber · June 7, 2021, 4:35pm

Ok, so yeah, your kernel is what’s causing this.
Newer kernels look like this instead:

stgraber@shell01:~$ ls -lh /sys/class/net/
total 0
lrwxrwxrwx 1 root   root       0 Jun  2 23:23 eth0 -> ../../devices/virtual/net/eth0
lrwxrwxrwx 1 root   root       0 Jun  2 23:23 lo -> ../../devices/virtual/net/lo

Which then allows udevd to trigger the events it needs on startup.
So in your case, if you can’t get a more recent kernel on that host, you may need to workaround things directly in the container by moving away from NetworkManager and maybe using systemd-networkd instead.

jsnjack · June 7, 2021, 9:14pm

Ah I see, the owner is different (nobody vs root).

I have found out something interesting. We have 2 nodes running Centos 8 as host system:

client@lxd-node1:~$ uname -a
Linux lxd-node1 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1 16:14:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[client@lxd-node2 ~]$uname -a
Linux lxd-node2 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1 16:14:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

One of them has this issue (unable to assign IP) and another one doesn’t! Both servers have the same version of kernel and systemd packages.

If it is not kernel, what could it be?

stgraber · June 7, 2021, 9:30pm

Are you using normal bridged networking on both systems or is one using macvlan or something like that?

jsnjack · June 7, 2021, 9:31pm

Both use normal bridged networking

stgraber · June 7, 2021, 10:00pm

Hmm, I guess compare ps fauxww to see if there’s any difference, if not, maybe look directly at the NetworkManager logs?

If NM isn’t critical to you, switching over to networkd may be the easy way out though.
The only reason why we didn’t do it by default in the image is that cloud-init requires NM and we wanted things to be consistent across the two image types.

jacob · June 8, 2021, 8:41am

I’m also having this issue.

It worked fine a few days ago - at this point I saved (lxc publish ..) a couple of images, that worked fine, but no longer do. Launching new images images:centos/8 is the same. CentOS 7 works fine.

I’m running OpenSuse Leap 15.2

[root@thorough-sponge ~]# ls -lh /sys/class/net
total 0
lrwxrwxrwx 1 nobody nobody 0 Jun  8 08:27 eth0 -> ../../devices/virtual/net/eth0
lrwxrwxrwx 1 root   root   0 Jun  8 06:36 lo -> ../../devices/virtual/net/lo
[root@thorough-sponge ~]# uname -a
Linux thorough-sponge 5.3.18-lp152.78-default #1 SMP Tue Jun 1 14:53:21 UTC 2021 (556d823) x86_64 x86_64 x86_64 GNU/Linux

Perhaps some relevant log lines:

Jun  8 06:36:02 thorough-sponge systemd-sysctl[428]: Couldn't write 'fq_codel' to 'net/core/default_qdisc', ignoring: No such file or directory
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4462] NetworkManager (version 1.30.0-7.el8) is starting... (for the first time)
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4463] Read config: /etc/NetworkManager/NetworkManager.conf
Jun  8 06:36:02 thorough-sponge systemd[1]: Started Network Manager.
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4480] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
Jun  8 06:36:02 thorough-sponge systemd[1]: Reached target Network.
Jun  8 06:36:02 thorough-sponge systemd[1]: Starting Permit User Sessions...
Jun  8 06:36:02 thorough-sponge systemd[1]: Starting Network Manager Wait Online...
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4498] manager[0x55b89c2750a0]: monitoring kernel firmware directory '/lib/firmware'.
Jun  8 06:36:02 thorough-sponge systemd[1]: systemd-hostnamed.service: Failed to reset devices.list: Operation not permitted
Jun  8 06:36:02 thorough-sponge systemd[1]: Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Jun  8 06:36:02 thorough-sponge systemd[1]: NetworkManager.service: Failed to reset devices.list: Operation not permitted
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4895] hostname: hostname: using hostnamed
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4898] dns-mgr[0x55b89c26c130]: init: dns=default,systemd-resolved rc-manager=symlink
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4905] rfkill1: found WWAN radio killswitch (at /sys/devices/platform/thinkpad_acpi/rfkill/rfkill1) (platform driver thinkpad_acpi)
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4920] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4921] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4921] manager: Networking is enabled by state file
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4922] dhcp-init: Using DHCP client 'internal'
Jun  8 06:36:02 thorough-sponge dbus-daemon[448]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.1' (uid=0 pid=452 comm="/usr/sbin/NetworkManager --no-daemon ")
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4933] settings: Loaded settings plugin: ifcfg-rh ("/usr/lib64/NetworkManager/1.30.0-7.el8/libnm-settings-plugin-ifcfg-rh.so")
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4933] settings: Loaded settings plugin: keyfile (internal)
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <warn>  [1623134162.4937] ifcfg-rh:     invalid MTU ''
Jun  8 06:36:02 thorough-sponge systemd[1]: NetworkManager-dispatcher.service: Failed to reset devices.list: Operation not permitted
Jun  8 06:36:02 thorough-sponge systemd[1]: Starting Network Manager Script Dispatcher Service...
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4949] device (lo): carrier: link connected
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4952] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Jun  8 06:36:02 thorough-sponge NetworkManager[452]: <info>  [1623134162.4961] manager: (eth0): new Veth device (/org/freedesktop/NetworkManager/Devices/2)
Jun  8 06:36:02 thorough-sponge dbus-daemon[448]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Jun  8 06:36:02 thorough-sponge systemd[1]: Started Network Manager Script Dispatcher Service.
Jun  8 06:36:12 thorough-sponge systemd[1]: NetworkManager-dispatcher.service: Succeeded.
Jun  8 06:36:20 thorough-sponge NetworkManager[452]: <info>  [1623134180.3182] agent-manager: agent[86a224099898b782,:1.7/nmcli-connect/0]: agent registered
Jun  8 06:36:20 thorough-sponge NetworkManager[452]: <info>  [1623134180.3201] audit: op="connection-activate" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="System eth0" result="fail" reason="No suitable device found for this connection (device lo not available because device is strictly unmanaged)."
Jun  8 06:36:32 thorough-sponge systemd[1]: systemd-hostnamed.service: Succeeded.
Jun  8 06:37:02 thorough-sponge systemd[1]: NetworkManager-wait-online.service: Main process exited, code=exited, status=1/FAILURE
Jun  8 06:37:02 thorough-sponge systemd[1]: NetworkManager-wait-online.service: Failed with result 'exit-code'.
Jun  8 06:37:02 thorough-sponge systemd[1]: Failed to start Network Manager Wait Online.
Jun  8 06:37:02 thorough-sponge systemd[1]: network-connection-activate.service: Failed to reset devices.list: Operation not permitted
Jun  8 06:37:02 thorough-sponge systemd[1]: Starting Activate connection...
Jun  8 06:37:02 thorough-sponge systemd[1]: Reached target Network is Online.
Jun  8 06:37:02 thorough-sponge NetworkManager[452]: <info>  [1623134222.5446] agent-manager: agent[f3f6c571ca5580ec,:1.8/nmcli-connect/0]: agent registered
Jun  8 06:37:02 thorough-sponge NetworkManager[452]: <info>  [1623134222.5452] audit: op="connection-activate" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="System eth0" result="fail" reason="No suitable device found for this connection (device lo not available because device is strictly unmanaged)."
Jun  8 06:37:02 thorough-sponge nmcli[494]: Error: Connection activation failed: No suitable device found for this connection (device lo not available because device is strictly unmanaged).

jsnjack · June 8, 2021, 9:40am

ps output is different indeed:

Container with ip

[root@fresh ~]# ps fauxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         616  0.0  0.0 232064  3760 pts/0    Ss   09:34   0:00 bash
root         630  0.0  0.0 264548  3836 pts/0    R+   09:34   0:00  \_ ps fauxww
root           1  0.0  0.0 101448 10656 ?        Ss   00:02   0:00 /sbin/init
root         438  0.0  0.0 100840 11784 ?        Ss   00:02   0:00 /usr/lib/systemd/systemd-journald
dbus         444  0.0  0.0  54196  5108 ?        Ss   00:02   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         445  0.0  0.0  90592  7064 ?        Ss   00:02   0:00 /usr/lib/systemd/systemd-logind
root         447  0.0  0.0 369008 20388 ?        Ssl  00:02   0:00 /usr/sbin/NetworkManager --no-daemon
root         456  0.0  0.0   6548  1724 pts/0    Ss+  00:02   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
root         458  0.0  0.0  22896  3124 ?        Ss   00:02   0:00 /usr/sbin/crond -n
root         467  0.0  0.0 211596  4368 ?        Ssl  00:02   0:00 /usr/sbin/rsyslogd -n

Container without IP

[root@fresh ~]# ps fauxww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         464  0.0  0.0 232064  3764 pts/0    Ss   09:34   0:00 bash
root         480  0.0  0.0 264548  3848 pts/0    R+   09:34   0:00  \_ ps fauxww
root           1  0.0  0.0 248768 10824 ?        Ss   09:33   0:00 /sbin/init
root         428  0.0  0.0  98724 11652 ?        Ss   09:33   0:00 /usr/lib/systemd/systemd-journald
root         444  0.0  0.0 104096  8308 ?        Ss   09:33   0:00 /usr/lib/systemd/systemd-udevd
dbus         445  0.0  0.0  54188  5436 ?        Ss   09:33   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         446  0.0  0.0  90592  7168 ?        Ss   09:33   0:00 /usr/lib/systemd/systemd-logind
root         448  0.0  0.0 368720 17732 ?        Ssl  09:33   0:00 /usr/sbin/NetworkManager --no-daemon
root         451  0.0  0.0 294704  8252 ?        Ssl  09:33   0:00 /usr/bin/nm-online -s -q
root         458  0.0  0.0  22896  3128 ?        Ss   09:33   0:00 /usr/sbin/crond -n
root         459  0.0  0.0   6548  1612 console  Ss+  09:33   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux

jsnjack · June 8, 2021, 9:52am

And also boot logs:

Container with ip

Journal started
Runtime journal (/run/log/journal/af0e94fb36ae4876ba4523d94b04b139) is 8.0M, max 3.1G, 3.1G free.
Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: Read-only file system
Couldn't write '|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' to 'kernel/core_pattern', ignoring: Read-only file system
Couldn't write '16' to 'kernel/sysrq', ignoring: Read-only file system
Couldn't write '1' to 'kernel/core_uses_pid', ignoring: Read-only file system
Couldn't write '1' to 'kernel/kptr_restrict', ignoring: Read-only file system
Couldn't write 'fq_codel' to 'net/core/default_qdisc', ignoring: No such file or directory
Couldn't write '1' to 'fs/protected_hardlinks', ignoring: Read-only file system
Couldn't write '1' to 'fs/protected_symlinks', ignoring: Read-only file system
Couldn't write '81920' to 'net/core/optmem_max', ignoring: No such file or directory
Couldn't write '4194304' to 'kernel/pid_max', ignoring: Read-only file system
Starting Flush Journal to Persistent Storage...
Started Rebuild Journal Catalog.
Runtime journal (/run/log/journal/af0e94fb36ae4876ba4523d94b04b139) is 8.0M, max 3.1G, 3.1G free.
Started Flush Journal to Persistent Storage.
Starting Create Volatile Files and Directories...
Started Create Volatile Files and Directories.
Starting Update UTMP about System Boot/Shutdown...
Started Update UTMP about System Boot/Shutdown.
Started Rebuild Dynamic Linker Cache.
Started Rebuild Hardware Database.
Starting Update is Completed...
Started Update is Completed.
Reached target System Initialization.
Listening on D-Bus System Message Bus Socket.
Reached target Sockets.
Reached target Basic System.
Starting Login Service...
Started D-Bus System Message Bus.
Started Daily Cleanup of Temporary Directories.
Starting Turn off network device...
Started dnf makecache --timer.
Reached target Timers.
Started Turn off network device.
Starting Network Manager...
New seat seat0.
Started Login Service.
  [1623145559.2558] NetworkManager (version 1.30.0-7.el8) is starting... (for the first time)
  [1623145559.2558] Read config: /etc/NetworkManager/NetworkManager.conf
Started Network Manager.
  [1623145559.2567] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
Starting Network Manager Wait Online...
Reached target Network.
  [1623145559.2572] manager[0x559c6ac730a0]: monitoring kernel firmware directory '/lib/firmware'.
Starting Permit User Sessions...
[system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.2' (uid=0 pid=445 comm="/usr/sbin/NetworkManager --no-daemon ")
Starting Hostname Service...
Started Permit User Sessions.
Started Console Getty.
Reached target Login Prompts.
Started Command Scheduler.
(CRON) STARTUP (1.5.2)
(CRON) INFO (Syslog will be used instead of sendmail.)
(CRON) INFO (RANDOM_DELAY will be scaled with factor 2% if used.)
(CRON) INFO (running with inotify support)
[system] Successfully activated service 'org.freedesktop.hostname1'
Started Hostname Service.
  [1623145559.2721] hostname: hostname: using hostnamed
  [1623145559.2723] dns-mgr[0x559c6ac6b130]: init: dns=default,systemd-resolved rc-manager=symlink
  [1623145559.2741] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
  [1623145559.2741] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
  [1623145559.2742] manager: Networking is enabled by state file
  [1623145559.2743] dhcp-init: Using DHCP client 'internal'
[system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.2' (uid=0 pid=445 comm="/usr/sbin/NetworkManager --no-daemon ")
  [1623145559.2747] settings: Loaded settings plugin: ifcfg-rh ("/usr/lib64/NetworkManager/1.30.0-7.el8/libnm-settings-plugin-ifcfg-rh.so")
  [1623145559.2748] settings: Loaded settings plugin: keyfile (internal)
  [1623145559.2749] ifcfg-rh:     invalid MTU ''
Starting Network Manager Script Dispatcher Service...
  [1623145559.2759] device (lo): carrier: link connected
  [1623145559.2760] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
  [1623145559.2764] manager: (eth0): new Veth device (/org/freedesktop/NetworkManager/Devices/2)
  [1623145559.2770] device (eth0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
  [1623145559.2774] device (eth0): carrier: link connected
  [1623145559.2798] device (eth0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'managed')
[system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Started Network Manager Script Dispatcher Service.
  [1623145559.2807] policy: auto-activating connection 'System eth0' (5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03)
  [1623145559.2811] device (eth0): Activation: starting connection 'System eth0' (5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03)
  [1623145559.2812] device (eth0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
  [1623145559.2815] manager: NetworkManager state is now CONNECTING
  [1623145559.2816] device (eth0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
  [1623145559.2818] device (eth0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
  [1623145559.2820] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
  [1623145559.2892] dhcp4 (eth0): state changed unknown -> bound, address=192.168.1.215
  [1623145559.2899] device (eth0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
  [1623145559.2909] device (eth0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
  [1623145559.2911] device (eth0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
  [1623145559.2913] manager: NetworkManager state is now CONNECTED_LOCAL
  [1623145559.2918] manager: NetworkManager state is now CONNECTED_SITE
  [1623145559.2919] policy: set 'System eth0' (eth0) as default for IPv4 routing and DNS
[system] Activating via systemd: service name='org.freedesktop.resolve1' unit='dbus-org.freedesktop.resolve1.service' requested by ':1.2' (uid=0 pid=445 comm="/usr/sbin/NetworkManager --no-daemon ")
  [1623145559.2922] device (eth0): Activation: successful, device activated.
[system] Activation via systemd failed for unit 'dbus-org.freedesktop.resolve1.service': Unit dbus-org.freedesktop.resolve1.service not found.
  [1623145559.2926] manager: NetworkManager state is now CONNECTED_GLOBAL
  [1623145559.2928] manager: startup complete
Started Network Manager Wait Online.
Reached target Network is Online.
Starting System Logging Service...
Starting Activate connection...
[origin software="rsyslogd" swVersion="8.1911.0-7.el8" x-pid="463" x-info="https://www.rsyslog.com"] start
Started System Logging Service.
imjournal: No statefile exists, /var/lib/rsyslog/imjournal.state will be created (ignore if this is first run): No such file or directory [v8.1911.0-7.el8 try https://www.rsyslog.com/e/2040 ]
Reached target Multi-User System.
imjournal: journal files changed, reloading...  [v8.1911.0-7.el8 try https://www.rsyslog.com/e/0 ]
  [1623145559.3030] agent-manager: agent[0f664da0a3242499,:1.7/nmcli-connect/0]: agent registered
  [1623145559.3035] device (eth0): state change: activated -> deactivating (reason 'new-activation', sys-iface-state: 'managed')
  [1623145559.3037] manager: NetworkManager state is now DISCONNECTING
  [1623145559.3045] device (eth0): disconnecting for new activation request.
  [1623145559.3046] audit: op="connection-activate" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="System eth0" pid=465 uid=0 result="success"
  [1623145559.3049] device (eth0): state change: deactivating -> disconnected (reason 'new-activation', sys-iface-state: 'managed')
  [1623145559.3173] dhcp4 (eth0): canceled DHCP transaction
  [1623145559.3174] dhcp4 (eth0): state changed bound -> done
  [1623145559.3350] manager: NetworkManager state is now DISCONNECTED
  [1623145559.3353] device (eth0): Activation: starting connection 'System eth0' (5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03)
  [1623145559.3363] device (eth0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
  [1623145559.3366] manager: NetworkManager state is now CONNECTING
  [1623145559.3367] device (eth0): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
  [1623145559.3370] device (eth0): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
  [1623145559.3372] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
  [1623145559.3521] dhcp4 (eth0): state changed unknown -> bound, address=192.168.1.215
  [1623145559.3529] device (eth0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
  [1623145559.3538] device (eth0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
  [1623145559.3540] device (eth0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
  [1623145559.3543] manager: NetworkManager state is now CONNECTED_LOCAL
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/2)
  [1623145559.3550] manager: NetworkManager state is now CONNECTED_SITE
  [1623145559.3551] policy: set 'System eth0' (eth0) as default for IPv4 routing and DNS
Started Activate connection.
Reached target Graphical Interface.
Starting Update UTMP about System Runlevel Changes...
systemd-update-utmp-runlevel.service: Succeeded.
Started Update UTMP about System Runlevel Changes.
Startup finished in 574ms.
  [1623145559.3699] device (eth0): Activation: successful, device activated.
  [1623145559.3703] manager: NetworkManager state is now CONNECTED_GLOBAL
NetworkManager-dispatcher.service: Succeeded.
systemd-hostnamed.service: Succeeded.

Container without ip

Journal started
Runtime journal (/run/log/journal/d26ca3d209274fb385c425246e3eb904) is 8.0M, max 4.0G, 3.9G free.
Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: Permission denied
Couldn't write '|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e' to 'kernel/core_pattern', ignoring: Permission denied
Couldn't write '16' to 'kernel/sysrq', ignoring: Permission denied
Couldn't write '1' to 'kernel/core_uses_pid', ignoring: Permission denied
Couldn't write '1' to 'kernel/kptr_restrict', ignoring: Permission denied
Couldn't write 'fq_codel' to 'net/core/default_qdisc', ignoring: No such file or directory
Couldn't write '1' to 'fs/protected_hardlinks', ignoring: Permission denied
Couldn't write '1' to 'fs/protected_symlinks', ignoring: Permission denied
Couldn't write '81920' to 'net/core/optmem_max', ignoring: No such file or directory
Couldn't write '4194304' to 'kernel/pid_max', ignoring: Permission denied
Starting Flush Journal to Persistent Storage...
Runtime journal (/run/log/journal/d26ca3d209274fb385c425246e3eb904) is 8.0M, max 4.0G, 3.9G free.
Started Flush Journal to Persistent Storage.
Starting Create Volatile Files and Directories...
Started Rebuild Journal Catalog.
Started Create Volatile Files and Directories.
Starting Update UTMP about System Boot/Shutdown...
Started Update UTMP about System Boot/Shutdown.
Started Rebuild Dynamic Linker Cache.
Started Rebuild Hardware Database.
Starting Update is Completed...
Starting udev Kernel Device Manager...
Started Update is Completed.
Started udev Kernel Device Manager.
Started udev Coldplug all Devices.
Reached target System Initialization.
Started Daily Cleanup of Temporary Directories.
Listening on D-Bus System Message Bus Socket.
Reached target Sockets.
Started dnf makecache --timer.
Reached target Timers.
Reached target Basic System.
Started D-Bus System Message Bus.
Starting Login Service...
Starting Turn off network device...
Started Turn off network device.
Starting Network Manager...
  [1623144838.8176] NetworkManager (version 1.30.0-7.el8) is starting... (for the first time)
  [1623144838.8177] Read config: /etc/NetworkManager/NetworkManager.conf
Started Network Manager.
  [1623144838.8191] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
New seat seat0.
Starting Network Manager Wait Online...
Reached target Network.
Starting Permit User Sessions...
  [1623144838.8205] manager[0x55bbc6a1a0a0]: monitoring kernel firmware directory '/lib/firmware'.
Started Login Service.
[system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.1' (uid=0 pid=448 comm="/usr/sbin/NetworkManager --no-daemon ")
system.slice: Failed to reset devices.list: Operation not permitted
systemd-hostnamed.service: Failed to reset devices.list: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Failed to set devices.allow on /system.slice/systemd-hostnamed.service: Operation not permitted
Starting Hostname Service...
init.scope: Failed to reset devices.list: Operation not permitted
user.slice: Failed to reset devices.list: Operation not permitted
systemd-journal-flush.service: Failed to reset devices.list: Operation not permitted
proc-swaps.mount: Failed to reset devices.list: Operation not permitted
sys-kernel-config.mount: Failed to reset devices.list: Operation not permitted
systemd-udevd-control.socket: Failed to reset devices.list: Operation not permitted
systemd-journald.socket: Failed to reset devices.list: Operation not permitted
selinux-autorelabel-mark.service: Failed to reset devices.list: Operation not permitted
proc-loadavg.mount: Failed to reset devices.list: Operation not permitted
proc-diskstats.mount: Failed to reset devices.list: Operation not permitted
dev-random.mount: Failed to reset devices.list: Operation not permitted
systemd-tmpfiles-setup-dev.service: Failed to reset devices.list: Operation not permitted
systemd-tmpfiles-setup.service: Failed to reset devices.list: Operation not permitted
systemd-update-done.service: Failed to reset devices.list: Operation not permitted
systemd-journald-dev-log.socket: Failed to reset devices.list: Operation not permitted
proc-meminfo.mount: Failed to reset devices.list: Operation not permitted
systemd-sysusers.service: Failed to reset devices.list: Operation not permitted
systemd-udev-trigger.service: Failed to reset devices.list: Operation not permitted
NetworkManager.service: Failed to reset devices.list: Operation not permitted
systemd-sysctl.service: Failed to reset devices.list: Operation not permitted
proc-uptime.mount: Failed to reset devices.list: Operation not permitted
dev-ptmx.mount: Failed to reset devices.list: Operation not permitted
dev-lxd.mount: Failed to reset devices.list: Operation not permitted
dev-mqueue.mount: Failed to reset devices.list: Operation not permitted
systemd-udevd-kernel.socket: Failed to reset devices.list: Operation not permitted
systemd-update-utmp.service: Failed to reset devices.list: Operation not permitted
dbus.service: Failed to reset devices.list: Operation not permitted
systemd-journald.service: Failed to reset devices.list: Operation not permitted
systemd-coredump.socket: Failed to reset devices.list: Operation not permitted
systemd-journal-catalog-update.service: Failed to reset devices.list: Operation not permitted
system-getty.slice: Failed to reset devices.list: Operation not permitted
sys-devices-system-cpu-online.mount: Failed to reset devices.list: Operation not permitted
dev-.lxd\x2dmounts.mount: Failed to reset devices.list: Operation not permitted
NetworkManager-wait-online.service: Failed to reset devices.list: Operation not permitted
systemd-logind.service: Failed to reset devices.list: Operation not permitted
systemd-user-sessions.service: Failed to reset devices.list: Operation not permitted
dev-fuse.mount: Failed to reset devices.list: Operation not permitted
dev-null.mount: Failed to reset devices.list: Operation not permitted
-.mount: Failed to reset devices.list: Operation not permitted
proc-cpuinfo.mount: Failed to reset devices.list: Operation not permitted
ldconfig.service: Failed to reset devices.list: Operation not permitted
dev-net-tun.mount: Failed to reset devices.list: Operation not permitted
sys-kernel-tracing.mount: Failed to reset devices.list: Operation not permitted
sys-fs-fuse-connections.mount: Failed to reset devices.list: Operation not permitted
proc-sys-kernel-random-boot_id.mount: Failed to reset devices.list: Operation not permitted
dev-urandom.mount: Failed to reset devices.list: Operation not permitted
dbus.socket: Failed to reset devices.list: Operation not permitted
dev-zero.mount: Failed to reset devices.list: Operation not permitted
sys-kernel-debug.mount: Failed to reset devices.list: Operation not permitted
systemd-initctl.socket: Failed to reset devices.list: Operation not permitted
dev-full.mount: Failed to reset devices.list: Operation not permitted
proc-stat.mount: Failed to reset devices.list: Operation not permitted
dev-tty.mount: Failed to reset devices.list: Operation not permitted
proc-sys-fs-binfmt_misc.mount: Failed to reset devices.list: Operation not permitted
systemd-udevd.service: Failed to reset devices.list: Operation not permitted
dracut-shutdown.service: Failed to reset devices.list: Operation not permitted
network-device-down.service: Failed to reset devices.list: Operation not permitted
systemd-hwdb-update.service: Failed to reset devices.list: Operation not permitted
Started Permit User Sessions.
crond.service: Failed to reset devices.list: Operation not permitted
Started Command Scheduler.
console-getty.service: Failed to reset devices.list: Operation not permitted
Started Console Getty.
Reached target Login Prompts.
systemd-user-sessions.service: Failed to reset devices.list: Operation not permitted
(CRON) STARTUP (1.5.2)
(CRON) INFO (Syslog will be used instead of sendmail.)
(CRON) INFO (RANDOM_DELAY will be scaled with factor 61% if used.)
(CRON) INFO (running with inotify support)
[system] Successfully activated service 'org.freedesktop.hostname1'
Started Hostname Service.
  [1623144839.0093] hostname: hostname: using hostnamed
  [1623144839.0095] dns-mgr[0x55bbc6a11130]: init: dns=default,systemd-resolved rc-manager=symlink
  [1623144839.0115] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
  [1623144839.0115] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
  [1623144839.0116] manager: Networking is enabled by state file
  [1623144839.0116] dhcp-init: Using DHCP client 'internal'
[system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.1' (uid=0 pid=448 comm="/usr/sbin/NetworkManager --no-daemon ")
  [1623144839.0127] settings: Loaded settings plugin: ifcfg-rh ("/usr/lib64/NetworkManager/1.30.0-7.el8/libnm-settings-plugin-ifcfg-rh.so")
  [1623144839.0127] settings: Loaded settings plugin: keyfile (internal)
  [1623144839.0130] ifcfg-rh:     invalid MTU ''
NetworkManager-dispatcher.service: Failed to reset devices.list: Operation not permitted
Starting Network Manager Script Dispatcher Service...
  [1623144839.0138] device (lo): carrier: link connected
  [1623144839.0140] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
  [1623144839.0145] manager: (eth0): new Veth device (/org/freedesktop/NetworkManager/Devices/2)
[system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Started Network Manager Script Dispatcher Service.
NetworkManager-dispatcher.service: Succeeded.
systemd-hostnamed.service: Succeeded.
NetworkManager-wait-online.service: Main process exited, code=exited, status=1/FAILURE
NetworkManager-wait-online.service: Failed with result 'exit-code'.
Failed to start Network Manager Wait Online.
Reached target Network is Online.
rsyslog.service: Failed to reset devices.list: Operation not permitted
Starting System Logging Service...
network-connection-activate.service: Failed to reset devices.list: Operation not permitted
Starting Activate connection...
[origin software="rsyslogd" swVersion="8.1911.0-7.el8" x-pid="482" x-info="https://www.rsyslog.com"] start
imjournal: No statefile exists, /var/lib/rsyslog/imjournal.state will be created (ignore if this is first run): No such file or directory [v8.1911.0-7.el8 try https://www.rsyslog.com/e/2040 ]
Started System Logging Service.
Reached target Multi-User System.
  [1623144898.8822] agent-manager: agent[0b1bcdad98344905,:1.7/nmcli-connect/0]: agent registered
  [1623144898.8826] audit: op="connection-activate" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="System eth0" result="fail" reason="No suitable device found for this connection (device lo not available because device is strictly unmanaged)."
Error: Connection activation failed: No suitable device found for this connection (device lo not available because device is strictly unmanaged).
Started Activate connection.
Reached target Graphical Interface.
systemd-update-utmp-runlevel.service: Failed to reset devices.list: Operation not permitted
Starting Update UTMP about System Runlevel Changes...
network-connection-activate.service: Failed to reset devices.list: Operation not permitted
imjournal: journal files changed, reloading...  [v8.1911.0-7.el8 try https://www.rsyslog.com/e/0 ]
systemd-update-utmp-runlevel.service: Succeeded.
Started Update UTMP about System Runlevel Changes.
Startup finished in 1min 778ms.
Stopping udev Kernel Device Manager...
systemd-udevd.service: Succeeded.
Stopped udev Kernel Device Manager.
  [1623145050.4210] agent-manager: agent[cd3aee8cb5297848,:1.8/nmcli-connect/0]: agent registered
  [1623145050.4214] audit: op="connection-activate" uuid="5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03" name="System eth0" result="fail" reason="No suitable device found for this connection (device lo not available because device is strictly unmanaged)."

jsnjack · June 8, 2021, 12:52pm

Making a container privileged allows it to get an ip address

stgraber · June 8, 2021, 1:16pm

I really have how fragile NetworkManager is in containers… Can you try removing systemd-udev from the broken container, see if that makes it work?

jsnjack · June 8, 2021, 1:39pm

Removed it, didn’t help. The container doesn’t have an ip address:

[root@fresh ~]# journalctl -u NetworkManager-wait-online
-- Logs begin at Tue 2021-06-08 13:35:58 UTC, end at Tue 2021-06-08 13:36:58 UTC. --
Jun 08 13:35:58 fresh systemd[1]: Starting Network Manager Wait Online...
Jun 08 13:35:58 fresh systemd[1]: NetworkManager-wait-online.service: Failed to reset devices.list: Operation not per
mitted
Jun 08 13:36:58 fresh systemd[1]: NetworkManager-wait-online.service: Main process exited, code=exited, status=1/FAIL
URE
Jun 08 13:36:58 fresh systemd[1]: NetworkManager-wait-online.service: Failed with result 'exit-code'.
Jun 08 13:36:58 fresh systemd[1]: Failed to start Network Manager Wait Online.

[root@fresh ~]# nmcli con up System\ eth0
Error: Connection activation failed: No suitable device found for this connection (device lo not available because device is strictly unmanaged).

trystan · July 12, 2021, 7:23pm

hitting this on all oracle/8 and oracle/8/cloud images as well.

the host is also an oracle 8 machine

host kernel : 5.4.17-2102.202.5.el8uek.x86_64 #2 SMP Sat May 22 16:16:03 PDT 2021 x86_64 x86_64 x86_64 GNU/Linux

container /sys/class/net:

    [root@test3 ~]# ls -lh /sys/class/net/
total 0
lrwxrwxrwx. 1 nobody nobody 0 Jul 12 18:46 eth0 -> ../../devices/virtual/net/eth0
lrwxrwxrwx. 1 root   root   0 Jul 12 18:46 lo -> ../../devices/virtual/net/lo

container nmcli:

[root@test3 ~]# nmcli device show
GENERAL.DEVICE:                         eth0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         00:16:3E:CD:3D:CB
GENERAL.MTU:                            1500
GENERAL.STATE:                          10 (unmanaged)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
WIRED-PROPERTIES.CARRIER:               off

GENERAL.DEVICE:                         lo
GENERAL.TYPE:                           loopback
GENERAL.HWADDR:                         00:00:00:00:00:00
GENERAL.MTU:                            65536
GENERAL.STATE:                          10 (unmanaged)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
IP4.ADDRESS[1]:                         127.0.0.1/8
IP4.GATEWAY:                            --
IP6.ADDRESS[1]:                         ::1/128
IP6.GATEWAY:                            --

Gilbert_Standen · July 19, 2021, 1:44am

The only solution that has worked so far for this DHCP issue on Oracle Linux 8 host running Oracle Linux 8 LXD containers, is the one from @jsnjack to set the LXD container to “privileged”. My use case is the same as @trystan.

Is there any other solution for getting Oracle Linux 8 LXD containers to obtain DHCP address successfully ?