After upgrading from Rocky Linux (RL) 9.0 to 9.1, container networking over the bridged interface stopped working

LXC
1

We are in an odd situation right now. On a host where LXC containers used to work while running Rocky Linux (RL) 9.0. After upgrading to RL 9.1, all containers still run, but their networking over the bridged interface doesn’t. More details are below. Any help/hints appreciated.

The setup

  1. We have two hosts running LXC containers. Both hosts have a bridged interface, both named br0. The utility brctl shows that br0 on this RL 9.1 host is up and has interfaces assigned to it. The assigned interface is up too.
  2. This host in trouble is running RL 9.1 (5.14.0-162.23.1.el9_1.x86_64)
  3. It runs LXC 4.0.12 (via lxc-ls –version)
  4. The other working host runs CentOS Linux release 7.9.2009 (Core) (3.10.0-1160.66.1.el7.x86_64)
  5. The working host runs LXC 3.2.1 (via lxc-ls –version)

The symptom

  1. The containers on the host running RL 9.1 are not reachable via ping from another host. Despite the fact that all containers and the two hosts use the same gateway, IP 192.168.20.3, on the same broadcast domain 192.168.20.0/24.
  2. Inside the LXC containers on the RL 9.1 host, it’s possible to ping br0 with its IP address 192.168.20.20
  3. But if the hostname of the host is used, there is a ‘Temporary failure in name resolution’ error
  4. In such LXC containers, ping the common gateway’s IP address 192.168.20.3 produces Destination Host Unreachable
  5. Ditto for ping 8.8.8.8.

More information

  1. Firewalld is not active
  2. SELinux is disabled per getenforce
  3. sudo iptables -L, sudo iptables -L -t nat, and sudo nft list ruleset show empty output
2

I’ll preface it with information that I’m not experienced with the “bridged” mode, so I’m not even sure what interfaces are supposed to be there, but…
Does the Rocky Linux 9.1 have the main interface up? If not, then it might have the same cause I reported here: https://github.com/lxc/distrobuilder/issues/701 . And in that case, to fix the main interface you would need to run nmcli c delete eth0 on boot in the Rocky Linux guest.

Or look into the connections if you don’t have any conflicting ones.

3

Please let me clearify: the host initially ran RL 9.0, then upgraded to 9.1. There are three containers, they respectively run

  1. RL 8
  2. Ubuntu 20.04
  3. Ubuntu 22.04

None of the containers run RL 9.x.

Currently, nmcli c show yields the following:

# nmcli c show
NAME                      UUID                                  TYPE        DEVICE      
System enp129s0f0         688bf5bf-d649-34b4-15eb-b07c50ac43f8  ethernet    enp129s0f0  
lo                        f0cb2335-83c4-45a4-9b47-5d3764c9b77c  loopback    lo          
br0                       6d2ad8e0-ccec-48f8-aa37-9e084a2f7c88  bridge      br0         
bridge-slave-enp6s0f0np0  61699151-f697-4c5f-a0cb-6df8ebbace33  ethernet    enp6s0f0np0 
enp129s0f1                51551853-78fd-443a-9f9d-b1efc71e49c9  ethernet    --          
ib0                       c372adca-c895-4930-bd47-0ff3b09cbda0  infiniband  --          
System enp0s25            340503b4-e679-f113-2778-985445badaee  ethernet    --          
Wired connection 1        dcf01433-dcf7-390e-aab3-4e90bf4a1e5e  ethernet    --          
Wired connection 2        99a13149-50b2-3757-ba27-f155465aec43  ethernet    --

ip a yields the following:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp16s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:b2:a8:d2:c5 brd ff:ff:ff:ff:ff:ff
3: enp129s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000
    link/ether 00:15:b2:a8:d2:c2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.12.200/24 brd 192.168.12.255 scope global noprefixroute enp129s0f0
       valid_lft forever preferred_lft forever
4: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 00:15:b2:a8:d2:c4 brd ff:ff:ff:ff:ff:ff
5: enp129s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:15:b2:a8:d2:c3 brd ff:ff:ff:ff:ff:ff
6: enp6s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master br0 state UP group default qlen 1000
    link/ether 24:8a:07:11:51:5c brd ff:ff:ff:ff:ff:ff
7: enp6s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:8a:07:11:51:5d brd ff:ff:ff:ff:ff:ff
8: ibp132s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc mq state UP group default qlen 256
    link/infiniband 00:00:00:67:fe:80:00:00:00:00:00:00:7c:fe:90:03:00:29:28:9e brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
11: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 24:8a:07:11:51:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.20/24 brd 192.168.20.255 scope global noprefixroute br0
       valid_lft forever preferred_lft forever
16: lxc_arm_ub2204@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether fe:36:30:2d:31:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 0

I don’t know if I missed anything. The two look OK to me.