Veth interfaces persist and multiply after container restart (with centos containers only?)

I m have no idea why this is happening so I just tell details maybe something will stick, this happens only with centos containers. . Ask anything.

This is dev server running lxd 3.0.3 deb
Linux dragon-ball-radar 5.3.0-25-generic #27~18.04.1-Ubuntu SMP Fri Dec 6 11:24:17 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux.

So far I have only used ubuntu 16/18 as virtualmachines but decided to try freeipa and used centos 8, because it seems reasonable.
I installed freeipa server and replica they have separate IP adresses x.x.x.100 and x.x.x.101 and for some reason pings start drop, or worse they keep going even after I shutdown that centos machine! This seems to happen after second freeipa installation, I can stop both containers and pings will still go for quite some time. Weird behaviour seems to start when starting stoping adding more centos machines.

There is like 10 other ubuntu machines and nothing like that happens, but freeipa messes a bit with network, still this shouldnt happen, because this is container after all.

Sometimes my pings dont drop, but some pings get replied by container others by?? I checked mac address with nmap and they match container’s mac address, maybe there is some cache something bug.

nmap which was run in centos shows many dropped packets by kernel.

5638 packets captured
8565 packets received by filter
2927 packets dropped by kernel

Any ideas would be welcome what I should try to check next?

Containers get IP address from router. Container info.
architecture: x86_64
config:
image.architecture: amd64
image.description: Centos 8 amd64 (20191204_07:08)
image.os: Centos
image.release: “8”
image.serial: “20191204_07:08”
limits.memory: 16GB
volatile.base_image: 54915ff915d2e9a1745f2df3d9a03abcf6a7d03437d6c1d1e726ebee3400e8cf
volatile.eth0.hwaddr: 00:16:3e:0b:70:bb
volatile.idmap.base: “0”
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]’
volatile.last_state.power: RUNNING
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: ceph-lxd
type: disk
ephemeral: false
profiles:
- default_ceph
stateful: false
description: “”

Ok found something interesting and weird :slight_smile: Veth interfaces they are not being shutdown. But only if container is centos. ipaclient is ubuntu machine.

andriusjurkus@dragon-ball-radar:~$ ip a | wc -l
110
andriusjurkus@dragon-ball-radar:~$ lxc restart ipaclient
andriusjurkus@dragon-ball-radar:~$ ip a | wc -l
110
andriusjurkus@dragon-ball-radar:~$ lxc restart centostest
andriusjurkus@dragon-ball-radar:~$ ip a | wc -l
114

Interestingly just launched clean centos image 8, restarted it, and then veth interfaces dont multiply. So it looks like something inside container (freeipa installation soemthing) causes this.

Its reproducable on clean centos 8 with
dnf install nfs-utils and all other dependencies can already be here but somehow nfs-utils package make this this happen.

Name : nfs-utils
Epoch : 1
Version : 2.3.3
Release : 14.el8_0
Arch : i686
Size : 488 k
Source : nfs-utils-2.3.3-14.el8_0.src.rpm
Repo : BaseOS
Summary : NFS utilities and supporting clients and daemons for the kernel NFS server
URL : http://linux-nfs.org/
License : MIT and GPLv2 and GPLv2+ and BSD
Description : The nfs-utils package provides a daemon for the kernel NFS server and
: related tools, which provides a much higher level of performance than the
: traditional Linux NFS server used by most users.
:
: This package also contains the showmount program. Showmount queries the
: mount daemon on a remote host for information about the NFS (Network File
: System) server on the remote host. For example, showmount can display the
: clients which are mounted on that host.
:
: This package also contains the mount.nfs and umount.nfs program.

That’s usually the sign of a kernel bug which is causing the network namespace to be kept around when the container is dead.

You can usually confirm it by looking at dmesg, you’ll see some odd refcounting errors around network, usually talking about your loopback device.

If that’s the case, there’s not much we can do, you should file a kernel bug and hopefully some of the recent work from @brauner can help figure out what’s keeping those network namespaces active.

Hmm I tried to look at dmesg of “clean lxc restart xxx” and failed one. Here are differences.

There is no errors with loopback, but failed container lack “veth disabled” lines.

uni058@K100:~/tmp$ diff -y gut.txt fail.txt                                                                                                                                       
XFS (rbd16): Unmounting Filesystem                              XFS (rbd16): Unmounting Filesystem                                                                                
audit: type=1400 audit(1576489595.568:2199): apparmor="STATUS | audit: type=1400 audit(1576489224.237:2183): apparmor="STATUS                                                     
br0: port 46(vethFYIKKU) entered disabled state               <                                                                                                                   
device vethFYIKKU left promiscuous mode                       <                                                                                                                   
br0: port 46(vethFYIKKU) entered disabled state               <                                                                                                                   
XFS (rbd16): Mounting V5 Filesystem                             XFS (rbd16): Mounting V5 Filesystem                                                                               
XFS (rbd16): Ending clean mount                                 XFS (rbd16): Ending clean mount                                                                                   
XFS (rbd16): Unmounting Filesystem                              XFS (rbd16): Unmounting Filesystem                                                                                
XFS (rbd16): Mounting V5 Filesystem                             XFS (rbd16): Mounting V5 Filesystem                                                                               
XFS (rbd16): Ending clean mount                                 XFS (rbd16): Ending clean mount                                                                                   
audit: type=1400 audit(1576489596.916:2200): apparmor="STATUS | audit: type=1400 audit(1576489228.829:2184): apparmor="STATUS                                                     
br0: port 46(veth3FO11N) entered blocking state               | br0: port 45(vethCAVTWE) entered blocking state                                                                   
br0: port 46(veth3FO11N) entered disabled state               | br0: port 45(vethCAVTWE) entered disabled state                                                                   
device veth3FO11N entered promiscuous mode                    | device vethCAVTWE entered promiscuous mode                                                                        
eth0: renamed from veth4MYRF3                                 | eth0: renamed from veth0DR3KD                                                                                     
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready         IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready                                                           
IPv6: ADDRCONF(NETDEV_CHANGE): veth3FO11N: link becomes ready | IPv6: ADDRCONF(NETDEV_CHANGE): vethCAVTWE: link becomes ready                                                     
br0: port 46(veth3FO11N) entered blocking state               | br0: port 45(vethCAVTWE) entered blocking state                                                                   
br0: port 46(veth3FO11N) entered forwarding state             | br0: port 45(vethCAVTWE) entered forwarding state                                                                 
IPv6: eth0: IPv6 duplicate address fe80::216:3eff:fe0a:c8c2 u   IPv6: eth0: IPv6 duplicate address fe80::216:3eff:fe0a:c8c2 u                                                     
audit: type=1400 audit(1576489597.828:2201): apparmor="DENIED | audit: type=1400 audit(1576489230.505:2185): apparmor="DENIED                                                     
br0: port 46(veth3FO11N) entered disabled state               | br0: port 45(vethCAVTWE) entered disabled state                                                                   
audit: type=1400 audit(1576489597.904:2202): apparmor="DENIED | audit: type=1400 audit(1576489230.569:2186): apparmor="DENIED                                                     
audit: type=1400 audit(1576489598.196:2203): apparmor="DENIED |                                                                                                                   
                                                              > audit: type=1400 audit(1576489230.853:2187): apparmor="DENIED                                                     
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready         IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready                                                           
br0: port 46(veth3FO11N) entered blocking state               | br0: port 45(vethCAVTWE) entered blocking state                                                                   
br0: port 46(veth3FO11N) entered forwarding state             | br0: port 45(vethCAVTWE) entered forwarding state
                                                              > IPv6: eth0: IPv6 duplicate address fe80::216:3eff:fe0a:c8c2 u

Hmm, so it’s most likely the same issue though for some reason the kernel isn’t logging those leaked namespaces somehow.

Normally when the last process in the container exits, the kernel will destroy its namespaces, including the network namespace, as this deletes one side of the veth pair, the other side goes away.

In your case, something is keeping your network namespace active at the kernel level, so the veth is never deleted.