Bridged network isn't down when CT is down

nobody · September 30, 2020, 6:55am

Very strange network issue… I have container (test-1) with 2 net interfaces in 2 bridges. I start ping one address of this container (192.168.0.1 for ex.) then i stop container (lxc-stop test-1), i see test-1 is stopped but ping still goes on. Timing and logs are below

09:33:54 - lxc-stop test-1
Sep 30 09:33:54 node-1 kernel: [2332605.285772] br0: port 5(veth2WC9PI) entered disabled state
Sep 30 09:33:57 node-1 kernel: [2332608.814034] kauditd_printk_skb: 11 callbacks suppressed
Sep 30 09:33:57 node-1 kernel: [2332608.814037] audit: type=1400 audit(1601447637.528:343): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/dev/" pid=5784 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.819328] audit: type=1400 audit(1601447637.536:344): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/sys/net/" pid=5786 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.821965] audit: type=1400 audit(1601447637.536:345): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/sys/" pid=5787 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.824589] audit: type=1400 audit(1601447637.540:346): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/sysrq-trigger" pid=5788 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.829946] audit: type=1400 audit(1601447637.544:347): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/sys/devices/virtual/net/" pid=5790 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.832684] audit: type=1400 audit(1601447637.548:348): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/sys/devices/virtual/net/" pid=5791 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.835318] audit: type=1400 audit(1601447637.552:349): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/cpuinfo" pid=5792 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.837940] audit: type=1400 audit(1601447637.552:350): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/diskstats" pid=5793 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.840628] audit: type=1400 audit(1601447637.556:351): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/meminfo" pid=5794 comm="mount" flags="ro, remount"
Sep 30 09:33:57 node-1 kernel: [2332608.843279] audit: type=1400 audit(1601447637.560:352): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/stat" pid=5795 comm="mount" flags="ro, remount"
Sep 30 09:34:32 node-1 kernel: [2332643.915404] br155: port 2(veth3HO0RY) entered disabled state
Sep 30 09:34:32 node-1 kernel: [2332643.915881] br0: port 3(vethGXLCMX) entered disabled state
Sep 30 09:34:32 node-1 kernel: [2332643.916482] device veth3HO0RY left promiscuous mode
Sep 30 09:34:32 node-1 kernel: [2332643.916485] br155: port 2(veth3HO0RY) entered disabled state
Sep 30 09:34:32 node-1 kernel: [2332643.987481] device vethGXLCMX left promiscuous mode
Sep 30 09:34:32 node-1 kernel: [2332643.987484] br0: port 3(vethGXLCMX) entered disabled state

09:39:02 - ping ends
Sep 30 09:39:02 node-1 kernel: [2332914.246982] br155: port 4(vethJH99DO) entered disabled state
Sep 30 09:39:02 node-1 kernel: [2332914.247491] br0: port 5(veth2WC9PI) entered disabled state
Sep 30 09:39:02 node-1 kernel: [2332914.248127] device vethJH99DO left promiscuous mode
Sep 30 09:39:02 node-1 kernel: [2332914.248129] br155: port 4(vethJH99DO) entered disabled state
Sep 30 09:39:03 node-1 kernel: [2332914.315108] device veth2WC9PI left promiscuous mode
Sep 30 09:39:03 node-1 kernel: [2332914.315111] br0: port 5(veth2WC9PI) entered disabled state

all this time, from 09:33:54 (when i shut off container) to 09:39:02 the ping continued. I saw an interface on hostnode

318: veth2WC9PI@if317: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether fe:f2:9d:1f:a6:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 15
    inet6 fe80::fcf2:9dff:fe1f:a6f0/64 scope link 
       valid_lft forever preferred_lft forever

So, why is this happening?

tomp · September 30, 2020, 8:40am

Can you post your container’s config file here please.

Also can you show the output of ip a on the host before and after the container is shutdown, as well as the output of ip a inside the container before shutdown.

nobody · September 30, 2020, 10:05am

Hi there, @tomp! Yeah, ofc

  # container config
# Common and Puppet specified includes
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.include = /var/lib/lxc/test-1/conf.d

# Distribution configuration
lxc.arch = x86_64

# Container specific configuration
lxc.rootfs.path = loop:/var/lib/lxc/test-1/rootdev
lxc.uts.name = test-1.local

# Local additions
lxc.monitor.unshare = 1
lxc.environment = TERM=linux
lxc.tty.max = 2
lxc.autodev = 1
lxc.start.auto = 1
lxc.prlimit.nofile = 10000

# Apparmor section
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 0
lxc.apparmor.raw = deny mount -> /proc/,
lxc.apparmor.raw = deny mount -> /sys/,

# Cgroups common
lxc.cgroup.dir.monitor = lxc.monitor/test-1
lxc.cgroup.dir.container = lxc/test-1
lxc.cgroup.dir.container.inner = ns

# Hooks & bindings
lxc.hook.mount = /var/lib/lxc/test-1/setup_routes
lxc.hook.pre-start = /usr/local/sbin/lxc/hook_pre_start_mkdir /srv/test
lxc.mount.entry = tmpfs srv/ramdisk/test tmpfs nodev,nosuid,size=1G,create=dir 0 0
lxc.mount.entry = /srv/test usr/share/test none ro,bind,create=dir 0 0

# From conf.d
lxc.cgroup.memory.limit_in_bytes = 5G
lxc.cgroup.memory.memsw.limit_in_bytes = 5G

lxc.net.0.flags = up
lxc.net.0.ipv4.address = 192.168.16.10/24
lxc.net.0.link = br0
lxc.net.0.name = eth0
lxc.net.0.type = veth

lxc.net.1.flags = up
lxc.net.1.ipv4.address = 10.0.0.2/24
lxc.net.1.ipv4.gateway = 10.0.0.253
lxc.net.1.link = br155
lxc.net.1.name = eth1
lxc.net.1.type = veth

#
# before stop
#
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 44:a8:42:27:51:43 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 44:a8:42:27:51:44 brd ff:ff:ff:ff:ff:ff
262: veth5b63a00@if261: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default 
    link/ether be:5e:9a:11:c7:ce brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::bc5e:9aff:fe11:c7ce/64 scope link 
       valid_lft forever preferred_lft forever
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
7: bond0.155@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br155 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
264: vethec52cbd@if263: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 36:85:65:dd:3b:42 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::3485:65ff:fedd:3b42/64 scope link 
       valid_lft forever preferred_lft forever
8: bond0.252@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.252.1/30 brd 192.168.252.3 scope global bond0.252
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.16.50/24 brd 192.168.16.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
10: br155: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br155
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
11: br-6697dc50ea30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:30:57:f1:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global br-6697dc50ea30
       valid_lft forever preferred_lft forever
    inet6 fe80::42:30ff:fe57:f103/64 scope link 
       valid_lft forever preferred_lft forever
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b8:ae:0b:18 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
14: vethd022fc9@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 56:1e:24:7f:d5:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::541e:24ff:fe7f:d5d2/64 scope link 
       valid_lft forever preferred_lft forever
16: veth70f80e5@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether fe:40:7f:dc:08:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fc40:7fff:fedc:83f/64 scope link 
       valid_lft forever preferred_lft forever
286: vethf2517fd@if285: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 8e:e4:a4:03:84:7f brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::8ce4:a4ff:fe03:847f/64 scope link 
       valid_lft forever preferred_lft forever
288: veth6f923a3@if287: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 5e:b5:7f:fc:95:6c brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::5cb5:7fff:fefc:956c/64 scope link 
       valid_lft forever preferred_lft forever
290: veth258b6c6@if289: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether d2:5a:a9:ca:75:77 brd ff:ff:ff:ff:ff:ff link-netnsid 6
    inet6 fe80::d05a:a9ff:feca:7577/64 scope link 
       valid_lft forever preferred_lft forever
292: vethfd9b486@if291: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 86:66:f4:7f:a9:ee brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::8466:f4ff:fe7f:a9ee/64 scope link 
       valid_lft forever preferred_lft forever
294: veth5d88bb8@if293: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 8a:bf:ad:79:ed:30 brd ff:ff:ff:ff:ff:ff link-netnsid 8
    inet6 fe80::88bf:adff:fe79:ed30/64 scope link 
       valid_lft forever preferred_lft forever
296: veth250919a@if295: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 02:24:46:ee:96:de brd ff:ff:ff:ff:ff:ff link-netnsid 9
    inet6 fe80::24:46ff:feee:96de/64 scope link 
       valid_lft forever preferred_lft forever
298: vethc600360@if297: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 0a:ea:0f:fc:cb:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 10
    inet6 fe80::8ea:fff:fefc:cba1/64 scope link 
       valid_lft forever preferred_lft forever
300: veth0e54aa2@if299: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 5e:59:0e:14:bd:ec brd ff:ff:ff:ff:ff:ff link-netnsid 11
    inet6 fe80::5c59:eff:fe14:bdec/64 scope link 
       valid_lft forever preferred_lft forever
302: veth4c3be0e@if301: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether a6:ed:c7:aa:e4:eb brd ff:ff:ff:ff:ff:ff link-netnsid 12
    inet6 fe80::a4ed:c7ff:feaa:e4eb/64 scope link 
       valid_lft forever preferred_lft forever
304: vethe39180c@if303: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether a2:af:61:59:ce:a8 brd ff:ff:ff:ff:ff:ff link-netnsid 13
    inet6 fe80::a0af:61ff:fe59:cea8/64 scope link 
       valid_lft forever preferred_lft forever
326: veth5WQ7E1@if325: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether fe:ed:4d:39:76:4e brd ff:ff:ff:ff:ff:ff link-netnsid 14
    inet6 fe80::fced:4dff:fe39:764e/64 scope link 
       valid_lft forever preferred_lft forever
328: veth9DCBCY@if327: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br155 state UP group default qlen 1000
    link/ether fe:e9:6c:ee:a1:4f brd ff:ff:ff:ff:ff:ff link-netnsid 14
    inet6 fe80::fce9:6cff:feee:a14f/64 scope link 
       valid_lft forever preferred_lft forever
#
# inside container 
#
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
325: eth0@if326: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 36:99:3c:8f:be:16 brd ff:ff:ff:ff:ff:ff
    inet 192.168.16.10/24 brd 192.168.16.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3499:3cff:fe8f:be16/64 scope link 
       valid_lft forever preferred_lft forever
327: eth1@if328: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 06:3a:1e:65:7b:14 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::43a:1eff:fe65:7b14/64 scope link 
       valid_lft forever preferred_lft forever

#
# after stop but net is still alive
#
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 44:a8:42:27:51:43 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 44:a8:42:27:51:44 brd ff:ff:ff:ff:ff:ff
262: veth5b63a00@if261: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default 
    link/ether be:5e:9a:11:c7:ce brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::bc5e:9aff:fe11:c7ce/64 scope link 
       valid_lft forever preferred_lft forever
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
7: bond0.155@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br155 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
264: vethec52cbd@if263: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 36:85:65:dd:3b:42 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::3485:65ff:fedd:3b42/64 scope link 
       valid_lft forever preferred_lft forever
8: bond0.252@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.252.1/30 brd 192.168.252.3 scope global bond0.252
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.16.50/24 brd 192.168.16.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
10: br155: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br155
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
11: br-6697dc50ea30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:30:57:f1:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global br-6697dc50ea30
       valid_lft forever preferred_lft forever
    inet6 fe80::42:30ff:fe57:f103/64 scope link 
       valid_lft forever preferred_lft forever
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b8:ae:0b:18 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
14: vethd022fc9@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 56:1e:24:7f:d5:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::541e:24ff:fe7f:d5d2/64 scope link 
       valid_lft forever preferred_lft forever
16: veth70f80e5@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether fe:40:7f:dc:08:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fc40:7fff:fedc:83f/64 scope link 
       valid_lft forever preferred_lft forever
286: vethf2517fd@if285: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 8e:e4:a4:03:84:7f brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::8ce4:a4ff:fe03:847f/64 scope link 
       valid_lft forever preferred_lft forever
288: veth6f923a3@if287: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 5e:b5:7f:fc:95:6c brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::5cb5:7fff:fefc:956c/64 scope link 
       valid_lft forever preferred_lft forever
290: veth258b6c6@if289: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether d2:5a:a9:ca:75:77 brd ff:ff:ff:ff:ff:ff link-netnsid 6
    inet6 fe80::d05a:a9ff:feca:7577/64 scope link 
       valid_lft forever preferred_lft forever
292: vethfd9b486@if291: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 86:66:f4:7f:a9:ee brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::8466:f4ff:fe7f:a9ee/64 scope link 
       valid_lft forever preferred_lft forever
294: veth5d88bb8@if293: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 8a:bf:ad:79:ed:30 brd ff:ff:ff:ff:ff:ff link-netnsid 8
    inet6 fe80::88bf:adff:fe79:ed30/64 scope link 
       valid_lft forever preferred_lft forever
296: veth250919a@if295: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 02:24:46:ee:96:de brd ff:ff:ff:ff:ff:ff link-netnsid 9
    inet6 fe80::24:46ff:feee:96de/64 scope link 
       valid_lft forever preferred_lft forever
298: vethc600360@if297: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 0a:ea:0f:fc:cb:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 10
    inet6 fe80::8ea:fff:fefc:cba1/64 scope link 
       valid_lft forever preferred_lft forever
300: veth0e54aa2@if299: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 5e:59:0e:14:bd:ec brd ff:ff:ff:ff:ff:ff link-netnsid 11
    inet6 fe80::5c59:eff:fe14:bdec/64 scope link 
       valid_lft forever preferred_lft forever
302: veth4c3be0e@if301: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether a6:ed:c7:aa:e4:eb brd ff:ff:ff:ff:ff:ff link-netnsid 12
    inet6 fe80::a4ed:c7ff:feaa:e4eb/64 scope link 
       valid_lft forever preferred_lft forever
304: vethe39180c@if303: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether a2:af:61:59:ce:a8 brd ff:ff:ff:ff:ff:ff link-netnsid 13
    inet6 fe80::a0af:61ff:fe59:cea8/64 scope link 
       valid_lft forever preferred_lft forever
326: veth5WQ7E1@if325: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether fe:ed:4d:39:76:4e brd ff:ff:ff:ff:ff:ff link-netnsid 14
    inet6 fe80::fced:4dff:fe39:764e/64 scope link 
       valid_lft forever preferred_lft forever
328: veth9DCBCY@if327: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br155 state UP group default qlen 1000
    link/ether fe:e9:6c:ee:a1:4f brd ff:ff:ff:ff:ff:ff link-netnsid 14
    inet6 fe80::fce9:6cff:feee:a14f/64 scope link 
       valid_lft forever preferred_lft forever


#
# after ping ends
#
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 44:a8:42:27:51:43 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 44:a8:42:27:51:44 brd ff:ff:ff:ff:ff:ff
262: veth5b63a00@if261: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default 
    link/ether be:5e:9a:11:c7:ce brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::bc5e:9aff:fe11:c7ce/64 scope link 
       valid_lft forever preferred_lft forever
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
7: bond0.155@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br155 state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
264: vethec52cbd@if263: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 36:85:65:dd:3b:42 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::3485:65ff:fedd:3b42/64 scope link 
       valid_lft forever preferred_lft forever
8: bond0.252@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.252.1/30 brd 192.168.252.3 scope global bond0.252
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.16.50/24 brd 192.168.16.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
10: br155: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 44:a8:42:27:51:41 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br155
       valid_lft forever preferred_lft forever
    inet6 fe80::46a8:42ff:fe27:5141/64 scope link 
       valid_lft forever preferred_lft forever
11: br-6697dc50ea30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:30:57:f1:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global br-6697dc50ea30
       valid_lft forever preferred_lft forever
    inet6 fe80::42:30ff:fe57:f103/64 scope link 
       valid_lft forever preferred_lft forever
12: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:b8:ae:0b:18 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
14: vethd022fc9@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 56:1e:24:7f:d5:d2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::541e:24ff:fe7f:d5d2/64 scope link 
       valid_lft forever preferred_lft forever
16: veth70f80e5@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether fe:40:7f:dc:08:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fc40:7fff:fedc:83f/64 scope link 
       valid_lft forever preferred_lft forever
286: vethf2517fd@if285: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 8e:e4:a4:03:84:7f brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::8ce4:a4ff:fe03:847f/64 scope link 
       valid_lft forever preferred_lft forever
288: veth6f923a3@if287: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 5e:b5:7f:fc:95:6c brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::5cb5:7fff:fefc:956c/64 scope link 
       valid_lft forever preferred_lft forever
290: veth258b6c6@if289: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether d2:5a:a9:ca:75:77 brd ff:ff:ff:ff:ff:ff link-netnsid 6
    inet6 fe80::d05a:a9ff:feca:7577/64 scope link 
       valid_lft forever preferred_lft forever
292: vethfd9b486@if291: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 86:66:f4:7f:a9:ee brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::8466:f4ff:fe7f:a9ee/64 scope link 
       valid_lft forever preferred_lft forever
294: veth5d88bb8@if293: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 8a:bf:ad:79:ed:30 brd ff:ff:ff:ff:ff:ff link-netnsid 8
    inet6 fe80::88bf:adff:fe79:ed30/64 scope link 
       valid_lft forever preferred_lft forever
296: veth250919a@if295: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 02:24:46:ee:96:de brd ff:ff:ff:ff:ff:ff link-netnsid 9
    inet6 fe80::24:46ff:feee:96de/64 scope link 
       valid_lft forever preferred_lft forever
298: vethc600360@if297: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 0a:ea:0f:fc:cb:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 10
    inet6 fe80::8ea:fff:fefc:cba1/64 scope link 
       valid_lft forever preferred_lft forever
300: veth0e54aa2@if299: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether 5e:59:0e:14:bd:ec brd ff:ff:ff:ff:ff:ff link-netnsid 11
    inet6 fe80::5c59:eff:fe14:bdec/64 scope link 
       valid_lft forever preferred_lft forever
302: veth4c3be0e@if301: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether a6:ed:c7:aa:e4:eb brd ff:ff:ff:ff:ff:ff link-netnsid 12
    inet6 fe80::a4ed:c7ff:feaa:e4eb/64 scope link 
       valid_lft forever preferred_lft forever
304: vethe39180c@if303: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6697dc50ea30 state UP group default 
    link/ether a2:af:61:59:ce:a8 brd ff:ff:ff:ff:ff:ff link-netnsid 13
    inet6 fe80::a0af:61ff:fe59:cea8/64 scope link 
       valid_lft forever preferred_lft forever

#
# logs
#
Sep 30 12:45:12 node-1 kernel: [2344084.198222] device eth1 left promiscuous mode
Sep 30 12:45:17 node-1 kernel: [2344088.451357] br0: port 4(veth5WQ7E1) entered disabled state
Sep 30 12:45:20 node-1 kernel: [2344092.026239] kauditd_printk_skb: 11 callbacks suppressed
Sep 30 12:45:20 node-1 kernel: [2344092.026242] audit: type=1400 audit(1601459120.759:396): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/dev/" pid=37489 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.032378] audit: type=1400 audit(1601459120.767:397): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/sys/net/" pid=37491 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.035466] audit: type=1400 audit(1601459120.767:398): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/sys/" pid=37492 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.038155] audit: type=1400 audit(1601459120.771:399): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/sysrq-trigger" pid=37493 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.044738] audit: type=1400 audit(1601459120.779:400): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/sys/devices/virtual/net/" pid=37495 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.047507] audit: type=1400 audit(1601459120.779:401): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/sys/devices/virtual/net/" pid=37496 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.050152] audit: type=1400 audit(1601459120.783:402): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/cpuinfo" pid=37497 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.053460] audit: type=1400 audit(1601459120.787:403): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/diskstats" pid=37498 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.056565] audit: type=1400 audit(1601459120.791:404): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/meminfo" pid=37499 comm="mount" flags="ro, remount"
Sep 30 12:45:20 node-1 kernel: [2344092.059235] audit: type=1400 audit(1601459120.791:405): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-test-1_</var/lib/lxc>" name="/proc/stat" pid=37500 comm="mount" flags="ro, remount"


Sep 30 12:49:30 node-1 kernel: [2344342.068192] br155: port 3(veth9DCBCY) entered disabled state
Sep 30 12:49:30 node-1 kernel: [2344342.068688] br0: port 4(veth5WQ7E1) entered disabled state
Sep 30 12:49:30 node-1 kernel: [2344342.069309] device veth9DCBCY left promiscuous mode
Sep 30 12:49:30 node-1 kernel: [2344342.069312] br155: port 3(veth9DCBCY) entered disabled state
Sep 30 12:49:30 node-1 kernel: [2344342.136177] device veth5WQ7E1 left promiscuous mode
Sep 30 12:49:30 node-1 kernel: [2344342.136180] br0: port 4(veth5WQ7E1) entered disabled state

As we can see, the pair of veth 327-328 (veth9DCBCY) is still alive over ~4-5min even after lxc-stop test-1 . The question is: Why? How to avoid this behavior?

tomp · September 30, 2020, 10:17am

If the address is still pinging, and its not bound on any of the host’s interfaces, then it suggests the container is still infact running (in some form).

What does ps aux | grep lxc show? It may be worth enabling the debug logging option in that container’s config to see if it can indicate what is going wrong.

nobody · September 30, 2020, 10:25am

# ps aux | grep lxc # before stop
root      1074  0.0  0.0 1756088 7712 ?        Ssl  Sep03  16:07 /usr/bin/lxcfs /var/lib/lxcfs/
root      2610  0.0  0.0      0     0 ?        S    Sep03   0:00 [drbd_w_lxc1]
root      2617  0.0  0.0      0     0 ?        S    Sep03   0:00 [drbd_w_lxc2]
root      2649  0.0  0.0      0     0 ?        S    Sep03   0:00 [drbd_r_lxc1]
root      2654  0.0  0.0      0     0 ?        S    Sep03   0:00 [drbd_r_lxc2]
root      5645  0.0  0.0  14284  3840 ?        Ss   13:20   0:00 [lxc monitor] /var/lib/lxc test-1
root      6050  0.0  0.0  14288  5244 pts/4    S+   13:20   0:00 lxc-attach test-1
root      7105  0.0  0.0   6208   828 pts/0    S+   13:21   0:00 grep --color=auto lxc
root     32213  0.0  0.0      0     0 ?        S    Sep24   0:03 [drbd_a_lxc2]
root     32214  0.0  0.0      0     0 ?        I<   Sep24   0:00 [drbd_as_lxc2]
root     32215  0.0  0.0      0     0 ?        S    Sep24   0:03 [drbd_a_lxc1]
root     32216  0.0  0.0      0     0 ?        I<   Sep24   0:00 [drbd_as_lxc1]

# ps aux | grep lxc # after stop
root      1074  0.0  0.0 1756088 7712 ?        Ssl  Sep03  16:07 /usr/bin/lxcfs /var/lib/lxcfs/
root      2610  0.0  0.0      0     0 ?        S    Sep03   0:00 [drbd_w_lxc1]
root      2617  0.0  0.0      0     0 ?        S    Sep03   0:00 [drbd_w_lxc2]
root      2649  0.0  0.0      0     0 ?        S    Sep03   0:00 [drbd_r_lxc1]
root      2654  0.0  0.0      0     0 ?        S    Sep03   0:00 [drbd_r_lxc2]
root      8076  0.0  0.0   6208   888 pts/0    S+   13:22   0:00 grep --color=auto lxc
root     32213  0.0  0.0      0     0 ?        S    Sep24   0:03 [drbd_a_lxc2]
root     32214  0.0  0.0      0     0 ?        I<   Sep24   0:00 [drbd_as_lxc2]
root     32215  0.0  0.0      0     0 ?        S    Sep24   0:03 [drbd_a_lxc1]
root     32216  0.0  0.0      0     0 ?        I<   Sep24   0:00 [drbd_as_lxc1]

nope, there is nothing…

tomp · September 30, 2020, 10:39am

And if you manually remove that veth interface using sudo ip link delete <ifname> the ping stops replying?

nobody · September 30, 2020, 10:52am

Yep, that is correct.

tomp · September 30, 2020, 10:59am

Is this reproducible? Is it possible to get access to the server to investigate?

nobody · September 30, 2020, 11:02am

Yeah, i can reproduce it any time, but no, sorry @tomp , unfortunately i can’t give you access cuz it’s prod environment :’(

tomp · September 30, 2020, 11:05am

It may be a kernel bug. But @brauner may have a suggestion for further investigation.

nobody · September 30, 2020, 11:10am

it’s a simple debian 10.5 : 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

nobody · October 14, 2020, 8:05am

Also, i have found lxc.net.[i].script.down isn’t work. It just didn’t execute without errors. I tried to avoid this situation with simple script

#!/bin/bash

# arguments example: test-1 net down veth br51 vethW3CNUX

echo "arguments: $*" > /tmp/test
echo "environment:" >> /tmp/test
env | grep LXC >> /tmp/test

if [[ $6 =~ ^veth.*$ ]]; then
    echo "Trying to remove $6 from $5" >> /tmp/test
    /usr/sbin/brctl delif $5 $6
    echo "Trying to delete $6" >> /tmp/test
    /usr/sbin/ip link delete $6
fi

But after lxc-stop there is no any /tmp/test only on hosts with this trouble. I think this is a really big bug, isn’t it?
Maybe anyone can give temporarily solution? For me it is a big trouble.

tomp · October 14, 2020, 8:30am

Have you tried it without specifying any lxc.net.[i].script.down?

Also, suggest you enable:

lxc.log.syslog = daemon
lxc.log.level = 4

And then restart your host so that you can be sure the container has stopped and restarted and then see what is failing.

nobody · October 14, 2020, 10:29am

Hi there @tomp again!
There are no errors in logs, but

Oct 14 13:13:38 node-1 lxc-start[39357]: test-1: conf - conf.c:run_buffer:335 - Script exited with status 126
Oct 14 13:13:38 node-1 lxc-start[39357]: test-1: network - network.c:lxc_delete_network_priv:2572 - Failed to deconfigure network device
Oct 14 13:13:38 node-1 lxc-start[39357]: test-1: network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 397
Oct 14 13:13:38 node-1 lxc-start[39357]: test-1: conf - conf.c:run_buffer:335 - Script exited with status 126
Oct 14 13:13:38 node-1 lxc-start[39357]: test-1: network - network.c:lxc_delete_network_priv:2572 - Failed to deconfigure network device
Oct 14 13:13:38 node-1 lxc-start[39357]: test-1: network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth1" with index 399

in any cases - with lxc.net.[i].script.down or without.
I’ll try to be clarifying: as we figure out earlier, LXC for some reason do not delete network interfaces after lxc-stop , but it can be done with ip link delete vethNAME_OF_VETH , so i thought to write simple hook to do that with lxc.net.[i].script.down but was failed, cuz for some reason lxc.net.[i].script.down do not execute.

nobody · October 14, 2020, 10:39am

I think i found a ve-e-e-e-ery, ve-e-e-ery dirty hack. I wrote 2 hooks:

# cat hook_net_up

#!/bin/bash

LXC_BASEDIR=$(dirname $LXC_ROOTFS_PATH)
LXC_RUNDIR="$LXC_BASEDIR/run"

[[ -d $LXC_RUNDIR ]] || mkdir $LXC_RUNDIR

echo $6 > ${LXC_RUNDIR}/net-${5}

and

# cat hook_down

#!/bin/bash

LXC_BASEDIR=$(dirname $LXC_ROOTFS_PATH)
LXC_RUNDIR="$LXC_BASEDIR/run"

for netname in `ls -d $LXC_BASEDIR/run/net-*`; do
    ifname=`cat $netname`
    echo "Trying to kill $ifname" >> /tmp/hook_net_down_2
    /usr/sbin/ip link delete $ifname
done

piece of container config

lxc.net.0.script.up = hook_net_up
lxc.net.1.script.up = hook_net_up

lxc.hook.stop = hook_down

First hook gave us a couple of files in /var/lib/lxc/test-1/run/ contains name of veth, and second hook just read it and remove interfaces after container stops. For now this works for me, but it pretty shitty. There is i have to use lxc.hook.stop instead of lxc.net.[i].script.down but lxc.hook.stop know nothing about network, so i have to use lxc.net.0.script.up before to get veth names, store it somewhere and then remove it from system. Maybe someone has a better idea?
So, the question is still actual state: why lxc.net.[i].script.down isn’t execute and why container do not delete veth interfaces after stop? Should i create issue on github?

tomp · October 14, 2020, 10:40am

Have you rebooted your host since this issue started?

nobody · October 14, 2020, 10:42am

host (node-1 in example)? Not a container?
No, i didn’t reboot host since this issue started. I don’t see reasons why it can help…

tomp · October 14, 2020, 10:43am

Well its just a guess I’d like to rule out, as lxc is having trouble removing the interface for some reason, which might hint at a kernel issue.

tomp · October 14, 2020, 10:44am

Also what version of lxc is this?

nobody · October 14, 2020, 10:50am

# lxc-info --version
3.0.3

# dpkg -l | grep lxc
ii  liblxc1                              1:3.1.0+really3.0.3-8        amd64        Linux Containers userspace tools (library)
ii  lxc                                  1:3.1.0+really3.0.3-8        amd64        Linux Containers userspace tools
ii  lxc-templates                        3.0.4-0+deb10u1              amd64        Linux Containers userspace tools (templates)
ii  lxcfs                                3.0.3-2                      amd64        FUSE based filesystem for LXC