LXD installation and init breaks Debian 11 network

Can you add the static neighbor entry after deleting the automatic one.

Sure, how?

sudo ip neigh flush dev enp3s0
sudo ip neigh add 10.0.10.138 lladdr 00:0d:b9:57:6d:66 dev enp3s0

Thanks! But not working:

root@rtems:~# ip nei flush dev enp3s0
root@rtems:~# ip nei
root@rtems:~# ip nei add 10.0.10.138 lladdr 00:0d:b9:57:6d:66 dev enp3s0
root@rtems:~# ip ne
10.0.10.138 dev enp3s0 lladdr 00:0d:b9:57:6d:66 PERMANENT
root@rtems:~# ip ne
10.0.10.138 dev enp3s0 lladdr 00:0d:b9:57:6d:66 PERMANENT
root@rtems:~# ip r
default via 10.0.10.138 dev enp3s0 proto dhcp metric 100 
10.0.10.0/24 dev enp3s0 proto kernel scope link src 10.0.10.34 metric 100 
192.168.2.0/24 dev lxdbr0 proto kernel scope link src 192.168.2.1 linkdown 
root@rtems:~# ping 10.0.10.138
PING 10.0.10.138 (10.0.10.138) 56(84) bytes of data.
^C
--- 10.0.10.138 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5118ms
root@rtems:~# 

and sorry for delay, forum enforced me to wait 19 hours before posting thisā€¦

OK can you repeat the tcpdump test with that static neighbour entry in place and see what is being sent.

Iā€™m starting to think there is something very odd going on with your system, perhaps a kernel issue.

Also can you show output of bridge link show I would like to see what, if anything, is connected to your lxdbr0 bridge.

Thomas,

thanks a lot for your help with the issue. As the issue turns not to be trivial or simple user error on my side, I think itā€™s a time to look into broader picture of how the network configuration looks here.

There are two workstation connected to one SG250 switch:

  • rtems (tests done on it)

  • silence.

RTEMS configuration:

  • debian 11 installed. Connected using enp3s0 to switch VLAN 10 port. IP from DHCP is then 10.0.10.x and router is 10.0.10.138

SILENCE configuration:

  • ubuntu 20.04.x LTS. Connected using enp0s31f6 to VLAN 30 port and with enp3s0 switch to VLAN 10 port. Only enp0s31f6 is activated so gets its IP from DHCP server. enp3s0 is not activated e.g. no IP on it set.
  • there are some VirtualBox based VMs running with bridging set and hooked to enp3s0. This means those VMs will run on VLAN 10 hence gets 10.0.10.xx IP from DHCP server. Idea behing enp3s0 NOT having IP address is that once silence tries to communicate with some of those VM, communication runs thorough router: 10.0.30.5 (silence) -> 10.0.30.138/10.0.10.138 (router) -> 10.0.10.x (VM).
  • there is LXD running on silence. There are even few containers there and they are using lxdbr0.
    Letā€™s look into few commands output from silence to show more details:

karel@silence:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 4c:52:62:af:b3:e0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::4e52:62ff:feaf:b3e0/64 scope link
valid_lft forever preferred_lft forever
3: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 4c:52:62:b4:74:29 brd ff:ff:ff:ff:ff:ff
inet 10.0.30.5/24 brd 10.0.30.255 scope global dynamic enp0s31f6
valid_lft 23697sec preferred_lft 23697sec
inet6 fe80::4e52:62ff:feb4:7429/64 scope link
valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:65:33:57 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:65:33:57 brd ff:ff:ff:ff:ff:ff
6: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:06:3b:bb brd ff:ff:ff:ff:ff:ff
inet 10.118.144.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:5a8a:84a2:6478::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe06:3bbb/64 scope link
valid_lft forever preferred_lft forever
8: vethc39ffa7f@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 32:81:54:df:2d:64 brd ff:ff:ff:ff:ff:ff link-netnsid 0
10: veth82608c61@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 7e:07:c1:ef:70:26 brd ff:ff:ff:ff:ff:ff link-netnsid 1
12: veth14238bd8@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether ba:ac:4d:c4:dc:35 brd ff:ff:ff:ff:ff:ff link-netnsid 2
14: veth2cb93038@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether ea:19:a6:5f:7e:31 brd ff:ff:ff:ff:ff:ff link-netnsid 3
16: vethbff5db80@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 0e:ce:da:47:a4:aa brd ff:ff:ff:ff:ff:ff link-netnsid 4
18: veth72608cf3@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether d2:59:3b:9e:99:1e brd ff:ff:ff:ff:ff:ff link-netnsid 5
20: veth7ab64eef@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 86:41:9b:ab:e6:0d brd ff:ff:ff:ff:ff:ff link-netnsid 6
22: veth306891ac@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 82:4b:27:37:6d:7b brd ff:ff:ff:ff:ff:ff link-netnsid 7
24: veth793bac1d@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 06:13:5c:38:a0:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 8
26: vethc439baf9@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether b6:8d:ea:1e:e4:36 brd ff:ff:ff:ff:ff:ff link-netnsid 9

karel@silence:~$ ip r
default via 10.0.30.138 dev enp0s31f6 proto dhcp src 10.0.30.5 metric 100
10.0.30.0/24 dev enp0s31f6 proto kernel scope link src 10.0.30.5
10.0.30.138 dev enp0s31f6 proto dhcp scope link src 10.0.30.5 metric 100
10.118.144.0/24 dev lxdbr0 proto kernel scope link src 10.118.144.1
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

karel@silence:~$ ip n
10.118.144.55 dev lxdbr0 lladdr 00:16:3e:c4:77:f7 STALE
10.0.30.24 dev enp0s31f6 lladdr 84:76:16:00:32:75 STALE
10.118.144.224 dev lxdbr0 lladdr 00:16:3e:e3:26:6c STALE
10.0.30.253 dev enp0s31f6 lladdr 00:77:8d:da:c0:ce STALE
10.118.144.79 dev lxdbr0 lladdr 00:16:3e:75:c0:b7 STALE
10.0.30.21 dev enp0s31f6 lladdr 90:1b:0e:47:2d:c1 STALE
10.118.144.50 dev lxdbr0 lladdr 00:16:3e:71:f9:11 STALE
10.0.30.114 dev enp0s31f6 lladdr 0c:54:15:f0:fc:fd STALE
10.0.30.138 dev enp0s31f6 lladdr 00:0d:b9:57:6d:66 REACHABLE
10.0.30.20 dev enp0s31f6 lladdr 90:1b:0e:47:01:61 STALE
10.118.144.65 dev lxdbr0 lladdr 00:16:3e:ac:fb:86 STALE
10.118.144.27 dev lxdbr0 lladdr 00:16:3e:85:69:8c STALE
10.118.144.174 dev lxdbr0 lladdr 00:16:3e:b1:58:7e STALE
10.0.30.125 dev enp0s31f6 lladdr 08:00:27:82:f3:c7 STALE
10.118.144.17 dev lxdbr0 lladdr 00:16:3e:f7:2a:12 STALE
10.0.30.23 dev enp0s31f6 lladdr b8:27:eb:4e:74:d4 STALE
10.118.144.103 dev lxdbr0 lladdr 00:16:3e:65:1d:ea STALE
10.118.144.113 dev lxdbr0 lladdr 00:16:3e:07:59:db STALE
10.0.30.25 dev enp0s31f6 lladdr e4:1d:2d:6b:1b:8a STALE
fe80::216:3eff:fee3:266c dev lxdbr0 lladdr 00:16:3e:e3:26:6c STALE
fe80::216:3eff:fe75:c0b7 dev lxdbr0 lladdr 00:16:3e:75:c0:b7 STALE
fd42:5a8a:84a2:6478:216:3eff:fe65:1dea dev lxdbr0 lladdr 00:16:3e:65:1d:ea STALE
fe80::216:3eff:fef7:2a12 dev lxdbr0 lladdr 00:16:3e:f7:2a:12 STALE
fe80::216:3eff:fe71:f911 dev lxdbr0 lladdr 00:16:3e:71:f9:11 STALE
fd42:5a8a:84a2:6478:216:3eff:feac:fb86 dev lxdbr0 lladdr 00:16:3e:ac:fb:86 STALE
fe80::216:3eff:fe07:59db dev lxdbr0 lladdr 00:16:3e:07:59:db STALE
fe80::216:3eff:fe85:698c dev lxdbr0 lladdr 00:16:3e:85:69:8c STALE
fe80::216:3eff:fec4:77f7 dev lxdbr0 lladdr 00:16:3e:c4:77:f7 STALE
fe80::216:3eff:fe06:3bbb dev lxdbr0 lladdr 00:16:3e:06:3b:bb router STALE
fe80::216:3eff:feb1:587e dev lxdbr0 lladdr 00:16:3e:b1:58:7e STALE
fd42:5a8a:84a2:6478:216:3eff:fee3:266c dev lxdbr0 lladdr 00:16:3e:e3:26:6c STALE
fe80::216:3eff:fe65:1dea dev lxdbr0 lladdr 00:16:3e:65:1d:ea STALE
fe80::ac8d:c0ff:feca:2564 dev enp3s0 lladdr ae:8d:c0:ca:25:64 STALE
fd42:5a8a:84a2:6478:216:3eff:fe75:c0b7 dev lxdbr0 lladdr 00:16:3e:75:c0:b7 STALE
fe80::c210:b1ff:fe47:2a40 dev enp3s0 lladdr c0:10:b1:47:2a:40 STALE
fe80::3428:c8e5:8dba:8fae dev enp3s0 lladdr 00:2b:67:7d:58:4c STALE
fd42:5a8a:84a2:6478:216:3eff:fef7:2a12 dev lxdbr0 lladdr 00:16:3e:f7:2a:12 STALE
fe80::216:3eff:feac:fb86 dev lxdbr0 lladdr 00:16:3e:ac:fb:86 STALE
fe80::c54a:3a2c:b08b:3d86 dev enp3s0 lladdr 0c:54:15:f0:fc:fd STALE
fd42:5a8a:84a2:6478:216:3eff:fe71:f911 dev lxdbr0 lladdr 00:16:3e:71:f9:11 STALE
fe80::a00:27ff:fe82:f3c7 dev enp0s31f6 lladdr 08:00:27:82:f3:c7 STALE
fd42:5a8a:84a2:6478:216:3eff:fe07:59db dev lxdbr0 lladdr 00:16:3e:07:59:db STALE
fd42:5a8a:84a2:6478:216:3eff:fe85:698c dev lxdbr0 lladdr 00:16:3e:85:69:8c STALE
fd42:5a8a:84a2:6478:216:3eff:fec4:77f7 dev lxdbr0 lladdr 00:16:3e:c4:77:f7 STALE
fe80::72f3:5aff:fe24:a2a8 dev enp0s31f6 lladdr 70:f3:5a:24:a2:a8 STALE
fd42:5a8a:84a2:6478:216:3eff:feb1:587e dev lxdbr0 lladdr 00:16:3e:b1:58:7e STALE
karel@silence:~$

root@silence:~# lxc list
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| alpine312 | STOPPED | | | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| centos7 | RUNNING | 10.118.144.113 (eth0) | fd42:5a8a:84a2:6478:216:3eff:fe07:59db (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| centos8 | RUNNING | 10.118.144.55 (eth0) | fd42:5a8a:84a2:6478:216:3eff:fec4:77f7 (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| debian11 | RUNNING | 10.118.144.50 (eth0) | fd42:5a8a:84a2:6478:216:3eff:fe71:f911 (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| debian11i386 | RUNNING | 10.118.144.103 (eth0) | fd42:5a8a:84a2:6478:216:3eff:fe65:1dea (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| debian32 | RUNNING | 10.118.144.27 (eth0) | fd42:5a8a:84a2:6478:216:3eff:fe85:698c (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| kali | RUNNING | 10.118.144.224 (eth0) | fd42:5a8a:84a2:6478:216:3eff:fee3:266c (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| opensuse152 | RUNNING | 10.118.144.17 (eth0) | fd42:5a8a:84a2:6478:d92:6b3c:4495:421d (eth0) | CONTAINER | 0 |
| | | | fd42:5a8a:84a2:6478:d8ba:3aef:ccca:b427 (eth0) | | |
| | | | fd42:5a8a:84a2:6478:c8c2:7301:85ef:e6ff (eth0) | | |
| | | | fd42:5a8a:84a2:6478:c7:44a5:e421:be93 (eth0) | | |
| | | | fd42:5a8a:84a2:6478:94b4:834:3295:2d41 (eth0) | | |
| | | | fd42:5a8a:84a2:6478:9457:1430:7a2e:5c04 (eth0) | | |
| | | | fd42:5a8a:84a2:6478:6910:3b67:3aa1:b7a2 (eth0) | | |
| | | | fd42:5a8a:84a2:6478:216:3eff:fef7:2a12 (eth0) | | |
| | | | fd42:5a8a:84a2:6478:15d8:c3c4:2d17:8a20 (eth0) | | |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| ubuntu18lxc | RUNNING | 10.118.144.65 (eth0) | fd42:5a8a:84a2:6478:216:3eff:feac:fb86 (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| ubuntu20lxc | RUNNING | 10.118.144.174 (eth0) | fd42:5a8a:84a2:6478:216:3eff:feb1:587e (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
| ubuntu32 | RUNNING | 10.118.144.79 (eth0) | fd42:5a8a:84a2:6478:216:3eff:fe75:c0b7 (eth0) | CONTAINER | 0 |
Ā±-------------Ā±--------Ā±----------------------Ā±-----------------------------------------------Ā±----------Ā±----------+
root@silence:~#

Now, why Iā€™m writing all this is: can silence/ubuntu lxd affectā€™s rtems/debian 11 lxd?

Anyway, here is command output you asked for:

root@rtems:~# bridge link show
root@rtems:~# 

It looks like there is some problem with tcpdump and ping. Since in tcpdump I donā€™t see ping request yet kernel should route them to enp3s0:

root@rtems:~# tcpdump -i enp3s0 > /tmp/dump.txt 2>&1 &
[1] 1497
root@rtems:~# ping 10.0.10.138
PING 10.0.10.138 (10.0.10.138) 56(84) bytes of data.
^C
--- 10.0.10.138 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9193ms

root@rtems:~# kill %1
root@rtems:~# cat /tmp/dump.txt 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:24:16.855271 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id  8000.00:77:8d:da:c0:ce.8006, length 36
12:24:18.664729 IP 10.0.30.110 > 239.255.255.250: igmp v2 report 239.255.255.250
12:24:18.855289 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.00:77:8d:da:c0:ce.8006, length 36
12:24:18.858260 IP 10.0.10.34.47381 > 10.0.10.138.domain: 39899+ PTR? 250.255.255.239.in-addr.arpa. (46)

4 packets captured
81 packets received by filter
52 packets dropped by kernel
root@rtems:~# 

So itā€™s indeed a bit strange. Kernel is debian distributed:

root@rtems:~# uname -a
Linux rtems 5.10.0-6-amd64 #1 SMP Debian 5.10.28-1 (2021-04-09) x86_64 GNU/Linux
root@rtems:~# 

If you think this is kernel related and is not affected by other workstation running lxd too, then let me know and Iā€™ll try vanilla kernel from kernel.org and see what happen.

Thanks!
Karel
PS: Iā€™ll probably be again blocked by forum clever tacticts so sorry for future delays as this message was deleyed tooā€¦

You should use tcpdump with the -n flag to avoid delaying output trying to do RDNS looksups, please can you retest to confirm that wasnā€™t the issue not seeing icmp packets outbound.

Indeed, you are absolutely right!

root@rtems:~# tcpdump -n -i enp3s0 > /tmp/dump.txt 2>&1 &                                                                                                                                     
[1] 1771
root@rtems:~# ping 10.0.10.138
PING 10.0.10.138 (10.0.10.138) 56(84) bytes of data.
^C
--- 10.0.10.138 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4101ms

root@rtems:~# kill %1
root@rtems:~# 
[1]+  Done                    tcpdump -n -i enp3s0 > /tmp/dump.txt 2>&1

root@rtems:~# cat /tmp/dump.txt 
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp3s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:37:16.895693 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.00:77:8d:da:c0:ce.8006, length 36
13:37:18.252771 ARP, Request who-has 10.0.0.101 tell 10.0.0.101, length 46
13:37:18.895174 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.00:77:8d:da:c0:ce.8006, length 36
13:37:18.911090 IP 10.0.0.101.5353 > 224.0.0.251.5353: 0- [0q] 1/0/0 TXT "type=0" "version=1" "refresh-age-timeout=0" "priority=6" "refresh-flag=0" "root-mac-address=00:77:8d:da:c0:ce" "cos)
13:37:18.912818 IP 10.0.30.253.5353 > 224.0.0.251.5353: 0- [0q] 1/0/0 TXT "type=0" "version=1" "refresh-age-timeout=0" "priority=6" "refresh-flag=0" "root-mac-address=00:77:8d:da:c0:ce" "co)
13:37:18.935133 IP 10.0.30.253.5353 > 224.0.0.251.5353: 0- [0q] 1/0/0 TXT "type=0" "version=1" "refresh-age-timeout=0" "priority=6" "refresh-flag=0" "root-mac-address=00:77:8d:da:c0:ce" "co)
13:37:20.895158 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.00:77:8d:da:c0:ce.8006, length 36
13:37:21.060226 IP 10.0.10.34 > 10.0.10.138: ICMP echo request, id 41243, seq 1, length 64
13:37:21.604920 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:77:8d:da:c0:ce, length 300
13:37:22.089554 IP 10.0.10.34 > 10.0.10.138: ICMP echo request, id 41243, seq 2, length 64
13:37:22.783203 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 3c:57:31:4c:00:cd, length 300
13:37:22.895148 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.00:77:8d:da:c0:ce.8006, length 36
13:37:23.113547 IP 10.0.10.34 > 10.0.10.138: ICMP echo request, id 41243, seq 3, length 64
13:37:24.137557 IP 10.0.10.34 > 10.0.10.138: ICMP echo request, id 41243, seq 4, length 64
13:37:24.895539 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.00:77:8d:da:c0:ce.8006, length 36
13:37:25.161550 IP 10.0.10.34 > 10.0.10.138: ICMP echo request, id 41243, seq 5, length 64
13:37:25.402401 ARP, Request who-has 10.0.0.101 tell 10.0.0.101, length 46
13:37:26.895120 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id  8000.00:77:8d:da:c0:ce.8006, length 36
13:37:28.895111 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.00:77:8d:da:c0:ce.8006, length 36

19 packets captured
23 packets received by filter
0 packets dropped by kernel
root@rtems:~#

OK so now do the same on your routerā€™s port (if you can) and check the icmp packets are actually making it there. Then see if the replies are failing to go back based on ARP learning failure (it feels like this is an ARP learning issue).

the other thing that would be interesting to try is to remove LXD and restore the configuration as it was, then create a manual bridge using:

sudo snap remove lxd --purge # will delete all instances
ip link delete lxdbr0 # in case its still around
ip link add name lxdbr0 type bridge
ip a add </24 address from LXD generated lxdbr0> dev lxdbr0
ip link set lxdbr0 up

If that breakā€™s your networking then its an issue with bridging.

Thomas,

(1) no, ICMP requests are not reaching router. They are dropped somewhere

(2) yes, your code to establish lxdbr0 on my system behaves exactly like lxd after init, so yes, you have proved that there are some issues on Debian 11 kernel with bridgeingā€¦

Thanks a lot!
Karel

PS: will probably need to report to debian somehowā€¦

1 Like

Thomas,

somehow still bugs me how the switch may be ā€œbrokenā€ by broken debian kernel. So I investigated more and everything is as it should be since the switch is cisco and it has its own ā€œintelligenceā€.

I told you, the machine is hooked to VLAN10, port6:

switchdac0ce#show vlan
Created by: D-Default, S-Static, G-GVRP, R-Radius Assigned VLAN, V-Voice VLAN

Vlan       Name           Tagged Ports      UnTagged Ports      Created by    
---- ----------------- ------------------ ------------------ ---------------- 
 1           1                              gi1,gi8,Po1-4           DV        
 10       VLAN10            gi1,gi8             gi5-7               S         
 30       VLAN30            gi1,gi8             gi2-4               S         

Now, when I use your lxdbr0 creation code and create the bridge, guess what happen? Debian/Linux kernel bridge sends probably out some packets informing switch about ā€œhey, Iā€™m bridge hereā€ and switch immediately switches from VLAN10 untagged port mode to the tagged port mode:

switchdac0ce#show vlan
Created by: D-Default, S-Static, G-GVRP, R-Radius Assigned VLAN, V-Voice VLAN

Vlan       Name           Tagged Ports      UnTagged Ports      Created by    
---- ----------------- ------------------ ------------------ ---------------- 
 1           1                            gi1,gi6,gi8,Po1-4         DV        
 10       VLAN10          gi1,gi6,gi8          gi5,gi7              S         
 30       VLAN30          gi1,gi6,gi8           gi2-4               S         

switchdac0ce#

That means that my pings to 10.0.10.138 were probably untagged (as Iā€™ve not instructed debian kernel that it should tag them with VLAN10 tag) and obviously switch drops them immediately since port is in tagged mode hence requiring tagged packets only. Also since this configuration change is not saved the switch becomes normal after restart again.

Debian community clearly warns about bridge/switch interaction issue on: BridgeNetworkConnections - Debian Wiki

Note: If, after trying to use the bridge interface, you find your network link becomes dead and refuses to work again, it might be that the router/switch upstream is blocking ā€œunauthorized switchesā€ in the network (for example, by detecting BPDU packets). Youā€™ll have to change its configuration to explicitly allow the host machine/network port as a ā€œswitchā€.

and although this is not exactly this issue, it pushed me to the right direction. So this is the life.

Anyway, thanks a lot to you and to Stephane for dealing with this issue. Without your help I would not be able to move forward.

Thanks!
Karel

PS: The switch is SG250-08HP 8-Port Gigabit PoE Smart Switch, firmware 2.5.7.85

Thanks!

When creating a private internal-only bridge (like lxdbr0) and not connecting any of your external interfaces to it, the host should not send anything out of the external interfaces about it, and so the external switch should have no knowledge of your private bridge. If it is sending something out then this seems like a bug to me.

Thomas,

indeed, if lxdbr0 is purely internal and not connected (like in the case), then if this is true, then indeed it should be a bug somewhere OR does it to have something to do with the fact that rtems is also running lldp daemon?

[a little bit of testing and]

Indeed, when I switch off lldp on rtems Iā€™m both able to successfully create lxdbr0 by hand using your code above and Iā€™m able to lxd init and then obtain list of images (which was not running before). So network is working!

Conclusion: installation of lldb leads to internal switch advertisement to managed switch which immediately reconfigured vlan config of the machine attached port which leads to broken network connection.

Is it a bug or not bug? That is completely over my head. :slight_smile:

Thanks for help!
Karel

1 Like

Iā€™m not familiar with lldp, what do you mean by ā€œrunningā€ it? Is it a process? Why are you running it?

Link Layer Discovery Protocol, read ubuntu man page here: https://manpages.ubuntu.com/manpages/xenial/man8/lldpd.8.html

And topic here: https://en.wikipedia.org/wiki/Link_Layer_Discovery_Protocol

Ah I see, well there is a -I interfaces argument which suggests you may need to run it more restrictively to only advertise certain links?

https://manpages.ubuntu.com/manpages/focal/man8/lldpd.8.html

ā€œIt is also possible to blacklist an interface by suffixing it with an exclamation mark.ā€

1 Like

Yeah, so instead of bug(s) we end in classical scenario of some error between chair and keyboard. :slight_smile:

Thanks for helping me out of it!

1 Like

Also -C interfaces seem relevant, as perhaps adding a new interface causes the chassis ID to change?

1 Like