LXC Containers Have No Ip Address on ARM machines

What does iptables -v show?

$ sudo iptables -v
iptables v1.8.7 (nf_tables): no command specified
Try `iptables -h' or 'iptables --help' for more information.

OK can you install nftables package and then show sudo nft list ruleset.

It’s important to note I ran a sudo snap remove lxd --purge and reinstalled on --channel latest/stable. Cluster was recreated. Wanted to check if it was not a bug which was fixed afterwards.

$ sudo nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
		ct state related,established counter packets 3312107 bytes 2436472627 accept
		meta l4proto icmp counter packets 0 bytes 0 accept
		iifname "lo" counter packets 5204 bytes 361483 accept
		meta l4proto udp udp sport 123 counter packets 0 bytes 0 accept
		meta l4proto tcp ip saddr 10.0.0.0/16 tcp dport 8443 ct state new  counter packets 202 bytes 12120 accept
		meta l4proto tcp ct state new tcp dport 22 counter packets 1250 bytes 73616 accept
		counter packets 578 bytes 183208 reject with icmp type host-prohibited
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		counter packets 0 bytes 0 reject with icmp type host-prohibited
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
		ip daddr 169.254.0.0/16 counter packets 5543 bytes 426430 jump InstanceServices
	}

	chain InstanceServices {
		meta l4proto tcp ip daddr 169.254.0.2 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.2.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.4.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.5.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.2 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 53  counter packets 968 bytes 102862 accept
		meta l4proto tcp ip daddr 169.254.169.254 tcp dport 53  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.3 skuid 0 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.4 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.169.254 tcp dport 80  counter packets 4551 bytes 321744 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 67  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 69  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 123  counter packets 24 bytes 1824 accept
		meta l4proto tcp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject with tcp reset
		meta l4proto udp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject
	}
}
table ip6 filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
table inet lxd {
	chain pstrt.lxdfan0 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 240.0.0.0/8 ip daddr != 240.0.0.0/8 masquerade
	}

	chain fwd.lxdfan0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdfan0" accept
		ip version 4 iifname "lxdfan0" accept
	}

	chain in.lxdfan0 {
		type filter hook input priority filter; policy accept;
		iifname "lxdfan0" tcp dport 53 accept
		iifname "lxdfan0" udp dport 53 accept
		iifname "lxdfan0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		iifname "lxdfan0" udp dport 67 accept
	}

	chain out.lxdfan0 {
		type filter hook output priority filter; policy accept;
		oifname "lxdfan0" tcp sport 53 accept
		oifname "lxdfan0" udp sport 53 accept
		oifname "lxdfan0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		oifname "lxdfan0" udp sport 67 accept
	}
}

Rules are definitely there. But everything I’m doing is quite new to me. So I really don’t understand everything that’s going on.

Yep, so this comes up quite a bit, and is likely to get more common as more people (without knowing) move to nftables as they move to newer OSes.

See

and

But basically if there is cause to drop/reject a packet in any of the nftables tables then it will be dropped/rejected even if its allowed in another table.

So LXD adds its allow rules to nftables in its own lxd table, but then your default reject rules in INPUT and FORWARD chains of the main filter table take over and block the traffic.

So going back to what we said at the start, you need to configure your existing firewall traffic to allow the same rules (or more) of what LXD adds.

I would prefer to let lxd manage its own rules.

I’ll try to remove the rules for REJECT from both INPUT and FORWARD.

I could try to put them back as nft rules in the end. Do you have any tips on how to do this?

EDIT: Removing the REJECT rules from both INPUT and FORWARD solved the problem. Thank you so much!!

Still trying to figure out nftables.

So I have:

$ sudo iptables -v
iptables v1.8.7 (nf_tables): no command specified
Try `iptables -h' or 'iptables --help' for more information.
$ sudo systemctl status nftables.service 
○ nftables.service - nftables
     Loaded: loaded (/lib/systemd/system/nftables.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:nft(8)
             http://wiki.nftables.org

Is nftables actually running? Is Oracle’s Ubuntu image broken? Anyone has any idea what’s going on? If nftables.service is not running, are lxd firewall rules being considered at all?

If it possible to change lxd to configure iptables rules instead of nftables?

Btw, just to clear up why I still have firewall problems. Although I can connect to the internet from my containers, containers can’t see each other. There’s still something wrong.

The iptables command you have on your system is a wrapper to nftables, so it is just populating rules in the nftable’s filter table. LXD will be adding rules to the nftable’s lxd table.

However based on my earlier posts, if anything in the nftable’s filter table (added by iptables) causes packets from LXD instances to be dropped then they will be blocked irrespective of the rules added by LXD.

The nftables service on your system is likely just a startup script that would read native nftables rules from /etc/nftables.conf and load them into nftables. But this isn’t needed as iptables and LXD are doing it for you when they start.

Please show sudo nft list ruleset now?

I’ll add some extra information if it’s necessary.

Running this cluster:

ubuntu@arm1:~$ lxc cluster list
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME |           URL           |      ROLES      | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| amd1 | https://10.0.1.135:8443 | database-leader | x86_64       | default        |             | ONLINE | Fully operational |
|      |                         | database        |              |                |             |        |                   |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| amd2 | https://10.0.1.85:8443  | database        | x86_64       | default        |             | ONLINE | Fully operational |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| arm1 | https://10.0.1.233:8443 | database        | aarch64      | default        |             | ONLINE | Fully operational |
+------+-------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+

Since it’s a cluster and there are three firewalls, wanted to show all three are configured the same.

Ruleset on arm1:

ubuntu@arm1:~$ sudo nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
		ct state related,established counter packets 737576 bytes 596015542 accept
		meta l4proto icmp counter packets 0 bytes 0 accept
		iifname "lo" counter packets 1843 bytes 171475 accept
		meta l4proto udp udp sport 123 counter packets 0 bytes 0 accept
		meta l4proto tcp ip saddr 10.0.0.0/16 tcp dport 8443 ct state new  counter packets 9678 bytes 580656 accept
		meta l4proto tcp ct state new tcp dport 22 counter packets 510 bytes 28632 accept
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
		ip daddr 169.254.0.0/16 counter packets 6998 bytes 536994 jump InstanceServices
	}

	chain InstanceServices {
		meta l4proto tcp ip daddr 169.254.0.2 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.2.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.4.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.5.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.2 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 53  counter packets 1195 bytes 126820 accept
		meta l4proto tcp ip daddr 169.254.169.254 tcp dport 53  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.3 skuid 0 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.4 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.169.254 tcp dport 80  counter packets 5773 bytes 407666 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 67  counter packets 1 bytes 304 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 69  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 123  counter packets 29 bytes 2204 accept
		meta l4proto tcp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject with tcp reset
		meta l4proto udp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject
	}
}
table ip6 filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
table inet lxd {
	chain pstrt.lxdfan0 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 240.0.0.0/8 ip daddr != 240.0.0.0/8 masquerade
	}

	chain fwd.lxdfan0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdfan0" accept
		ip version 4 iifname "lxdfan0" accept
	}

	chain in.lxdfan0 {
		type filter hook input priority filter; policy accept;
		iifname "lxdfan0" tcp dport 53 accept
		iifname "lxdfan0" udp dport 53 accept
		iifname "lxdfan0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		iifname "lxdfan0" udp dport 67 accept
	}

	chain out.lxdfan0 {
		type filter hook output priority filter; policy accept;
		oifname "lxdfan0" tcp sport 53 accept
		oifname "lxdfan0" udp sport 53 accept
		oifname "lxdfan0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		oifname "lxdfan0" udp sport 67 accept
	}
}

Ruleset on amd1:

ubuntu@amd1:~$ sudo nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
		ct state related,established counter packets 3020529 bytes 736219684 accept
		meta l4proto icmp counter packets 0 bytes 0 accept
		iifname "lo" counter packets 6621 bytes 457728 accept
		meta l4proto udp udp sport 123 counter packets 0 bytes 0 accept
		meta l4proto tcp ip saddr 10.0.0.0/16 tcp dport 8443 ct state new  counter packets 50 bytes 3000 accept
		meta l4proto tcp ct state new tcp dport 22 counter packets 445 bytes 25340 accept
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
		ip daddr 169.254.0.0/16 counter packets 6994 bytes 536303 jump InstanceServices
	}

	chain InstanceServices {
		meta l4proto tcp ip daddr 169.254.0.2 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.2.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.4.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.5.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.2 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 53  counter packets 1238 bytes 131529 accept
		meta l4proto tcp ip daddr 169.254.169.254 tcp dport 53  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.3 skuid 0 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.4 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.169.254 tcp dport 80  counter packets 5726 bytes 402266 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 67  counter packets 1 bytes 304 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 69  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 123  counter packets 29 bytes 2204 accept
		meta l4proto tcp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject with tcp reset
		meta l4proto udp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject
	}
}
table ip6 filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
table inet lxd {
	chain pstrt.lxdfan0 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 240.0.0.0/8 ip daddr != 240.0.0.0/8 masquerade
	}

	chain fwd.lxdfan0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdfan0" accept
		ip version 4 iifname "lxdfan0" accept
	}

	chain in.lxdfan0 {
		type filter hook input priority filter; policy accept;
		iifname "lxdfan0" tcp dport 53 accept
		iifname "lxdfan0" udp dport 53 accept
		iifname "lxdfan0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		iifname "lxdfan0" udp dport 67 accept
	}

	chain out.lxdfan0 {
		type filter hook output priority filter; policy accept;
		oifname "lxdfan0" tcp sport 53 accept
		oifname "lxdfan0" udp sport 53 accept
		oifname "lxdfan0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		oifname "lxdfan0" udp sport 67 accept
	}
}

Ruleset on amd2:

ubuntu@amd2:~$ sudo nft list ruleset
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
		ct state related,established counter packets 690179 bytes 607718840 accept
		meta l4proto icmp counter packets 0 bytes 0 accept
		iifname "lo" counter packets 1801 bytes 168370 accept
		meta l4proto udp udp sport 123 counter packets 0 bytes 0 accept
		meta l4proto tcp ip saddr 10.0.0.0/16 tcp dport 8443 ct state new  counter packets 9690 bytes 581400 accept
		meta l4proto tcp ct state new tcp dport 22 counter packets 1242 bytes 72472 accept
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
		ip daddr 169.254.0.0/16 counter packets 7011 bytes 538496 jump InstanceServices
	}

	chain InstanceServices {
		meta l4proto tcp ip daddr 169.254.0.2 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.2.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.4.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.5.0/24 skuid 0 tcp dport 3260  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.2 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 53  counter packets 1254 bytes 133197 accept
		meta l4proto tcp ip daddr 169.254.169.254 tcp dport 53  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.3 skuid 0 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.0.4 tcp dport 80  counter packets 0 bytes 0 accept
		meta l4proto tcp ip daddr 169.254.169.254 tcp dport 80  counter packets 5727 bytes 402791 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 67  counter packets 1 bytes 304 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 69  counter packets 0 bytes 0 accept
		meta l4proto udp ip daddr 169.254.169.254 udp dport 123  counter packets 29 bytes 2204 accept
		meta l4proto tcp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject with tcp reset
		meta l4proto udp ip daddr 169.254.0.0/16   counter packets 0 bytes 0 reject
	}
}
table ip6 filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}

	chain FORWARD {
		type filter hook forward priority filter; policy accept;
	}

	chain OUTPUT {
		type filter hook output priority filter; policy accept;
	}
}
table inet lxd {
	chain pstrt.lxdfan0 {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 240.0.0.0/8 ip daddr != 240.0.0.0/8 masquerade
	}

	chain fwd.lxdfan0 {
		type filter hook forward priority filter; policy accept;
		ip version 4 oifname "lxdfan0" accept
		ip version 4 iifname "lxdfan0" accept
	}

	chain in.lxdfan0 {
		type filter hook input priority filter; policy accept;
		iifname "lxdfan0" tcp dport 53 accept
		iifname "lxdfan0" udp dport 53 accept
		iifname "lxdfan0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		iifname "lxdfan0" udp dport 67 accept
	}

	chain out.lxdfan0 {
		type filter hook output priority filter; policy accept;
		oifname "lxdfan0" tcp sport 53 accept
		oifname "lxdfan0" udp sport 53 accept
		oifname "lxdfan0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
		oifname "lxdfan0" udp sport 67 accept
	}
}

The issue I’m having:

ubuntu@arm1:~$ lxc list
+-------+---------+----------------------+------+-----------+-----------+----------+
| NAME  |  STATE  |         IPV4         | IPV6 |   TYPE    | SNAPSHOTS | LOCATION |
+-------+---------+----------------------+------+-----------+-----------+----------+
| ceph1 | RUNNING | 240.233.0.31 (eth0)  |      | CONTAINER | 0         | arm1     |
+-------+---------+----------------------+------+-----------+-----------+----------+
| ceph2 | RUNNING | 240.135.0.223 (eth0) |      | CONTAINER | 0         | amd1     |
+-------+---------+----------------------+------+-----------+-----------+----------+
| ceph3 | RUNNING | 240.85.0.61 (eth0)   |      | CONTAINER | 0         | amd2     |
+-------+---------+----------------------+------+-----------+-----------+----------+
| test  | RUNNING | 240.233.0.234 (eth0) |      | CONTAINER | 0         | arm1     |
+-------+---------+----------------------+------+-----------+-----------+----------+

All instaces were created with ubuntu:22.04. No changes have been made to them.

ubuntu@arm1:~$ lxc exec ceph1 ping 240.135.0.223
PING 240.135.0.223 (240.135.0.223) 56(84) bytes of data.
From 240.233.0.31 icmp_seq=1 Destination Host Unreachable
From 240.233.0.31 icmp_seq=2 Destination Host Unreachable
From 240.233.0.31 icmp_seq=3 Destination Host Unreachable
^C
--- 240.135.0.223 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3060ms
pipe 4
ubuntu@arm1:~$ lxc exec ceph1 ping ceph2
ping: ceph2: Temporary failure in name resolution

On the same host:

ubuntu@arm1:~$ lxc exec ceph1 ping 240.233.0.234
PING 240.233.0.234 (240.233.0.234) 56(84) bytes of data.
64 bytes from 240.233.0.234: icmp_seq=1 ttl=64 time=0.087 ms
64 bytes from 240.233.0.234: icmp_seq=2 ttl=64 time=0.059 ms
64 bytes from 240.233.0.234: icmp_seq=3 ttl=64 time=0.056 ms
64 bytes from 240.233.0.234: icmp_seq=4 ttl=64 time=0.055 ms
^C
--- 240.233.0.234 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3050ms
rtt min/avg/max/mdev = 0.055/0.064/0.087/0.013 ms
ubuntu@arm1:~$ lxc exec ceph1 ping test
PING test.lxd (240.233.0.234) 56(84) bytes of data.
64 bytes from test.lxd (240.233.0.234): icmp_seq=1 ttl=64 time=0.034 ms
64 bytes from test.lxd (240.233.0.234): icmp_seq=2 ttl=64 time=0.055 ms
64 bytes from test.lxd (240.233.0.234): icmp_seq=3 ttl=64 time=0.059 ms
64 bytes from test.lxd (240.233.0.234): icmp_seq=4 ttl=64 time=0.055 ms
^C
--- test.lxd ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3051ms
rtt min/avg/max/mdev = 0.034/0.050/0.059/0.009 ms

OK so DHCP is working now, and ping between instances on same host is working, but the fan overlay isn’t.

Can you get a ping going between containers on different LXD hosts and then in a separate window run sudo tcpdump -i <external interface> -nn and see if you can see the encapsulated packets heading out to the other server?

Is this it?

ubuntu@arm1:~$ lxc network list
+---------+----------+---------+------+------+-------------+---------+---------+
|  NAME   |   TYPE   | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |  STATE  |
+---------+----------+---------+------+------+-------------+---------+---------+
| enp0s3  | physical | NO      |      |      |             | 0       |         |
+---------+----------+---------+------+------+-------------+---------+---------+
| lxdfan0 | bridge   | YES     |      |      |             | 5       | CREATED |
+---------+----------+---------+------+------+-------------+---------+---------+

On one window:

ubuntu@arm1:~$ lxc exec ceph1 ping 240.135.0.223
PING 240.135.0.223 (240.135.0.223) 56(84) bytes of data.
From 240.233.0.31 icmp_seq=1 Destination Host Unreachable
From 240.233.0.31 icmp_seq=2 Destination Host Unreachable
From 240.233.0.31 icmp_seq=3 Destination Host Unreachable
From 240.233.0.31 icmp_seq=4 Destination Host Unreachable
From 240.233.0.31 icmp_seq=5 Destination Host Unreachable
From 240.233.0.31 icmp_seq=6 Destination Host Unreachable
From 240.233.0.31 icmp_seq=7 Destination Host Unreachable
From 240.233.0.31 icmp_seq=8 Destination Host Unreachable
From 240.233.0.31 icmp_seq=9 Destination Host Unreachable
From 240.233.0.31 icmp_seq=10 Destination Host Unreachable
From 240.233.0.31 icmp_seq=11 Destination Host Unreachable
From 240.233.0.31 icmp_seq=12 Destination Host Unreachable
From 240.233.0.31 icmp_seq=13 Destination Host Unreachable
From 240.233.0.31 icmp_seq=17 Destination Host Unreachable

On the other:

ubuntu@arm1:~$ sudo tcpdump -i enp0s3 -nn
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:12:22.618773 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 4062748733:4062748929, ack 4133986014, win 469, options [nop,nop,TS val 3452184787 ecr 2428726898], length 196
10:12:22.663871 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 196, win 501, options [nop,nop,TS val 2428726949 ecr 3452184787], length 0
10:12:22.696560 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 196:568, ack 1, win 469, options [nop,nop,TS val 3452184865 ecr 2428726949], length 372
10:12:22.741149 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 568, win 501, options [nop,nop,TS val 2428727026 ecr 3452184865], length 0
10:12:22.744897 IP 10.0.1.135.8443 > 10.0.1.233.35174: Flags [.], ack 1345206500, win 12293, options [nop,nop,TS val 3345033004 ecr 416088970], length 0
10:12:22.744919 IP 10.0.1.233.35174 > 10.0.1.135.8443: Flags [.], ack 1, win 16502, options [nop,nop,TS val 416092042 ecr 3345030033], length 0
10:12:22.798012 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 568:916, ack 1, win 469, options [nop,nop,TS val 3452184967 ecr 2428727026], length 348
10:12:22.798088 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 916:1256, ack 1, win 469, options [nop,nop,TS val 3452184967 ecr 2428727026], length 340
10:12:22.813928 IP 10.0.1.233.40105 > 10.0.1.135.8472: OTV, flags [I] (0x08), overlay 0, instance 15728640
ARP, Request who-has 240.135.0.223 tell 240.233.0.31, length 28
10:12:22.814388 IP 10.0.1.233.22 > 177.37.251.57.28599: Flags [P.], seq 1986647938:1986648094, ack 774653113, win 469, options [nop,nop,TS val 3452184983 ecr 2428724072], length 156
10:12:22.842457 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 916, win 501, options [nop,nop,TS val 2428727128 ecr 3452184967], length 0
10:12:22.842458 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 1256, win 501, options [nop,nop,TS val 2428727128 ecr 3452184967], length 0
10:12:22.845856 IP 10.0.1.233.44294 > 10.0.1.85.8443: Flags [.], ack 1240556396, win 2652, options [nop,nop,TS val 2463504953 ecr 4289511227], length 0
10:12:22.845856 IP 10.0.1.233.35174 > 10.0.1.135.8443: Flags [.], ack 1, win 16502, options [nop,nop,TS val 416092143 ecr 3345030033], length 0
10:12:22.846270 IP 10.0.1.135.8443 > 10.0.1.233.35174: Flags [.], ack 1, win 12293, options [nop,nop,TS val 3345033105 ecr 416092042], length 0
10:12:22.846275 IP 10.0.1.85.8443 > 10.0.1.233.44294: Flags [.], ack 1, win 482, options [nop,nop,TS val 4289514474 ecr 2463501706], length 0
10:12:22.858648 IP 177.37.251.57.28599 > 10.0.1.233.22: Flags [.], ack 156, win 501, options [nop,nop,TS val 2428727145 ecr 3452184983], length 0
10:12:22.897963 IP 10.0.1.135.8443 > 10.0.1.233.35184: Flags [P.], seq 4044149233:4044149266, ack 1180710104, win 480, options [nop,nop,TS val 3345033157 ecr 416088283], length 33
10:12:22.898001 IP 10.0.1.233.35184 > 10.0.1.135.8443: Flags [.], ack 33, win 443, options [nop,nop,TS val 416092195 ecr 3345033157], length 0
10:12:22.898194 IP 10.0.1.233.35184 > 10.0.1.135.8443: Flags [P.], seq 1:38, ack 33, win 443, options [nop,nop,TS val 416092195 ecr 3345033157], length 37
10:12:22.898477 IP 10.0.1.135.8443 > 10.0.1.233.35184: Flags [.], ack 38, win 480, options [nop,nop,TS val 3345033157 ecr 416092195], length 0
10:12:22.899892 IP 10.0.1.135.42628 > 10.0.1.233.8443: Flags [P.], seq 3185255445:3185255531, ack 3769648991, win 483, options [nop,nop,TS val 3345033159 ecr 416091696], length 86
10:12:22.899902 IP 10.0.1.233.8443 > 10.0.1.135.42628: Flags [.], ack 86, win 3554, options [nop,nop,TS val 416092197 ecr 3345033159], length 0
10:12:22.900105 IP 10.0.1.233.35168 > 10.0.1.135.8443: Flags [P.], seq 3650419252:3650419314, ack 1420952192, win 483, options [nop,nop,TS val 416092197 ecr 3345032659], length 62
10:12:22.900401 IP 10.0.1.135.8443 > 10.0.1.233.35168: Flags [.], ack 62, win 442, options [nop,nop,TS val 3345033159 ecr 416092197], length 0
10:12:22.902058 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 1256:1620, ack 1, win 469, options [nop,nop,TS val 3452185071 ecr 2428727128], length 364
10:12:22.902709 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 1620:1768, ack 1, win 469, options [nop,nop,TS val 3452185071 ecr 2428727128], length 148
10:12:22.902879 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 1768:2052, ack 1, win 469, options [nop,nop,TS val 3452185072 ecr 2428727128], length 284
10:12:22.902920 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 2052:2384, ack 1, win 469, options [nop,nop,TS val 3452185072 ecr 2428727128], length 332
10:12:22.902949 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 2384:2716, ack 1, win 469, options [nop,nop,TS val 3452185072 ecr 2428727128], length 332
10:12:22.903134 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 2716:4464, ack 1, win 469, options [nop,nop,TS val 3452185072 ecr 2428727128], length 1748
10:12:22.946837 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 1620, win 501, options [nop,nop,TS val 2428727232 ecr 3452185071], length 0
10:12:22.947792 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 1768, win 501, options [nop,nop,TS val 2428727233 ecr 3452185071], length 0
10:12:22.947792 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 2052, win 501, options [nop,nop,TS val 2428727233 ecr 3452185072], length 0
10:12:22.947834 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 2384, win 499, options [nop,nop,TS val 2428727233 ecr 3452185072], length 0
10:12:22.947877 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 2716, win 497, options [nop,nop,TS val 2428727233 ecr 3452185072], length 0
10:12:22.947877 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 4464, win 499, options [nop,nop,TS val 2428727233 ecr 3452185072], length 0
10:12:23.006290 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 4464:5924, ack 1, win 469, options [nop,nop,TS val 3452185175 ecr 2428727233], length 1460
10:12:23.006380 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 5924:6408, ack 1, win 469, options [nop,nop,TS val 3452185175 ecr 2428727233], length 484
10:12:23.012603 IP 169.254.169.254.80 > 10.0.1.233.48620: Flags [F.], seq 3057849076, ack 1672661540, win 111, options [nop,nop,TS val 323185418 ecr 1257919166], length 0
10:12:23.012689 IP 10.0.1.233.48620 > 169.254.169.254.80: Flags [F.], seq 1, ack 1, win 468, options [nop,nop,TS val 1257984164 ecr 323185418], length 0
10:12:23.012841 IP 169.254.169.254.80 > 10.0.1.233.48620: Flags [.], ack 2, win 111, options [nop,nop,TS val 323185418 ecr 1257984164], length 0
10:12:23.053156 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 5924, win 501, options [nop,nop,TS val 2428727336 ecr 3452185175], length 0
10:12:23.053173 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 6408, win 498, options [nop,nop,TS val 2428727336 ecr 3452185175], length 0
10:12:23.110057 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 6408:6772, ack 1, win 469, options [nop,nop,TS val 3452185279 ecr 2428727336], length 364
10:12:23.110108 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 6772:7576, ack 1, win 469, options [nop,nop,TS val 3452185279 ecr 2428727336], length 804
10:12:23.156565 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 6772, win 501, options [nop,nop,TS val 2428727440 ecr 3452185279], length 0
10:12:23.156565 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 7576, win 501, options [nop,nop,TS val 2428727440 ecr 3452185279], length 0
10:12:23.202321 IP 140.204.12.251.443 > 10.0.1.233.32914: Flags [P.], seq 1546823992:1546824023, ack 1087737390, win 854, options [nop,nop,TS val 1235010892 ecr 3199565063], length 31
10:12:23.202322 IP 140.204.12.251.443 > 10.0.1.233.32914: Flags [F.], seq 31, ack 1, win 854, options [nop,nop,TS val 1235010892 ecr 3199565063], length 0
10:12:23.202378 IP 10.0.1.233.32914 > 140.204.12.251.443: Flags [.], ack 31, win 460, options [nop,nop,TS val 3199570101 ecr 1235010892], length 0
10:12:23.202502 IP 10.0.1.233.32914 > 140.204.12.251.443: Flags [P.], seq 1:32, ack 32, win 460, options [nop,nop,TS val 3199570101 ecr 1235010892], length 31
10:12:23.202545 IP 10.0.1.233.32914 > 140.204.12.251.443: Flags [F.], seq 32, ack 32, win 460, options [nop,nop,TS val 3199570101 ecr 1235010892], length 0
10:12:23.202799 IP 140.204.12.251.443 > 10.0.1.233.32914: Flags [R], seq 1546824024, win 0, length 0
10:12:23.214225 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 7576:9044, ack 1, win 469, options [nop,nop,TS val 3452185383 ecr 2428727440], length 1468
10:12:23.214435 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 9044:9184, ack 1, win 469, options [nop,nop,TS val 3452185383 ecr 2428727440], length 140
10:12:23.258882 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 9044, win 501, options [nop,nop,TS val 2428727544 ecr 3452185383], length 0
10:12:23.258883 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 9184, win 501, options [nop,nop,TS val 2428727544 ecr 3452185383], length 0
10:12:23.318073 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 9184:9844, ack 1, win 469, options [nop,nop,TS val 3452185487 ecr 2428727544], length 660
10:12:23.357870 IP 10.0.1.233.35178 > 10.0.1.135.8443: Flags [.], ack 4151969888, win 4750, options [nop,nop,TS val 416092655 ecr 3345030528], length 0
10:12:23.358284 IP 10.0.1.135.8443 > 10.0.1.233.35178: Flags [.], ack 1, win 482, options [nop,nop,TS val 3345033617 ecr 416089565], length 0
10:12:23.364195 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 9844, win 501, options [nop,nop,TS val 2428727649 ecr 3452185487], length 0
10:12:23.401408 IP 10.0.1.135.42628 > 10.0.1.233.8443: Flags [P.], seq 86:172, ack 1, win 483, options [nop,nop,TS val 3345033660 ecr 416092197], length 86
10:12:23.401427 IP 10.0.1.233.8443 > 10.0.1.135.42628: Flags [.], ack 172, win 3554, options [nop,nop,TS val 416092698 ecr 3345033660], length 0
10:12:23.401737 IP 10.0.1.233.35168 > 10.0.1.135.8443: Flags [P.], seq 62:124, ack 1, win 483, options [nop,nop,TS val 416092698 ecr 3345033159], length 62
10:12:23.402049 IP 10.0.1.135.8443 > 10.0.1.233.35168: Flags [.], ack 124, win 442, options [nop,nop,TS val 3345033661 ecr 416092698], length 0
10:12:23.422096 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 9844:10648, ack 1, win 469, options [nop,nop,TS val 3452185591 ecr 2428727649], length 804
10:12:23.422177 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 10648:10828, ack 1, win 469, options [nop,nop,TS val 3452185591 ecr 2428727649], length 180
10:12:23.422272 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 10828:11168, ack 1, win 469, options [nop,nop,TS val 3452185591 ecr 2428727649], length 340
10:12:23.467535 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 11168, win 501, options [nop,nop,TS val 2428727752 ecr 3452185591], length 0
10:12:23.526113 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 11168:11852, ack 1, win 469, options [nop,nop,TS val 3452185695 ecr 2428727752], length 684
10:12:23.570929 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 11852, win 501, options [nop,nop,TS val 2428727856 ecr 3452185695], length 0
10:12:23.630033 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 11852:12200, ack 1, win 469, options [nop,nop,TS val 3452185799 ecr 2428727856], length 348
10:12:23.675243 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 12200, win 501, options [nop,nop,TS val 2428727960 ecr 3452185799], length 0
10:12:23.734054 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 12200:12548, ack 1, win 469, options [nop,nop,TS val 3452185903 ecr 2428727960], length 348
10:12:23.778617 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 12548, win 501, options [nop,nop,TS val 2428728064 ecr 3452185903], length 0
10:12:23.837908 IP 10.0.1.233.40105 > 10.0.1.135.8472: OTV, flags [I] (0x08), overlay 0, instance 15728640
ARP, Request who-has 240.135.0.223 tell 240.233.0.31, length 28
10:12:23.838193 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 12548:12896, ack 1, win 469, options [nop,nop,TS val 3452186007 ecr 2428728064], length 348
10:12:23.883929 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 12896, win 501, options [nop,nop,TS val 2428728168 ecr 3452186007], length 0
10:12:23.901939 IP 10.0.1.135.42628 > 10.0.1.233.8443: Flags [P.], seq 172:258, ack 1, win 483, options [nop,nop,TS val 3345034161 ecr 416092698], length 86
10:12:23.901960 IP 10.0.1.233.8443 > 10.0.1.135.42628: Flags [.], ack 258, win 3554, options [nop,nop,TS val 416093199 ecr 3345034161], length 0
10:12:23.902190 IP 10.0.1.233.35168 > 10.0.1.135.8443: Flags [P.], seq 124:186, ack 1, win 483, options [nop,nop,TS val 416093199 ecr 3345033661], length 62
10:12:23.902517 IP 10.0.1.135.8443 > 10.0.1.233.35168: Flags [.], ack 186, win 442, options [nop,nop,TS val 3345034161 ecr 416093199], length 0
10:12:23.942152 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 12896:13420, ack 1, win 469, options [nop,nop,TS val 3452186111 ecr 2428728168], length 524
10:12:23.942355 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 13420:14064, ack 1, win 469, options [nop,nop,TS val 3452186111 ecr 2428728168], length 644
10:12:23.988299 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 14064, win 501, options [nop,nop,TS val 2428728273 ecr 3452186111], length 0
10:12:24.029856 IP 10.0.1.233.44296 > 10.0.1.85.8443: Flags [.], ack 1329496515, win 2652, options [nop,nop,TS val 2463506137 ecr 4289512640], length 0
10:12:24.030300 IP 10.0.1.85.8443 > 10.0.1.233.44296: Flags [.], ack 1, win 480, options [nop,nop,TS val 4289515658 ecr 2463503119], length 0
10:12:24.046087 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 14064:14876, ack 1, win 469, options [nop,nop,TS val 3452186215 ecr 2428728273], length 812
10:12:24.091624 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 14876, win 501, options [nop,nop,TS val 2428728376 ecr 3452186215], length 0
10:12:24.150047 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 14876:15224, ack 1, win 469, options [nop,nop,TS val 3452186319 ecr 2428728376], length 348
10:12:24.195963 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 15224, win 501, options [nop,nop,TS val 2428728480 ecr 3452186319], length 0
10:12:24.254019 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 15224:15572, ack 1, win 469, options [nop,nop,TS val 3452186423 ecr 2428728480], length 348
10:12:24.299335 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 15572, win 501, options [nop,nop,TS val 2428728584 ecr 3452186423], length 0
10:12:24.358100 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 15572:15920, ack 1, win 469, options [nop,nop,TS val 3452186527 ecr 2428728584], length 348
10:12:24.401152 IP 10.0.1.135.8443 > 10.0.1.233.35178: Flags [P.], seq 1:229, ack 1, win 482, options [nop,nop,TS val 3345034660 ecr 416089565], length 228
10:12:24.401152 IP 10.0.1.135.8443 > 10.0.1.233.35178: Flags [P.], seq 229:540, ack 1, win 482, options [nop,nop,TS val 3345034660 ecr 416089565], length 311
10:12:24.401323 IP 10.0.1.135.8443 > 10.0.1.233.35184: Flags [P.], seq 33:261, ack 38, win 480, options [nop,nop,TS val 3345034660 ecr 416092195], length 228
10:12:24.401361 IP 10.0.1.233.35178 > 10.0.1.135.8443: Flags [.], ack 540, win 4750, options [nop,nop,TS val 416093698 ecr 3345034660], length 0
10:12:24.401388 IP 10.0.1.135.8443 > 10.0.1.233.35184: Flags [P.], seq 261:572, ack 38, win 480, options [nop,nop,TS val 3345034660 ecr 416092195], length 311
10:12:24.401543 IP 10.0.1.233.35184 > 10.0.1.135.8443: Flags [.], ack 572, win 443, options [nop,nop,TS val 416093698 ecr 3345034660], length 0
10:12:24.402066 IP 10.0.1.135.42628 > 10.0.1.233.8443: Flags [P.], seq 258:344, ack 1, win 483, options [nop,nop,TS val 3345034661 ecr 416093199], length 86
10:12:24.402083 IP 10.0.1.233.8443 > 10.0.1.135.42628: Flags [.], ack 344, win 3554, options [nop,nop,TS val 416093699 ecr 3345034661], length 0
10:12:24.402224 IP 10.0.1.233.35168 > 10.0.1.135.8443: Flags [P.], seq 186:248, ack 1, win 483, options [nop,nop,TS val 416093699 ecr 3345034161], length 62
10:12:24.402536 IP 10.0.1.135.8443 > 10.0.1.233.35168: Flags [.], ack 248, win 442, options [nop,nop,TS val 3345034661 ecr 416093699], length 0
10:12:24.403647 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 15920, win 501, options [nop,nop,TS val 2428728688 ecr 3452186527], length 0
10:12:24.462142 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 15920:17052, ack 1, win 469, options [nop,nop,TS val 3452186631 ecr 2428728688], length 1132
10:12:24.462212 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 17052:17248, ack 1, win 469, options [nop,nop,TS val 3452186631 ecr 2428728688], length 196
10:12:24.462236 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 17248:17884, ack 1, win 469, options [nop,nop,TS val 3452186631 ecr 2428728688], length 636
10:12:24.508030 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 17884, win 501, options [nop,nop,TS val 2428728793 ecr 3452186631], length 0
10:12:24.566091 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 17884:18568, ack 1, win 469, options [nop,nop,TS val 3452186735 ecr 2428728793], length 684
10:12:24.610346 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 18568, win 501, options [nop,nop,TS val 2428728896 ecr 3452186735], length 0
10:12:24.637856 IP 10.0.1.233.8443 > 10.0.1.85.40082: Flags [.], ack 1741525298, win 481, options [nop,nop,TS val 2463506745 ecr 4289513194], length 0
10:12:24.637857 IP 10.0.1.233.44290 > 10.0.1.85.8443: Flags [.], ack 513779282, win 483, options [nop,nop,TS val 2463506745 ecr 4289513194], length 0
10:12:24.638313 IP 10.0.1.85.8443 > 10.0.1.233.44290: Flags [.], ack 1, win 481, options [nop,nop,TS val 4289516266 ecr 2463504468], length 0
10:12:24.638337 IP 10.0.1.85.40082 > 10.0.1.233.8443: Flags [.], ack 1, win 483, options [nop,nop,TS val 4289516266 ecr 2463503700], length 0
10:12:24.665087 IP 10.0.1.85.40082 > 10.0.1.233.8443: Flags [.], ack 1, win 483, options [nop,nop,TS val 4289516293 ecr 2463503700], length 0
10:12:24.665106 IP 10.0.1.233.8443 > 10.0.1.85.40082: Flags [.], ack 1, win 481, options [nop,nop,TS val 2463506772 ecr 4289516266], length 0
10:12:24.670070 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 18568:18916, ack 1, win 469, options [nop,nop,TS val 3452186839 ecr 2428728896], length 348
10:12:24.670125 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 18916:19256, ack 1, win 469, options [nop,nop,TS val 3452186839 ecr 2428728896], length 340
10:12:24.670159 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 19256:19580, ack 1, win 469, options [nop,nop,TS val 3452186839 ecr 2428728896], length 324
10:12:24.670299 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 19580:19904, ack 1, win 469, options [nop,nop,TS val 3452186839 ecr 2428728896], length 324
10:12:24.715757 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 19904, win 501, options [nop,nop,TS val 2428729000 ecr 3452186839], length 0
10:12:24.774058 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 19904:20436, ack 1, win 469, options [nop,nop,TS val 3452186943 ecr 2428729000], length 532
10:12:24.774140 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 20436:20784, ack 1, win 469, options [nop,nop,TS val 3452186943 ecr 2428729000], length 348
10:12:24.819078 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 20784, win 501, options [nop,nop,TS val 2428729104 ecr 3452186943], length 0
10:12:24.861904 IP 10.0.1.233.40105 > 10.0.1.135.8472: OTV, flags [I] (0x08), overlay 0, instance 15728640
ARP, Request who-has 240.135.0.223 tell 240.233.0.31, length 28
10:12:24.878125 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 20784:21476, ack 1, win 469, options [nop,nop,TS val 3452187047 ecr 2428729104], length 692
10:12:24.903446 IP 10.0.1.135.42628 > 10.0.1.233.8443: Flags [P.], seq 344:430, ack 1, win 483, options [nop,nop,TS val 3345035162 ecr 416093699], length 86
10:12:24.903476 IP 10.0.1.233.8443 > 10.0.1.135.42628: Flags [.], ack 430, win 3554, options [nop,nop,TS val 416094200 ecr 3345035162], length 0
10:12:24.903738 IP 10.0.1.233.35168 > 10.0.1.135.8443: Flags [P.], seq 248:310, ack 1, win 483, options [nop,nop,TS val 416094200 ecr 3345034661], length 62
10:12:24.904063 IP 10.0.1.135.8443 > 10.0.1.233.35168: Flags [.], ack 310, win 442, options [nop,nop,TS val 3345035163 ecr 416094200], length 0
10:12:24.924422 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 21476, win 501, options [nop,nop,TS val 2428729208 ecr 3452187047], length 0
10:12:24.982080 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 21476:21984, ack 1, win 469, options [nop,nop,TS val 3452187151 ecr 2428729208], length 508
10:12:24.982131 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 21984:22512, ack 1, win 469, options [nop,nop,TS val 3452187151 ecr 2428729208], length 528
10:12:25.027827 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 22512, win 501, options [nop,nop,TS val 2428729312 ecr 3452187151], length 0
10:12:25.086133 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 22512:23028, ack 1, win 469, options [nop,nop,TS val 3452187255 ecr 2428729312], length 516
10:12:25.131128 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 23028, win 501, options [nop,nop,TS val 2428729416 ecr 3452187255], length 0
10:12:25.177090 IP 10.0.1.85.40086 > 10.0.1.233.8443: Flags [.], ack 2594142123, win 443, options [nop,nop,TS val 4289516805 ecr 2463504052], length 0
10:12:25.177108 IP 10.0.1.233.8443 > 10.0.1.85.40086: Flags [.], ack 1, win 482, options [nop,nop,TS val 2463507284 ecr 4289510559], length 0
10:12:25.190040 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 23028:23672, ack 1, win 469, options [nop,nop,TS val 3452187359 ecr 2428729416], length 644
10:12:25.235451 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 23672, win 501, options [nop,nop,TS val 2428729520 ecr 3452187359], length 0
10:12:25.294053 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 23672:24020, ack 1, win 469, options [nop,nop,TS val 3452187463 ecr 2428729520], length 348
10:12:25.308981 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [S], seq 3668981669, win 62720, options [mss 8960,sackOK,TS val 3345035568 ecr 0,nop,wscale 7], length 0
10:12:25.309015 IP 10.0.1.135.8443 > 10.0.1.233.35184: Flags [P.], seq 572:787, ack 38, win 480, options [nop,nop,TS val 3345035568 ecr 416093698], length 215
10:12:25.309064 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [S.], seq 2806070109, ack 3668981670, win 62636, options [mss 8960,sackOK,TS val 416094606 ecr 3345035568,nop,wscale 7], length 0
10:12:25.309125 IP 10.0.1.135.8443 > 10.0.1.233.35178: Flags [P.], seq 540:755, ack 1, win 482, options [nop,nop,TS val 3345035568 ecr 416093698], length 215
10:12:25.309408 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [.], ack 1, win 490, options [nop,nop,TS val 3345035568 ecr 416094606], length 0
10:12:25.309668 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [P.], seq 1:251, ack 1, win 490, options [nop,nop,TS val 3345035568 ecr 416094606], length 250
10:12:25.309692 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [.], ack 251, win 488, options [nop,nop,TS val 416094606 ecr 3345035568], length 0
10:12:25.311886 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [P.], seq 1:970, ack 251, win 488, options [nop,nop,TS val 416094609 ecr 3345035568], length 969
10:12:25.312228 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [.], ack 970, win 483, options [nop,nop,TS val 3345035571 ecr 416094609], length 0
10:12:25.318680 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [P.], seq 251:997, ack 970, win 483, options [nop,nop,TS val 3345035577 ecr 416094609], length 746
10:12:25.318691 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [.], ack 997, win 483, options [nop,nop,TS val 416094615 ecr 3345035577], length 0
10:12:25.318847 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [P.], seq 997:1776, ack 970, win 483, options [nop,nop,TS val 3345035578 ecr 416094609], length 779
10:12:25.318863 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [.], ack 1776, win 477, options [nop,nop,TS val 416094616 ecr 3345035578], length 0
10:12:25.322330 IP 10.0.1.233.8443 > 10.0.1.135.42642: Flags [P.], seq 3473654050:3473654360, ack 2449113861, win 482, options [nop,nop,TS val 416094619 ecr 3345029565], length 310
10:12:25.322344 IP 10.0.1.233.8443 > 10.0.1.85.40086: Flags [P.], seq 1:311, ack 1, win 482, options [nop,nop,TS val 2463507429 ecr 4289510559], length 310
10:12:25.322682 IP 10.0.1.233.8443 > 10.0.1.135.42642: Flags [P.], seq 310:614, ack 1, win 482, options [nop,nop,TS val 416094619 ecr 3345029565], length 304
10:12:25.322714 IP 10.0.1.233.8443 > 10.0.1.85.40086: Flags [P.], seq 311:615, ack 1, win 482, options [nop,nop,TS val 2463507429 ecr 4289510559], length 304
10:12:25.323067 IP 10.0.1.135.42642 > 10.0.1.233.8443: Flags [.], ack 614, win 2652, options [nop,nop,TS val 3345035582 ecr 416094619], length 0
10:12:25.323394 IP 10.0.1.85.40086 > 10.0.1.233.8443: Flags [.], ack 615, win 443, options [nop,nop,TS val 4289516951 ecr 2463507429], length 0
10:12:25.326057 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [P.], seq 970:1086, ack 1776, win 477, options [nop,nop,TS val 416094623 ecr 3345035578], length 116
10:12:25.326159 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [P.], seq 1086:1110, ack 1776, win 477, options [nop,nop,TS val 416094623 ecr 3345035578], length 24
10:12:25.326199 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [F.], seq 1110, ack 1776, win 477, options [nop,nop,TS val 416094623 ecr 3345035578], length 0
10:12:25.326442 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [.], ack 1086, win 483, options [nop,nop,TS val 3345035585 ecr 416094623], length 0
10:12:25.326443 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [.], ack 1110, win 483, options [nop,nop,TS val 3345035585 ecr 416094623], length 0
10:12:25.326510 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [P.], seq 1776:1800, ack 1111, win 483, options [nop,nop,TS val 3345035585 ecr 416094623], length 24
10:12:25.326537 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [R], seq 2806071220, win 0, length 0
10:12:25.326576 IP 10.0.1.135.36366 > 10.0.1.233.8443: Flags [F.], seq 1800, ack 1111, win 483, options [nop,nop,TS val 3345035585 ecr 416094623], length 0
10:12:25.326584 IP 10.0.1.233.8443 > 10.0.1.135.36366: Flags [R], seq 2806071220, win 0, length 0
10:12:25.326609 IP 10.0.1.135.8443 > 10.0.1.233.35178: Flags [P.], seq 755:964, ack 1, win 482, options [nop,nop,TS val 3345035585 ecr 416093698], length 209
10:12:25.326651 IP 10.0.1.233.35178 > 10.0.1.135.8443: Flags [.], ack 964, win 4750, options [nop,nop,TS val 416094623 ecr 3345035568], length 0
10:12:25.327124 IP 10.0.1.135.8443 > 10.0.1.233.35184: Flags [P.], seq 787:996, ack 38, win 480, options [nop,nop,TS val 3345035586 ecr 416093698], length 209
10:12:25.327160 IP 10.0.1.233.35184 > 10.0.1.135.8443: Flags [.], ack 996, win 443, options [nop,nop,TS val 416094624 ecr 3345035568], length 0
10:12:25.328574 IP 10.0.1.135.42628 > 10.0.1.233.8443: Flags [P.], seq 430:4676, ack 1, win 483, options [nop,nop,TS val 3345035587 ecr 416094200], length 4246
10:12:25.328592 IP 10.0.1.233.8443 > 10.0.1.135.42628: Flags [.], ack 4676, win 3547, options [nop,nop,TS val 416094625 ecr 3345035587], length 0
10:12:25.330586 IP 10.0.1.233.35168 > 10.0.1.135.8443: Flags [P.], seq 310:372, ack 1, win 483, options [nop,nop,TS val 416094627 ecr 3345035163], length 62
10:12:25.330926 IP 10.0.1.135.8443 > 10.0.1.233.35168: Flags [.], ack 372, win 442, options [nop,nop,TS val 3345035590 ecr 416094627], length 0
10:12:25.339811 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 24020, win 501, options [nop,nop,TS val 2428729624 ecr 3452187463], length 0
10:12:25.398163 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 24020:25040, ack 1, win 469, options [nop,nop,TS val 3452187567 ecr 2428729624], length 1020
10:12:25.398245 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 25040:26032, ack 1, win 469, options [nop,nop,TS val 3452187567 ecr 2428729624], length 992
10:12:25.398401 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 26032:27812, ack 1, win 469, options [nop,nop,TS val 3452187567 ecr 2428729624], length 1780
10:12:25.398445 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 27812:28444, ack 1, win 469, options [nop,nop,TS val 3452187567 ecr 2428729624], length 632
10:12:25.398631 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 28444:30104, ack 1, win 469, options [nop,nop,TS val 3452187567 ecr 2428729624], length 1660
10:12:25.433103 IP 10.0.1.85.8443 > 10.0.1.233.44290: Flags [.], ack 1, win 481, options [nop,nop,TS val 4289517061 ecr 2463504468], length 0
10:12:25.433125 IP 10.0.1.233.44290 > 10.0.1.85.8443: Flags [.], ack 1, win 483, options [nop,nop,TS val 2463507540 ecr 4289516266], length 0
10:12:25.444177 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 30104, win 501, options [nop,nop,TS val 2428729729 ecr 3452187567], length 0
10:12:25.502088 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 30104:30476, ack 1, win 469, options [nop,nop,TS val 3452187671 ecr 2428729729], length 372
10:12:25.502179 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 30476:31448, ack 1, win 469, options [nop,nop,TS val 3452187671 ecr 2428729729], length 972
10:12:25.547501 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 31448, win 501, options [nop,nop,TS val 2428729832 ecr 3452187671], length 0
10:12:25.606073 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 31448:31964, ack 1, win 469, options [nop,nop,TS val 3452187775 ecr 2428729832], length 516
10:12:25.652864 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 31964, win 501, options [nop,nop,TS val 2428729936 ecr 3452187775], length 0
10:12:25.710070 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 31964:32312, ack 1, win 469, options [nop,nop,TS val 3452187879 ecr 2428729936], length 348
10:12:25.756195 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 32312, win 501, options [nop,nop,TS val 2428730041 ecr 3452187879], length 0
10:12:25.814074 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 32312:32660, ack 1, win 469, options [nop,nop,TS val 3452187983 ecr 2428730041], length 348
10:12:25.816928 IP 10.0.1.135.8443 > 10.0.1.233.35174: Flags [.], ack 1, win 12293, options [nop,nop,TS val 3345036076 ecr 416092042], length 0
10:12:25.816953 IP 10.0.1.233.35174 > 10.0.1.135.8443: Flags [.], ack 1, win 16502, options [nop,nop,TS val 416095114 ecr 3345033105], length 0
10:12:25.859531 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 32660, win 501, options [nop,nop,TS val 2428730144 ecr 3452187983], length 0
10:12:25.885930 IP 10.0.1.233.40105 > 10.0.1.135.8472: OTV, flags [I] (0x08), overlay 0, instance 15728640
ARP, Request who-has 240.135.0.223 tell 240.233.0.31, length 28
10:12:25.886336 IP 10.0.1.233.22 > 177.37.251.57.28599: Flags [P.], seq 156:376, ack 1, win 469, options [nop,nop,TS val 3452188055 ecr 2428727145], length 220
10:12:25.903857 IP 10.0.1.135.42628 > 10.0.1.233.8443: Flags [P.], seq 4676:4762, ack 1, win 483, options [nop,nop,TS val 3345036163 ecr 416094625], length 86
10:12:25.903885 IP 10.0.1.233.8443 > 10.0.1.135.42628: Flags [.], ack 4762, win 3554, options [nop,nop,TS val 416095201 ecr 3345036163], length 0
10:12:25.905962 IP 10.0.1.233.35168 > 10.0.1.135.8443: Flags [P.], seq 372:434, ack 1, win 483, options [nop,nop,TS val 416095203 ecr 3345035590], length 62
10:12:25.906315 IP 10.0.1.135.8443 > 10.0.1.233.35168: Flags [.], ack 434, win 442, options [nop,nop,TS val 3345036165 ecr 416095203], length 0
10:12:25.917862 IP 10.0.1.233.35174 > 10.0.1.135.8443: Flags [.], ack 1, win 16502, options [nop,nop,TS val 416095215 ecr 3345033105], length 0
10:12:25.917862 IP 10.0.1.233.44294 > 10.0.1.85.8443: Flags [.], ack 1, win 2652, options [nop,nop,TS val 2463508025 ecr 4289514474], length 0
10:12:25.918239 IP 10.0.1.135.8443 > 10.0.1.233.35174: Flags [.], ack 1, win 12293, options [nop,nop,TS val 3345036177 ecr 416095114], length 0
10:12:25.918267 IP 10.0.1.85.8443 > 10.0.1.233.44294: Flags [.], ack 1, win 482, options [nop,nop,TS val 4289517546 ecr 2463501706], length 0
10:12:25.918314 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 32660:33792, ack 1, win 469, options [nop,nop,TS val 3452188087 ecr 2428730144], length 1132
10:12:25.918450 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 33792:34572, ack 1, win 469, options [nop,nop,TS val 3452188087 ecr 2428730144], length 780
10:12:25.928962 IP 177.37.251.57.28599 > 10.0.1.233.22: Flags [.], ack 376, win 501, options [nop,nop,TS val 2428730216 ecr 3452188055], length 0
10:12:25.962867 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 34572, win 501, options [nop,nop,TS val 2428730248 ecr 3452188087], length 0
10:12:25.963882 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [P.], seq 1:37, ack 34572, win 501, options [nop,nop,TS val 2428730248 ecr 3452188087], length 36
10:12:25.963893 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [.], ack 37, win 469, options [nop,nop,TS val 3452188133 ecr 2428730248], length 0
10:12:26.022067 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 34572:35232, ack 37, win 469, options [nop,nop,TS val 3452188191 ecr 2428730248], length 660
10:12:26.022110 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 35232:35564, ack 37, win 469, options [nop,nop,TS val 3452188191 ecr 2428730248], length 332
10:12:26.022126 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 35564:35912, ack 37, win 469, options [nop,nop,TS val 3452188191 ecr 2428730248], length 348
10:12:26.067228 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 35564, win 501, options [nop,nop,TS val 2428730352 ecr 3452188191], length 0
10:12:26.067228 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 35912, win 501, options [nop,nop,TS val 2428730352 ecr 3452188191], length 0
10:12:26.126016 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 35912:36284, ack 37, win 469, options [nop,nop,TS val 3452188295 ecr 2428730352], length 372
10:12:26.126064 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 36284:36640, ack 37, win 469, options [nop,nop,TS val 3452188295 ecr 2428730352], length 356
10:12:26.126099 IP 10.0.1.233.22 > 177.37.251.57.49998: Flags [P.], seq 36640:36828, ack 37, win 469, options [nop,nop,TS val 3452188295 ecr 2428730352], length 188
10:12:26.171593 IP 177.37.251.57.49998 > 10.0.1.233.22: Flags [.], ack 36828, win 501, options [nop,nop,TS val 2428730456 ecr 3452188295], length 0
^C
224 packets captured
228 packets received by filter
0 packets dropped by kernel

I just thought of something which might be an issue here. As mentioned before, this is running on Oracle Cloud and everything is blocked by default there. This is running inside a VCN which has it’s own firewall rules. The VCN has IPv4 CIDR Block: 10.0.0.0/16 and all these instances are in a subnet with IPv4 CIDR Block: 10.0.1.0/24. Every subnet on Oracle Cloud has it’s own firewall ingress and egress rules and what is not there is blocked by default.

I added two sets of ingress rules to the subnet firewall:

  • Source Type: CIDR; Source CIDR: 10.0.0.0/16; IP Protocol: TCP; Source Port Range: All; Destination Port Range: 8443; Description: Victor: lxd inside subnet
  • Source Type: CIDR; Source CIDR: 10.0.0.0/16; IP Protocol: ICMP; Type: All; Code: All; Description: Victor: allow ping inside vcn

Should I add any other rules here? They might be in play on this issue.

I would suggest allowing all internal traffic until you have resolved the issue then you can lock it down after that.

Allowed all local traffic and everything was solved now.

So how do I figure out exactly what to allow? Ideally it should be as little as possible. But is this documented anywhere for lxd?

Just tried TCP 53 and UDP 53 and 67 and it was not enough. No ping to container ip or container name.

You need to allow the fan encapsulation protocol between hosts. Thus is vxlan which I believe is UDP port 4789 , bit you can confirm that using tcpdump to observe the traffic.

Figured it out! The only ports I have to open on Oracle Cloud firewall are TCP 8443 and UDP 8472. Just these and the default ports which are already open and I can ping between nodes in a cluster through ubuntu-fan.

1 Like