Can' tftp between containers in diferent managed networks

Incus
1

My goal is to build a PXE server running tftp.
Let’s say that the PXE server is running in a managed network named pxe-net.
Everything works fine from host, I can ping from host and the command below succeded.

host:~$ tftp -m binary <ip_pxe> -c get ipxe.efi

The problem comes when I try to run the same command from another container B in another managed network, let’s say incusbr0.
Pinging from B to pxe_server succeded but tftp from B to pxe_server fails.

My first thought is that UDP traffic is blocked between networks, but he following test is successfull.

pxe_srv:~# nc -u -l 4000

B:~# nc -u <ip_pxe_server>  4000

Here is the dump from tcpdump in both containers and the same request from host with success.

B requesting ipxe.efi*

16:12:51.007707 IP 10.149.165.224.41962 > 10.0.10.66.69: TFTP, length 17, RRQ "ipxe.efi" octet
16:12:53.447630 IP 10.149.165.1.60414 > 239.255.255.250.1900: UDP, length 101
16:12:53.518926 IP 10.149.165.1.35915 > 10.149.165.255.32412: UDP, length 21
16:12:53.518928 IP 10.149.165.1.39617 > 10.149.165.255.32414: UDP, length 21

pxe_server listening on port 69 (tftp) on request from B FAILS*

16:12:56.007939 IP 10.0.10.100.41962 > 10.0.10.66.69: TFTP, length 17, RRQ "ipxe.efi" octet
16:12:56.012454 IP 10.0.10.66.50948 > 10.0.10.100.41962: UDP, length 516
16:12:56.012546 IP 10.0.10.100 > 10.0.10.66: ICMP 10.0.10.100 udp port 41962 unreachable, length 552

pxe_server listening on port 69 (tftp) on request from host SUCCESS*

6:41:52.863961 IP 10.0.10.100.51531 > 10.0.10.66.69: TFTP, length 17, RRQ "ipxe.efi" octet
16:41:52.869931 IP 10.0.10.66.51562 > 10.0.10.100.51531: UDP, length 516
16:41:52.870137 IP 10.0.10.100.51531 > 10.0.10.66.51562: UDP, length 4
16:41:52.870239 IP 10.0.10.66.51562 > 10.0.10.100.51531: UDP, length 516
16:41:52.870349 IP 10.0.10.100.51531 > 10.0.10.66.51562: UDP, length 4
16:41:52.870405 IP 10.0.10.66.51562 > 10.0.10.100.51531: UDP, length 516
16:41:52.870549 IP 10.0.10.100.51531 > 10.0.10.66.51562: UDP, length 4

I can see that tftp request from B fails because related efimeral port 41692 to gateway isn’t unreachable.

Any ideas?
Might be firewal rules that incus creates wrong?

Thanks.

2

tftp is a very dump protocol having troubles with nat as far I know, this might be related: How best to disable NAT between incus managed networks

3

Yeah, that’s definitely the NAT at fault. The target sees a request coming from 100.0.10.100 so it tries to connect back to port 41962 on that address, but that won’t work; it should connect to 10.149.165.224 which is the real client address.

1 Like
4

@candlerb do I see it right ipv4.nat=false and ipv6.nat=false for both bridges would fix his issue?

Or does he need to disable nat for all bridges he got?

5

Yes, setting ipv4.nat=false for those two bridges should fix the issue.

However, containers on those bridges will no longer be able to make outbound connections to the Internet, unless he manually adds suitable NAT rules to nftables or iptables.

Issue to watch: #3513. For configuring NAT manually see this comment.