Problem with UDP proxying [SOLVED]

Incus
,
1

I’m trying to get UDP proxying (non-NAT) to work, and I’m not sure what I’m doing wrong. TCP proxying is fine.

I am using incus 6.0.2 LTS from Zabbly packages under Ubuntu 22.04.5.

The network “bridge0” has IP address 192.0.2.254, and the container “nfsen” has IP address 192.0.2.3 (full details at end of post).

I have added the following proxy:

incus config device add nfsen udp9996 proxy 'listen=udp:[::]:29996' 'connect=udp:[192.0.2.3]:9996'

Running tcpdump on the host, I see incoming packets on UDP port 29996. But tcpdump on bridge0, or on eth0 inside the container, does not see any packets on 9996 (or 29996).

ufw has rules to allow everything:

Anywhere on bridge0        ALLOW       Anywhere
Anywhere (v6) on bridge0   ALLOW       Anywhere (v6)
Anywhere                   ALLOW FWD   Anywhere on bridge0
Anywhere (v6)              ALLOW FWD   Anywhere (v6) on bridge0

although to be sure, I’ve also tried setting the policy on INPUT/FORWARD/OUTPUT chains to “ACCEPT” and flushing all the rules - and sysctl net.ipv4.conf.bridge0.rp_filter=0. No difference.

However, if I send a packet using nc:

echo "boing" | nc -u 192.0.2.3 9996

… then this does appear in both tcpdump sessions.

14:59:57.036196 IP 192.0.2.254.36016 > 192.0.2.3.9996: UDP, length 6

I can see that incus is listening:

# ss -naup | grep :29996
UNCONN 0      0                                           *:29996            *:*    users:(("incusd",pid=426472,fd=8),("incusd",pid=426472,fd=3))

And if I run strace on that pid, I see packets arriving on fd 8 and being sent to fd 10.

# strace -tt -f -p 426472
...
[pid 426487] 15:01:47.922548 futex(0x3154980, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 426475] 15:01:47.922775 <... restart_syscall resumed>) = 0
[pid 426487] 15:01:47.922945 recvfrom(8,  <unfinished ...>
[pid 426475] 15:01:47.923025 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 426487] 15:01:47.923202 <... recvfrom resumed>"\0\t\0\5Gb\251\254f\355\216\244\0\0\10\243\0\0\0\0\1\0\1\264GbijGb2~"..., 32768, 0, {sa_family=AF_INET6, sin6_port=htons(29996), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::ffff:10.12.253.1", &sin6_addr), sin6_scope_id=0}, [112 => 28]) = 456
[pid 426475] 15:01:47.923387 <... nanosleep resumed>NULL) = 0
[pid 426487] 15:01:47.923547 write(10, "\0\t\0\5Gb\251\254f\355\216\244\0\0\10\243\0\0\0\0\1\0\1\264GbijGb2~"..., 456 <unfinished ...>
[pid 426475] 15:01:47.923619 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 426487] 15:01:47.923834 <... write resumed>) = 456
[pid 426475] 15:01:47.923862 <... nanosleep resumed>NULL) = 0
[pid 426487] 15:01:47.923933 recvfrom(8,  <unfinished ...>
[pid 426475] 15:01:47.923963 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 426487] 15:01:47.923996 <... recvfrom resumed>"\0\t\0\2Gb\251\254f\355\216\244\0\0\10\244\0\0\0\0\1\1\0\364\6*\7\2D\0006\312"..., 32768, 0, {sa_family=AF_INET6, sin6_port=htons(29996), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::ffff:10.12.253.1", &sin6_addr), sin6_scope_id=0}, [112 => 28]) = 264
[pid 426475] 15:01:47.924097 <... nanosleep resumed>NULL) = 0
[pid 426487] 15:01:47.924119 write(10, "\0\t\0\2Gb\251\254f\355\216\244\0\0\10\244\0\0\0\0\1\1\0\364\6*\7\2D\0006\312"..., 264 <unfinished ...>
[pid 426475] 15:01:47.924149 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 426487] 15:01:47.924263 <... write resumed>) = 264
[pid 426475] 15:01:47.924287 <... nanosleep resumed>NULL) = 0
[pid 426475] 15:01:47.924335 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 426487] 15:01:47.924374 recvfrom(8, 0xc000884000, 32768, 0, 0xc000891b34, [112]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 426475] 15:01:47.924691 <... nanosleep resumed>NULL) = 0
[pid 426475] 15:01:47.924742 nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 426487] 15:01:47.924777 epoll_pwait(5, [{events=EPOLLOUT, data={u32=471334913, u64=9176656171968757761}}], 128, 0, NULL, 0) = 1
[pid 426475] 15:01:47.924901 <... nanosleep resumed>NULL) = 0
[pid 426487] 15:01:47.924925 epoll_pwait(5,  <unfinished ...>
[pid 426475] 15:01:47.924947 futex(0x3154980, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>

I can’t see what the destination address/port of that socket is though:

# ls -l /proc/426472/fd
total 0
lr-x------ 1 root root 64 Sep 20 14:37 0 -> /dev/null
lrwx------ 1 root root 64 Sep 20 14:37 1 -> /var/log/incus/nsrc-builder_nfsen/proxy.udp9996.log
lrwx------ 1 root root 64 Sep 20 14:37 10 -> 'socket:[1429620]'
lrwx------ 1 root root 64 Sep 20 14:37 11 -> 'socket:[36134]'
lrwx------ 1 root root 64 Sep 20 14:37 2 -> /var/log/incus/nsrc-builder_nfsen/proxy.udp9996.log
lrwx------ 1 root root 64 Sep 20 14:37 3 -> 'socket:[1426694]'
lrwx------ 1 root root 64 Sep 20 14:37 4 -> 'anon_inode:[pidfd]'
lrwx------ 1 root root 64 Sep 20 14:37 44 -> 'anon_inode:[pidfd]'
lrwx------ 1 root root 64 Sep 20 14:37 5 -> 'anon_inode:[eventpoll]'
lrwx------ 1 root root 64 Sep 20 14:37 6 -> 'anon_inode:[eventfd]'
lrwx------ 1 root root 64 Sep 20 14:37 7 -> 'anon_inode:[eventpoll]'
lrwx------ 1 root root 64 Sep 20 14:37 8 -> 'socket:[1426694]'
lrwx------ 1 root root 64 Sep 20 14:37 9 -> 'socket:[1429619]'
# cat /var/log/incus/nsrc-builder_nfsen/proxy.udp9996.log
Status: Started
#

Any ideas where I can look next?

Full container details:

$ incus config show nfsen
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (cloud) (20240919_07:42)
  image.name: ubuntu-noble-amd64-cloud-20240919_07:42
  image.os: ubuntu
  image.release: noble
  image.serial: "20240919_07:42"
  image.variant: cloud
  volatile.base_image: 2fac9ede6c4eb75e9f6af0f88f45eb1ffaf32c8e6f72d6e6f01b73aec647f763
  volatile.cloud-init.instance-id: c0ee8a6e-a98d-4021-bec8-517d4e1f769c
  volatile.eth0.host_name: vethd0828766
  volatile.eth0.hwaddr: 00:16:3e:cf:b5:f4
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: 37eddb8f-a3a5-4d18-8244-cf7b1668163f
  volatile.uuid.generation: 37eddb8f-a3a5-4d18-8244-cf7b1668163f
devices:
  eth0:
    ipv4.address: 192.0.2.3
    name: eth0
    network: bridge0
    type: nic
  root:
    path: /
    pool: zfs
    type: disk
  udp9996:
    connect: udp:[192.0.2.3]:9996
    listen: udp:[::]:29996
    type: proxy
ephemeral: false
profiles:
- default
stateful: false
description: ""

$ incus network show bridge0
config:
  dns.domain: nmm.internal
  ipv4.address: 192.0.2.254/24
  ipv4.dhcp.ranges: 192.0.2.200-192.0.2.250
  ipv4.nat: "true"
  ipv6.address: fd42:ed44:301:241c::1/64
  ipv6.nat: "true"
description: ""
name: bridge0
type: bridge
used_by:
- /1.0/instances/nfsen?project=nsrc-builder
<< SNIP >>
managed: true
status: Created
locations:
- none
project: default
2

Digging further with lsof (pid has changed as I’ve removed and re-added the proxy device):

# lsof -i -a -p 459723 -n 2>/dev/null
COMMAND    PID     USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
incusd  459723  1000000    3u  IPv6 1588756      0t0  UDP *:29996
incusd  459723  1000000    8u  IPv6 1588756      0t0  UDP *:29996
# lsof -d 8,10 -a -p 459723 -n 2>/dev/null
COMMAND    PID     USER   FD   TYPE  DEVICE SIZE/OFF    NODE NAME
incusd  459723  1000000    8u  IPv6 1588756      0t0     UDP *:29996
incusd  459723  1000000   10u  sock     0,8      0t0 1592619 protocol: UDP

That’s not helping me much: the outbound on FD 10 looks different (it’s not picked up as an internet socket with “-i”)

Oh, and I also tried listening on [0.0.0.0] instead of [::] since the incoming packets are on IPv4, but that didn’t help.

3

Sigh, I’m being dumb. The incoming proxied packets are visible with tcpdump on the lo interface, not eth0. Sorry for the noise!

2 Likes