Proxy device IPv4/IPv6 issue. Container should serve nginx specific site only in IPv6, but does both or none

dionysius · November 20, 2021, 11:51am

How can I explicitly split IPv4 and IPv6 traffic with a proxy device? Is it possible that I can’t define the IPV6_V6ONLY behaviour (or just a wrong assumption)?

Fact so far (see below): Nginx seems to differentiate IP version from received traffic, which depends on the the way the proxy device is configured.

Background

I have a container serving various nginx sites, so they all serve it via:

root@container:~# cat /etc/nginx/sites-enabled/normal.site | grep listen
    listen 443 ssl proxy_protocol;
    listen [::]:443 ssl proxy_protocol;
    listen 80 proxy_protocol;
    listen [::]:80 proxy_protocol;

and a site which I want to explicitly only serve in ipv6:

root@container:~# cat /etc/nginx/sites-enabled/ipv6.site | grep listen
    listen [::]:443 ssl proxy_protocol;
    listen [::]:80 proxy_protocol;

Any unknown site not matching a server directive gets returned 444 Empty reply from server.

And for testing purposes the following /etc/hosts file:

root@server:~# cat /etc/hosts | grep ipv
::1     ip6-localhost   ip6-loopback   ipv6.site

case 1: with 127.0.01

If the device config is as follows:

devices:
  http:
    connect: tcp:127.0.0.1:80
    listen: tcp:0.0.0.0:80
    proxy_protocol: "true"
    type: proxy
  https:
    connect: tcp:127.0.0.1:443
    listen: tcp:0.0.0.0:443
    proxy_protocol: "true"
    type: proxy

I get:

root@server:~# curl -4 ipv6.site
curl: (52) Empty reply from server
root@server:~# curl -6 ipv6.site
curl: (52) Empty reply from server

Additionally for this case the tcpdump:

root@server:~# tcpdump -i any port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
13:12:34.272833 IP localhost.42706 > localhost.http: Flags [S], seq 1588935550, win 65495, options [mss 65495,sackOK,TS val 4004873144 ecr 0,nop,wscale 7], length 0
13:12:34.272860 IP localhost.http > localhost.42706: Flags [S.], seq 3736461613, ack 1588935551, win 65483, options [mss 65495,sackOK,TS val 4004873144 ecr 4004873144,nop,wscale 7], length 0
13:12:34.272882 IP localhost.42706 > localhost.http: Flags [.], ack 1, win 512, options [nop,nop,TS val 4004873144 ecr 4004873144], length 0
13:12:34.272962 IP localhost.42706 > localhost.http: Flags [P.], seq 1:81, ack 1, win 512, options [nop,nop,TS val 4004873144 ecr 4004873144], length 80: HTTP: GET / HTTP/1.1
13:12:34.273709 IP localhost.http > localhost.42706: Flags [F.], seq 1, ack 81, win 512, options [nop,nop,TS val 4004873145 ecr 4004873144], length 0
13:12:34.273975 IP localhost.42706 > localhost.http: Flags [F.], seq 81, ack 2, win 512, options [nop,nop,TS val 4004873145 ecr 4004873145], length 0
13:12:34.273999 IP localhost.http > localhost.42706: Flags [.], ack 82, win 512, options [nop,nop,TS val 4004873145 ecr 4004873145], length 0
13:12:46.320884 IP6 ip6-localhost.41432 > ip6-localhost.http: Flags [S], seq 670525252, win 65476, options [mss 65476,sackOK,TS val 729838226 ecr 0,nop,wscale 7], length 0
13:12:46.320908 IP6 ip6-localhost.http > ip6-localhost.41432: Flags [S.], seq 3237775875, ack 670525253, win 65464, options [mss 65476,sackOK,TS val 729838226 ecr 729838226,nop,wscale 7], length 0
13:12:46.320929 IP6 ip6-localhost.41432 > ip6-localhost.http: Flags [.], ack 1, win 512, options [nop,nop,TS val 729838226 ecr 729838226], length 0
13:12:46.321008 IP6 ip6-localhost.41432 > ip6-localhost.http: Flags [P.], seq 1:81, ack 1, win 512, options [nop,nop,TS val 729838226 ecr 729838226], length 80: HTTP: GET / HTTP/1.1
13:12:46.321753 IP6 ip6-localhost.http > ip6-localhost.41432: Flags [F.], seq 1, ack 81, win 512, options [nop,nop,TS val 729838227 ecr 729838226], length 0
13:12:46.322037 IP6 ip6-localhost.41432 > ip6-localhost.http: Flags [F.], seq 81, ack 2, win 512, options [nop,nop,TS val 729838227 ecr 729838227], length 0
13:12:46.322056 IP6 ip6-localhost.http > ip6-localhost.41432: Flags [.], ack 82, win 512, options [nop,nop,TS val 729838227 ecr 729838227], length 0

which shows that the first curl is effectively using IPv4 and the second one is using IPv6.

Inside the container though everything is IPv4 now:

root@container:~# tcpdump -i any port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
12:12:34.273075 IP localhost.42708 > localhost.http: Flags [S], seq 1834994760, win 65495, options [mss 65495,sackOK,TS val 4004873144 ecr 0,nop,wscale 7], length 0
12:12:34.273093 IP localhost.http > localhost.42708: Flags [S.], seq 3146860621, ack 1834994761, win 65483, options [mss 65495,sackOK,TS val 4004873144 ecr 4004873144,nop,wscale 7], length 0
12:12:34.273109 IP localhost.42708 > localhost.http: Flags [.], ack 1, win 512, options [nop,nop,TS val 4004873144 ecr 4004873144], length 0
12:12:34.273215 IP localhost.42708 > localhost.http: Flags [P.], seq 1:42, ack 1, win 512, options [nop,nop,TS val 4004873144 ecr 4004873144], length 41: HTTP
12:12:34.273628 IP localhost.42708 > localhost.http: Flags [F.], seq 122, ack 2, win 512, options [nop,nop,TS val 4004873145 ecr 4004873145], length 0
12:12:34.273643 IP localhost.http > localhost.42708: Flags [.], ack 123, win 512, options [nop,nop,TS val 4004873145 ecr 4004873145], length 0
12:12:46.321083 IP localhost.42712 > localhost.http: Flags [S], seq 3319134991, win 65495, options [mss 65495,sackOK,TS val 4004885192 ecr 0,nop,wscale 7], length 0
12:12:46.321100 IP localhost.http > localhost.42712: Flags [S.], seq 3551039829, ack 3319134992, win 65483, options [mss 65495,sackOK,TS val 4004885192 ecr 4004885192,nop,wscale 7], length 0
12:12:46.321113 IP localhost.42712 > localhost.http: Flags [.], ack 1, win 512, options [nop,nop,TS val 4004885192 ecr 4004885192], length 0
12:12:46.321245 IP localhost.42712 > localhost.http: Flags [P.], seq 1:30, ack 1, win 512, options [nop,nop,TS val 4004885192 ecr 4004885192], length 29: HTTP
12:12:46.321253 IP localhost.http > localhost.42712: Flags [.], ack 30, win 512, options [nop,nop,TS val 4004885192 ecr 4004885192], length 0
12:12:46.321426 IP localhost.42712 > localhost.http: Flags [P.], seq 30:110, ack 1, win 512, options [nop,nop,TS val 4004885193 ecr 4004885192], length 80: HTTP: GET / HTTP/1.1
12:12:46.321437 IP localhost.http > localhost.42712: Flags [.], ack 110, win 512, options [nop,nop,TS val 4004885193 ecr 4004885193], length 0
12:12:46.321619 IP localhost.http > localhost.42712: Flags [F.], seq 1, ack 110, win 512, options [nop,nop,TS val 4004885193 ecr 4004885193], length 0
12:12:46.321674 IP localhost.42712 > localhost.http: Flags [F.], seq 110, ack 2, win 512, options [nop,nop,TS val 4004885193 ecr 4004885193], length 0
12:12:46.321685 IP localhost.http > localhost.42712: Flags [.], ack 111, win 512, options [nop,nop,TS val 4004885193 ecr 4004885193], length 0

Since nginx received the request as localhost.http, it does not serve the request for this site

case 2: with [::1]

If I set the device config as follows:

devices:
  http:
    connect: tcp:[::1]:80
    listen: tcp:[::]:80
    proxy_protocol: "true"
    type: proxy
  https:
    connect: tcp:[::1]:443
    listen: tcp:[::]:443
    proxy_protocol: "true"
    type: proxy

I get:

root@server:~# curl -4 ipv6.site
<!DOCTYPE html>
...
root@server:~# curl -6 ipv6.site
<!DOCTYPE html>
...

Additionally for this case the tcpdump:

root@server:~# tcpdump -i any port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
13:06:16.182619 IP localhost.42692 > localhost.http: Flags [S], seq 1938281950, win 65495, options [mss 65495,sackOK,TS val 4004495054 ecr 0,nop,wscale 7], length 0
13:06:16.182645 IP localhost.http > localhost.42692: Flags [S.], seq 1605870375, ack 1938281951, win 65483, options [mss 65495,sackOK,TS val 4004495054 ecr 4004495054,nop,wscale 7], length 0
13:06:16.182667 IP localhost.42692 > localhost.http: Flags [.], ack 1, win 512, options [nop,nop,TS val 4004495054 ecr 4004495054], length 0
13:06:16.182754 IP localhost.42692 > localhost.http: Flags [P.], seq 1:81, ack 1, win 512, options [nop,nop,TS val 4004495054 ecr 4004495054], length 80: HTTP: GET / HTTP/1.1
13:06:16.183555 IP localhost.http > localhost.42692: Flags [P.], seq 1:1150, ack 81, win 512, options [nop,nop,TS val 4004495055 ecr 4004495054], length 1149: HTTP: HTTP/1.1 200 OK
13:06:16.183584 IP localhost.42692 > localhost.http: Flags [.], ack 1150, win 504, options [nop,nop,TS val 4004495055 ecr 4004495055], length 0
13:06:16.183803 IP localhost.42692 > localhost.http: Flags [F.], seq 81, ack 1150, win 512, options [nop,nop,TS val 4004495055 ecr 4004495055], length 0
13:06:16.183988 IP localhost.http > localhost.42692: Flags [F.], seq 1150, ack 82, win 512, options [nop,nop,TS val 4004495055 ecr 4004495055], length 0
13:06:16.184004 IP localhost.42692 > localhost.http: Flags [.], ack 1151, win 512, options [nop,nop,TS val 4004495055 ecr 4004495055], length 0
13:06:20.950320 IP6 ip6-localhost.41418 > ip6-localhost.http: Flags [S], seq 3251368108, win 65476, options [mss 65476,sackOK,TS val 729452856 ecr 0,nop,wscale 7], length 0
13:06:20.950341 IP6 ip6-localhost.http > ip6-localhost.41418: Flags [S.], seq 1666642826, ack 3251368109, win 65464, options [mss 65476,sackOK,TS val 729452856 ecr 729452856,nop,wscale 7], length 0
13:06:20.950358 IP6 ip6-localhost.41418 > ip6-localhost.http: Flags [.], ack 1, win 512, options [nop,nop,TS val 729452856 ecr 729452856], length 0
13:06:20.950421 IP6 ip6-localhost.41418 > ip6-localhost.http: Flags [P.], seq 1:81, ack 1, win 512, options [nop,nop,TS val 729452856 ecr 729452856], length 80: HTTP: GET / HTTP/1.1
13:06:20.951276 IP6 ip6-localhost.http > ip6-localhost.41418: Flags [P.], seq 1:1150, ack 81, win 512, options [nop,nop,TS val 729452857 ecr 729452856], length 1149: HTTP: HTTP/1.1 200 OK
13:06:20.951302 IP6 ip6-localhost.41418 > ip6-localhost.http: Flags [.], ack 1150, win 504, options [nop,nop,TS val 729452857 ecr 729452857], length 0
13:06:20.951488 IP6 ip6-localhost.41418 > ip6-localhost.http: Flags [F.], seq 81, ack 1150, win 512, options [nop,nop,TS val 729452857 ecr 729452857], length 0
13:06:20.951574 IP6 ip6-localhost.http > ip6-localhost.41418: Flags [F.], seq 1150, ack 82, win 512, options [nop,nop,TS val 729452857 ecr 729452857], length 0
13:06:20.951585 IP6 ip6-localhost.41418 > ip6-localhost.http: Flags [.], ack 1151, win 512, options [nop,nop,TS val 729452857 ecr 729452857], length 0

which shows that the first curl is effectively using IPv4 and the second one is using IPv6.

Inside the container though everything is IPv6 now:

root@container:~# tcpdump -i any port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
12:06:16.182881 IP6 ip6-localhost.41416 > ip6-localhost.http: Flags [S], seq 1799105677, win 65476, options [mss 65476,sackOK,TS val 729448088 ecr 0,nop,wscale 7], length 0
12:06:16.182898 IP6 ip6-localhost.http > ip6-localhost.41416: Flags [S.], seq 262452753, ack 1799105678, win 65464, options [mss 65476,sackOK,TS val 729448088 ecr 729448088,nop,wscale 7], length 0
12:06:16.182912 IP6 ip6-localhost.41416 > ip6-localhost.http: Flags [.], ack 1, win 512, options [nop,nop,TS val 729448088 ecr 729448088], length 0
12:06:16.183044 IP6 ip6-localhost.41416 > ip6-localhost.http: Flags [P.], seq 1:42, ack 1, win 512, options [nop,nop,TS val 729448088 ecr 729448088], length 41: HTTP
12:06:16.183495 IP6 ip6-localhost.http > ip6-localhost.41416: Flags [P.], seq 1:1150, ack 122, win 512, options [nop,nop,TS val 729448089 ecr 729448088], length 1149: HTTP: HTTP/1.1 200 OK
12:06:16.183507 IP6 ip6-localhost.41416 > ip6-localhost.http: Flags [.], ack 1150, win 504, options [nop,nop,TS val 729448089 ecr 729448089], length 0
12:06:16.183895 IP6 ip6-localhost.41416 > ip6-localhost.http: Flags [F.], seq 122, ack 1150, win 512, options [nop,nop,TS val 729448089 ecr 729448089], length 0
12:06:16.183942 IP6 ip6-localhost.http > ip6-localhost.41416: Flags [F.], seq 1150, ack 123, win 512, options [nop,nop,TS val 729448089 ecr 729448089], length 0
12:06:16.183955 IP6 ip6-localhost.41416 > ip6-localhost.http: Flags [.], ack 1151, win 512, options [nop,nop,TS val 729448089 ecr 729448089], length 0
12:06:20.950613 IP6 ip6-localhost.41420 > ip6-localhost.http: Flags [S], seq 1603331564, win 65476, options [mss 65476,sackOK,TS val 729452856 ecr 0,nop,wscale 7], length 0
12:06:20.950628 IP6 ip6-localhost.http > ip6-localhost.41420: Flags [S.], seq 2934414596, ack 1603331565, win 65464, options [mss 65476,sackOK,TS val 729452856 ecr 729452856,nop,wscale 7], length 0
12:06:20.950642 IP6 ip6-localhost.41420 > ip6-localhost.http: Flags [.], ack 1, win 512, options [nop,nop,TS val 729452856 ecr 729452856], length 0
12:06:20.950751 IP6 ip6-localhost.41420 > ip6-localhost.http: Flags [P.], seq 1:30, ack 1, win 512, options [nop,nop,TS val 729452856 ecr 729452856], length 29: HTTP
12:06:20.950762 IP6 ip6-localhost.http > ip6-localhost.41420: Flags [.], ack 30, win 512, options [nop,nop,TS val 729452856 ecr 729452856], length 0
12:06:20.950969 IP6 ip6-localhost.41420 > ip6-localhost.http: Flags [P.], seq 30:110, ack 1, win 512, options [nop,nop,TS val 729452856 ecr 729452856], length 80: HTTP: GET / HTTP/1.1
12:06:20.950981 IP6 ip6-localhost.http > ip6-localhost.41420: Flags [.], ack 110, win 512, options [nop,nop,TS val 729452856 ecr 729452856], length 0
12:06:20.951235 IP6 ip6-localhost.http > ip6-localhost.41420: Flags [P.], seq 1:1150, ack 110, win 512, options [nop,nop,TS val 729452856 ecr 729452856], length 1149: HTTP: HTTP/1.1 200 OK
12:06:20.951242 IP6 ip6-localhost.41420 > ip6-localhost.http: Flags [.], ack 1150, win 504, options [nop,nop,TS val 729452856 ecr 729452856], length 0
12:06:20.951535 IP6 ip6-localhost.41420 > ip6-localhost.http: Flags [F.], seq 110, ack 1150, win 512, options [nop,nop,TS val 729452857 ecr 729452856], length 0
12:06:20.951562 IP6 ip6-localhost.http > ip6-localhost.41420: Flags [F.], seq 1150, ack 111, win 512, options [nop,nop,TS val 729452857 ecr 729452857], length 0
12:06:20.951573 IP6 ip6-localhost.41420 > ip6-localhost.http: Flags [.], ack 1151, win 512, options [nop,nop,TS val 729452857 ecr 729452857], length 0

Since nginx received the request as ip-localhost.http, it serves the request for this site for both IPv4 and IPv6

case 3: defining both

If i try to set the device config as follows:

devices:
  http-4:
    connect: tcp:127.0.0.1:80
    listen: tcp:0.0.0.0:80
    proxy_protocol: "true"
    type: proxy
  https-4:
    connect: tcp:127.0.0.1:443
    listen: tcp:0.0.0.0:443
    proxy_protocol: "true"
    type: proxy
  http-6:
    connect: tcp:[::1]:80
    listen: tcp:[::]:80
    proxy_protocol: "true"
    type: proxy
  https-6:
    connect: tcp:[::1]:443
    listen: tcp:[::]:443
    proxy_protocol: "true"
    type: proxy

I get the error:

Config parsing error: Failed to start device "http-6": Error occurred when starting proxy device: Error: Failed to listen on [::]:80: listen tcp 0.0.0.0:80: bind: address already in use
Press enter to open the editor again or ctrl+c to abort change

Maybe related:

tomp · November 22, 2021, 9:04am

I think the key thing to realise here is that for both the proxy device and the nginx config, the following listen directive:

Will cause the socket to listen on both the IPv6 AND IPv4 wildcard addresses.

So for example, the following LXD proxy device:

lxc config device add c1 p1 proxy listen=tcp:[::]:80 connect=tcp:[::1]:80

Will listen on the wildcard IPv4 address on the host:

sudo ss -tlpn
LISTEN  0        4096                              *:80                  *:*      users:(("lxd",pid=87968,fd=8),("lxd",pid=87968,fd=3))

And because the proxy device will lose the source address of the remote connection by default, all requests will appear to come from the address of the host.

Nginx appears to have a similar behaviour, but as you mention, has an additional option to specify that [::] means IPv6 only (which you don’t appear to be using).

Anyway, to get the proxy and nginx to truly listen only on IPv6, then I suggest binding the proxy and nginx to a non-wildcard IPv6 address. For nginx, as you’re using the proxy to connect to it, you could use the local-loopback address [::1] and for the proxy I suggest you use an IPv6 address on the host.

stgraber · November 23, 2021, 10:12pm

Right, on Linux IPv6 binds the matching IPv4 address by default. You can change that setting through some sysctl, but it’s somewhat unusual.

dionysius · November 24, 2021, 3:05am

Makes sense, I was hoping for a generic solution without explicitly defining an address, so I can move the container without ever needing to think about its IP. It’s a rather an edge case wanting nginx to explicitly differ those traffic.