You were likely running proxy in non-nat mode, nat mode will preserve source IP.
See Making sure that IP's "connected" to the containers gameserver proxy shows users real IP? for examples.
Just tried this kind of setup:
lxc config device add alp p7 proxy listen=tcp:a.b.c.d:7777 connect=tcp:0.0.0.0:7777
lxc config device set alp p7 nat=true listen=tcp:a.b.c.d:7777 connect=tcp:0.0.0.0:7777
In this case telnet a.b.c.d 7777 works ok from outside, it works from within the container itself, but it does not work from inside the host or from inside other containers.
I run internal DNS on local IP 172.16.172.233:53. I did this kind of setup for port 53, tcp and udp. Now container can talk to its dns via external IP address, I can connect to external IP and resolve from outside, but a) host itself can’t query DNS via external address and other containers can’t do that as well.
Can you check your LXD logs and see if there is a warning about br_netfilter not being enabled?
Jul 6 15:38:09 ns lxd.daemon[31233]: t=2020-07-06T15:38:09+0300 lvl=warn msg="Proxy bridge netfilter not enabled: br_netfilte
r not loaded: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory. Instances using the bridge will no
t be able to connect to the proxy's listen IP"
ls -al /etc/alternatives|grep iptables [/etc/alternatives 15:41 Mon 6]
lrwxrwxrwx 1 root root 25 Jun 13 03:31 iptables -> /usr/sbin/iptables-legacy
lrwxrwxrwx 1 root root 33 Jun 13 03:31 iptables-restore -> /usr/sbin/iptables-legacy-restore
lrwxrwxrwx 1 root root 30 Jun 13 03:31 iptables-save -> /usr/sbin/iptables-legacy-save
OK so you will need that enabled, see Resolve Host Top Level Domain to Bridge IP inside LXD network? for a similar issue.
Yeah, with
modprobe br_filter
it works. BTW, does that proxy mechanism work with ipv6 addresses?
Yes it should do.
Ok, thanks. I think it should be documented somewhere clearly, because whole process is not obvious and scattered around this forum… including this tiny detail about kernel module not loaded by default. And that you have to use two commands to create a proxy and then to set it into nat mode.
Yes the br_netfilter optional module could be documented here https://linuxcontainers.org/lxd/docs/master/instances#type-proxy
What do you mean about “you have to use two commands to create a proxy and then to set it into nat mode.”?
Nevermind. For some reasons I used add and then set as two commands. I looked up my zsh history and it seems that first hints I found on the net used ‘lxc config device set’, lxc complained about missing device, I became confused and used two commands – created proxy without nat and then set it into nat mode.
Anyway, clear guide how to set up port forwarding and preserve remote IPs would be great.
Great thanks.
I will look at adding a mention for br_netfilter to the proxy doc.
If you were able to write a post on the forum for setting up a proxy in nat mode then I could promote that into the Tutorials category for the benefit of other users.
I’ll try to write something, but my English is lousy
lxc config device add alp proxy80v6 proxy listen=tcp:2604:280:1:5f8::2:80 connect=tcp:0.0.0.0:80
Error: Invalid devices: Device validation failed "proxy80v6": Invalid value for device option "listen": address 2604:280:1:5f8::2:80: too many colons in address
Ugh.
You need to use square brackets, e.g. listen=tcp:[2604:280:1:5f8::2]:80
lxc config device add alp proxy80v6 proxy nat=true "listen=tcp:[2604:280:1:5f8::2]:80" "connect=tcp:[::]:80"
Error: Failed to start device "proxy80v6": Proxy connect IP cannot be used with any of the instance NICs static IPs
I was able to add v4 proxy to this container without problems.
You need to use static IPs for both IPv4 and IPv6 (when using proxy in nat mode), please show the output of lxc config show <container> --expanded
for the container you added the IPv4 proxy device on.
architecture: x86_64
config:
image.architecture: amd64
image.description: Alpine 3.9 amd64 (20190321_13:00)
image.os: Alpine
image.release: "3.9"
image.serial: "20190321_13:00"
volatile.base_image: 0cbd911b5a203c7e475241b8b22cc5332d10fd30ae27916bae1558bcb118c9ce
volatile.eth0.host_name: veth196022e7
volatile.eth0.hwaddr: 00:16:3e:6b:04:12
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.power: RUNNING
devices:
eth0:
ipv4.address: 172.16.172.116
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: tank
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
Yep, so your eth0 NIC has a static IP defined already. So when you added a proxy in NAT mode with a connect option of tcp:0.0.0.0:7777
will use the static IP to setup the NAT rule.
In order to define a static IPv6 address, you also need to enable stateful DHCPv6 on the LXD managed network (otherwise the container may use SLAAC to generate a random IP). You will get an error to that effect when you try and define a static IPv6 address on a bridged NIC.