News feature 3.0 proxy question

networking

(Chartre) #1

Hello,

In 3.0: “proxy device type to forward network connections” .

I would like to create by this feature a “virtual dns/ip”. Is it possible?

Example:
all 3306 request on my host or cluster will be redirect to haproxy container and in my haproxy config, redirect to galera multi-nodes.

Thanks,


(Stéphane Graber) #2

If I understand you correctly, yes, you can have some IPs on your host that you use for the binding side of the proxy device, which would then let you move the traffic to another container by removing that device from one container and adding it to another.


(Chartre) #3

OK thanks!

lxc config device add haproxy http proxy listen=tcp:0.0.0.0:3306 connect=127.0.0.1:3306
Error: Proxy device doesn't support the connection type: 127.0.0.1

Sorry, but something I don’t understand in command line.

Thanks,


(Chartre) #5

OK, it’s little bit more clear.

It’s like IPtables rules?

I thought that was catch all trafic in LXD lan by 0.0.0.0. I don’t want external trafic.

I will try and maybe ask you again for help.

Thanks,


(Stéphane Graber) #6

The error is because your connect= is missing the tcp: suffix.


(Stéphane Graber) #7
lxc config device add haproxy http proxy listen=tcp:0.0.0.0:3306 connect=tcp:127.0.0.1:3306

(Chartre) #8

Thanks Mr stgraber!

Maybe, in LXD 3.0.0 released post, you miss something?
lxc config device add c1 http proxy listen=tcp:0.0.0.0:80 connect=127.0.0.1:80

Thanks,


#9

My post earlier about the IP addresses was not correct and I erased it.
So LXD 3.0.0 can auto-figure out both the IP address of the host and the IP address of the container.
Much simpler than using iptables for the forwarding TCP connections.


(Stéphane Graber) #10

Oh, oops. I’ve now fixed the announcement.


(Chartre) #11

How I can remove after add?

lxc config device remove haproxy http1 proxy listen=tcp:0.0.0.0:80 connect=127.0.0.1:80
or
lxc config device delete haproxy http1 proxy listen=tcp:0.0.0.0:80 connect=127.0.0.1:80

But:

Error: The device doesn’t exist

How I can delete this rules?

Thanks


(Stéphane Graber) #12

lxc config device remove haproxy http1 would do it. In fact what you ran likely removed the device despite the error. That’s because lxc config device remove can remove multiple devices at once so it tried to remove devices named:

  • http1
  • proxy
  • listen=tcp:0.0.0.0:80
  • connect=127.0.0.1:80

The first one likely succeeeded as it’s an actual device, all the others don’t exist, causing the error.


Publicly Available Containers via Ubuntu 18.04 -- individual website containers
(Express1) #13

This is a really great feature, thanks!
I see a problem here where the source address is hidden.

In the example below I did forward the port 2222 from the LXD host (UN 18.04, LXC 3.01) to port 22 of a container and ssf from outside to the LXD host. Check the last entries from the SSH log -they originate from 127.0.0.1 instead of the real address.
Will be great to have a way to see the originating IP in order to make this feature better.

root@ub18:/var/lib# lxc config device add ub18test1 ssh proxy listen=tcp:0.0.0.0:2222 connect=tcp:127.0.0.1:22
Device ssh added to ub18test1

Jul 13 17:52:28 ub18test1 sshd[384]: error: Received disconnect from 127.0.0.1 port 42752:14: No supported authentication methods
Jul 13 17:52:28 ub18test1 sshd[384]: Disconnected from authenticating user root 127.0.0.1 port 42752 [preauth]


#14

This thing where the service does not know the real source IP is a common issue with load balancers and proxies in general.

Read a good background on this at https://blog.cloudflare.com/mmproxy-creative-way-of-preserving-client-ips-in-spectrum/ where they introduce mmproxy.

Specifically, the OpenSSH software does not support the PROXY protocol that normally solves the problem. That means that you would need mmproxy on top of the PROXY protocol.

What needs to be done, is for LXD to support the PROXY protocol in the proxy device, then specifically for SSHD (OpenSSH) to add the mmproxy TCP proxy that makes SSHD PROXY protocol-aware.

I have added a feature request for LXD at https://github.com/lxc/lxd/issues/4786

You can use now the following workaround (I have not tested this myself):

  1. Install HAProxy on the host and set it up to enable the HAProxy protocol.
  2. Have HAProxy to forward TCP connections to the container with the OpenSSH server.
  3. Install mmproxy in the container of the OpenSSH server as described in the Cloudflare article.

(Express1) #15

If the connection does go thru a proxy the source address of the packet will be the one of the proxy and a supplementary mechanism is necessary - this is how typically a http proxy is set up and you are absolutely right.

Our case is a simple one and I am looking for a simple port forward, not a proxy - so back to a simple solution, using iptables
Host interface ens5, port 2220 forward to
Dest (on bridgel xdbr0), 10.202.82.203:22

sudo iptables -A PREROUTING -p tcp -m tcp --dport 2220 -j DNAT --to-destination 10.202.82.203:22

Jul 16 21:53:12 ub18test1 sshd[1824]: error: Received disconnect from 192.168.1.153 port 52760:14: No supported authentication me
Jul 16 21:53:12 ub18test1 sshd[1824]: Disconnected from authenticating user ubuntu 192.168.1.153 port 52760 [preauth]

Looking at the Container config keys: https://lxd.readthedocs.io/en/latest/containers/
is easy to configure a static IP for the container (ipv4.address)
I do not see a way to run a startup script in order to set the forwarding automatically when the container does start/remove when it does stop - if I remember well it use to be possible in LXC 2 to have startup/shutdown hooks but I do not see them in 3.0 doc above - any idea?

Thank you.


#16

In practice, I did not notice LXD giving a different IP address when a container gets restarted.
However, it is good to use the feature to set a static IP address to a container.

To set the iptables rule, use a recipe like

PORT=2220 CONTAINER_PORT=22 PUBLIC_IP=your_server_ip CONTAINER_IP=your_container_ip \
sudo -E bash -c 'iptables -t nat -I PREROUTING -i eth0 -p TCP -d $PUBLIC_IP --dport $PORT -j DNAT --to-destination $CONTAINER_IP:$CONTAINER_PORT -m comment --comment "forward to a LXD container"'

For the discussion about hooks, see https://github.com/lxc/lxd/issues/3391 (and the referenced issue in there).