It seems both network forward and proxy device can achieve same, forwarding a source Ip/port/protocol to a target container/port.
What exactly is the use case for either those?
Whilst proxy device is described sufficiently, the network forward lack of CLI command pattern or examples.
The design doc for this feature may help to illuminate the design rationale behind this feature:
The proxy device is instance specific and can forward connections in either direction, between protocols, and when operating in non Nat mode doesnât require a network connection between host and instance.
A forward on the other hand is more like a proxy device operating in Nat mode. But its defined at the network level rather than instance level and allows sharing an ip between multiple instances because it can forward different ports to different ips.
Thanks.
Was helpful. Just in order to find out the CLI syntax, needed to fast forward to end of it, as a lot of optioned were discussed, tossed or modified.
Seems CLI takes too many parameters, so either edit it in yaml or API would be easier for that.
Is network forward translating the input into iptables rules?
Iâve tried proxy device. It worked fine. Just cant determine where on system those rules happen, as iptables hasnt shown any additional entries.
You can use lxc network forward edit <network> <listen_address> if you prefer.
For bridge networks it adds iptables or nftables depending on the firewall driver being used lxc info | grep -i firewall:
On ovn networks it uses the built in forwarding feature.
Whilst proxy device can define source protocol and target protocol in particular (which can defer), there only is one protocol key in network forward. I guess, its value applies to both, source and target.
Network forward:
tcp <-> tcp
udp <-> udp
Proxy device:
tcp <-> tcp
udp <-> udp
udp <-> tcp
tcp <-> udp
right?
How to achieve this in network forward:
source:53 UDP â target:53 UDP
source:53 TCP â target:53 TCP
Need to create two network forwards with same listen address or only one forward of listen address and multiple ports items as children?
Are the strings âlisten_portâ: â80,81,8080-8090â, and âtarget_portâ: â80,81,8080-8090â mapped by (,)position?
Correct, as it is network (DNAT) based rather than using a proxy process to translate connection types.
To create a port based network forward, lets image we have an IP on the LXD host of 192.168.1.200 and an instance connected to the lxdbr0 network with a static IP of 10.147.102.2/24.
lxc network show lxdbr0
config:
ipv4.address: 10.147.102.1/24
ipv4.nat: "true"
ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
lxc init images:ubuntu/focal c1
lxc config device override c1 eth0 ipv4.address=10.147.102.2
lxc start c1
lxc ls
+------+---------+---------------------+----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+
| c1 | RUNNING | 10.147.102.2 (eth0) | fd42:ab1f:7a24:8e73:216:3eff:fe70:a44 (eth0) | CONTAINER | 0 |
+------+---------+---------------------+----------------------------------------------+-----------+-----------+
Now setup the forward listening on 192.168.1.200 forwarding to 10.147.102.2:
lxc network forward create lxdbr0 192.168.1.200
lxc network forward port add lxdbr0 192.168.1.200 udp 53 10.147.102.2 53
lxc network forward port add lxdbr0 192.168.1.200 tcp 53 10.147.102.2 53
lxc network forward show lxdbr0 192.168.1.200
description: ""
config: {}
ports:
- description: ""
protocol: udp
listen_port: "53"
target_port: "53"
target_address: 10.147.102.2
- description: ""
protocol: tcp
listen_port: "53"
target_port: "53"
target_address: 10.147.102.2
listen_address: 192.168.1.200
location: none
You can also do multi port forwards:
lxc network forward port add lxdbr0 192.168.1.200 tcp 80,81,8080-8090 10.147.102.2
or multi ports to a different set of multi ports:
lxc network forward port add lxdbr0 192.168.1.200 tcp 80,81,8080-8090 10.147.102.2 90,91,9080-9090
or multi ports to a single port:
lxc network forward port add lxdbr0 192.168.1.200 tcp 80,81,8080-8090 10.147.102.2 80
1 Like
Both, proxy device and network forward very interesting features.
I am trying to find out which to apply, as in many cases they are redundant when aiming an IP/ Port/Protocol forward.
Whilst network forward offers a centralized organisation and overview, proxies are hidden in container config and need to be fetched from containers.
At the other hand, as proxy is part of container config, by sudden change of container IP (new Lease or dynamic to static âŚ) it adapts the new situation when target was iface.
Also by stopping or deleting container, the proxy disapears and donât hang around as zombie in network forward.
Yes there is certainly some overlap, especially with bridged networks.
Network forwards can be useful with OVN networks in restricted projects where proxy devices are not allowed, as the administrator can delegate which IPs can be used for network level forwards and and then the project user can setup their own forwards to their instances using only the allowed external IPs.
I would point out that when using proxy device in the more performant nat=true mode it will also require the instanceâs NIC IP to be statically configured in LXD as it uses the same underlying mechanism (firewall dnat) to forward traffic.
In proxy, there is a bind option, how does it work?
bind=<host/instance>
What is recommended as target wildcard?
0.0.0.0 or 127.0.0.1
The proxy docs say:
âWhich side to bind on (host/instance)â
Meaning do you want the listen socket opened on the LXD host side or inside the container?
I donât quite understand what you mean, one address youâve specified is a wildcard address, and another is the local loopback address?
lxc config device add container port80 proxy listen=tcp:0.0.0.0:80 connect=tcp:127.0.0.1:80
lxc config device add container port80 proxy listen=tcp:0.0.0.0:80 connect=tcp:0.0.0.0:80
As target wildcard, both 127.0.0.1 and 0.0.0.0 work fine and seems being interchangeable.
As you mentioned already, proxy device in NAT mode and network forward are gradually same. Though
- proxy is defined in instance config as device, naming is flexible and of course only apply to the container. Not sure it works in profile as well.
- network forward name takes the source ip only, but can handle all sorts of forwards throughout all containers as sets of port definitions.
A non-nat proxy produces an lxd fork main process and at least 7 to 12 child processes per idle device, havent tested whether it increases by ingress/egress). Having numerous containers with each multiple proxy dev will be quite CPU and MEM consuming on Host.
Certainly an L7 or application level proxy supposed to consume, but not sure about min. child processes in idle.
Network forward on the other hand as expected for L3 very humble and performant, just so very limited to OVN and bridges and to my surprise only managed bridges:
Error: Invalid devices: Device validation failed for "eth1": Cannot use manually specified ipv4.address when using unmanaged parent bridge
The iptables entries made by lxd are only based on source IP and container IP, which are given in an unmanaged bridge as well.
- is there a way, forward works for unmanaged bridges as well?
- Is it necessary having a static IP for container? Iptables can as well forward to any bridge address (dhcp pulled incl.).
- Is it possible to forward all ip traffic (all ports) from source to container and vice versa per network forward? This would be an alternative to ipvlan/macvlan/routed/physical.
Something like:
-A PREROUTING -d sourceIP/32 -j DNAT --to-destination containerIP
-A POSTROUTING -s containerIP/32 -j SNAT --to-source sourceIP
These two setups are the same, they both connect to 127.0.0.1.
That is not true, it produces 1 process per non-nat proxy device. What you may be seeing in your process list is threads.
The forkproxy process is very lighweight and consumes little resources.
The error you are seeing is because when specifying a static IP address on a containerâs NIC device, LXD does not actually configure the NIC interface inside the container, but rather creates a static DHCP allocation on its local (managed) DHCP server for the NICâs MAC address, so it always gets the same IP.
For unmanaged bridges, as the name suggests, LXD does not manage the network and so does not provide DHCP or DNS services to it, and without DHCP it cannot create static DHCP allocations, and so specifying a static IP address in LXD config would be meaningless and misleading.
- Currently network forwarding does not work for unmanaged bridges because there is no ânetworkâ concept in the database for an unmanaged parent bridge (as by definition they are unmanaged) and so there is nowhere to store the config for it.
- The instance NIC requires a static endpoint IP otherwise if it changes the forward may go elsewhere.
- Network forwards allow can allow all inbound traffic to be forwarded, and for output traffic on managed bridges you can set
ipv4.nat.addresson the managed bridge to specify the SNAT address for outbound traffic.
However to actually have the container âownâ the IP you can use routed in many scenarios.
Thatâs right. Indeed it might need a managed network. The reason I donât use managed bridge, still is the partly interference with my own iptables rules, and a complex way of dealing with dnsmasq config. Some implemented but getiing passed in same line as -c when starting dnsmasq.
A lot of other dnsmasq central config options/files/hosts/resolve/logs/leases missing, which is an easier way to centrally manage hosts/leases / static leases etc. rather passing it as config to single container.
Didnât understand this really. How do I achieve passing all traffic of a source IP to target IP (container) through network forwards?
Source based forwarding is not available at the moment.
But as the forward in bridged network is a firewall DNAT entry, you can still add additional rules to prohibit certain traffic from using the forward.