How to use a gateway not in same network of the container ip address

LXC
#1

Some big hosting company (OVH, Online.net) use this way to add more public IP for their dedicated servers.

My context is unprivileged LXC 3 containers.

lxc.net.0.type = veth
lxc.net.0.hwaddr = 00:16:3E:Ea:32:a6
lxc.net.0.link = lxcbr15
lxc.net.0.ipv4.address = 1.2.3.4/32
lxc.net.0.ipv4.gateway = 172.16.0.15
lxc.net.0.flags = up

With this configuration I’m not reaching 172.16.0.15 (lxcbr15) from container (1.2.3.4/32) ; it seems because 172.16.0.15 isn’t part of 1.2.3.4/32 … but it’s the right things to do in my case and LXC routes seem to be there.

[vps]$ ip route show
default via 172.16.0.15 dev eth0 
172.16.0.15 dev eth0 scope link
#2

How is your lxcbr15 configured, is it connected to the external network? And can you confirm whether your hosting provider allows multiple MAC addresses on each physical external port?

#3 
[host]$ cat /etc/network/interfaces.d/lxcbr15
auto lxcbr15
iface lxcbr15 inet static
        bridge_ports none
        bridge_fd 0
        bridge_stp off
        bridge_maxwait 0
        address 172.16.0.15/24
        gateway 172.16.0.1

172.16.0.1 is my own path to WAN.

#4

So is the host’s WAN port connected to lxcbr15?

I’m trying to get an idea of your LAN layout, and understand what you’re trying to achieve.

Is your ISP requiring you to do this, or are you doing it out of choice for your network design?

#5

Also can you show output of ip a and ip r on the host please?

#6

I’m trying to give containers a single network device (eth0) with a single IP address (public address from Internet frontend host). It’s for my network design.

I have no problem to route to/from traffic Internet if container’s IP address is in same subnet as container’s bridge. But I need assign public address to container when my LXC hoster computer is in a LAN environment.

#7

I see, so the external IP traffic is already being routed to your LXC host?

In that case I would suggest using the router NIC veth mode, which feels well suited to this, as it allows using external IPs inside a container without the need for a bridge.

See lxc.net.[i].type in https://linuxcontainers.org/lxc/manpages//man5/lxc.container.conf.5.html

So, first, remove the bridge entirely.

Then use this in your container config:

lxc.net.0.type = veth
lxc.net.0.veth.mode = router
lxc.net.0.l2proxy = 1 # This enables proxy ARP/proxy NDP advertisement, not needed if IPs routed to host's LAN address already
lxc.net.0.link = eth1 # External interface (not the bridge)
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.script.up = /usr/share/lxc/hooks/lxc-router-up
lxc.net.0.ipv4.gateway = 169.254.0.1 # This is a link-local next-hop address for IPv4
lxc.net.0.ipv6.gateway = fe80::1 # This is a link-local next-hop address for IPv6
lxc.net.0.ipv4.address = n.n.n.n.n/32 # Single IPv4 address
lxc.net.0.ipv6.address = .../128 # Single IPv6 address

Contents of /usr/share/lxc/hooks/lxc-router-up:

#!/bin/sh
if [ -z "${LXC_NET_PEER}" ]
then
        echo "LXC_NET_PEER not set"
        exit 1
fi

sysctl net.ipv6.conf."${LXC_NET_PEER}".autoconf=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".accept_dad=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".accept_ra=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".dad_transmits=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".addr_gen_mode=1
ip a flush local dev "${LXC_NET_PEER}" scope link
ip a add fe80::1/64 dev "${LXC_NET_PEER}"
ip a add 169.254.0.1 dev "${LXC_NET_PEER}"

You’ll also need to ensure that the OS inside the container doesn’t remove the IPs configured by LXC, such as when triggering DHCP client request. And you’ll need to ensure DNS resolver IPs are configured manually.

#8

Thank you for kind explanation and config details. As I said, my context is LXC 3.0 unprivileged containers, and this version does not seeem to support lxc.net.[i].veth.mode
I need this to work on Debian 10 Stable (LXC 3) and if possible on Debian 9 (LXC 2) too.

#9

Hrm, yes I didn’t see that part.

Well you could use just a normal veth pair, without a link property (that would connect it to a bridge) and instead setup the equivalent settings as router mode does using the various hooks LXC provides.

Basically what router mode does is:

  1. Adds a static route on the host to the container’s IP address pointing towards the container’s veth interface on the host.
  2. If needed, adds proxy ARP/proxy NDP entries ip neigh proxy to advertise the address on the external l2, but I suspect this isn’t needed in your case as the addresses are in different subnet to the LAN and are AFAIK already routed to the host.
  3. Adds link-local addresses to host-side veth interface so the container’s network config can use those addresses as the next-hop address for the default gateway inside the container.
#10

Or you can use LXD which has a snap package that works with Debian and has a NIC type called routed which achieves the same thing.