Google Cloud, Alias IP Ranges and LXC/LXD

1

Hi,

Google uses Alias IP Ranges for their own container and pod hosting, and I did manage to get it to work with LXC/LXD last year but can’t get it to work anymore right now. The intention would be to have a primary subnet for the hosts, and a secondary subnet from which each host will get an Alias IP Range, will host all the containers. Each container should get an internal IP from the secondary subnet/alias IP range, and it should be able to reach other containers inside the secondary subnet, etc. Google takes care of all the routing within the VPC, and the underlying OS automatically gets a route added for the alias IP range already.

I’ve tried all possible combinations from a bridge with or without NAT, macvlan, ipvlan, routed, and so on to get the containers to actually use an IP address from the secondary/alias IP range.

Does anyone have any tips? Hints? Any tutorials around somewhere?

EDIT: this is now resolved. You’ll need to modify the instance_configs.cfg file to disable alias IP ranges on the instance level. Simply creating a nat bridge works fine after this.

2

Can you elaborate on what this looked like or what was needed to make it work? :slight_smile:

3

I run into the same problem when trying to set up bridged networking for LCX containers on Google Compute VM. OP’s post update (thanks!) pointed me into the right direction but there were many more things to realize and put together. I might not understand all pieces yet but here is what I have learned so far.

When IPv4 alias range is assigned to the VM, by default Google guest agent GitHub - GoogleCloudPlatform/guest-agent installs a local route on the VM’s primary network interface.

The route creation can be disabled in /etc/default/instance_configs.cfg → IpForwarding → set ip_aliases to false.

Next we can create a bridge interface using netplan.
Example configuration by using /etc/netplan/10-NAME.yaml file:

network:
  version: 2
  renderer: networkd
  bridges:
    br0:
      addresses:
        - IPv4-alias-range.host/XY
        - IPv6-subnet.host/XYZ
      parameters:
        stp: false
        forward-delay: 0
      optional: true

Note: When using dual stack, /96 IPv6 range is routed to the VM. Unlike IPv4 alias range there is no need for any adjustments to the guest agent. /96 prefix can be spliced into multiple smaller ones. The first one would contain the /128 address from the VM primary interface. Rest of them can be used for the bridge(s).

Of course don’t forget to enable routing in the kernel - net.ipv4.ip_forward=1 and net.ipv6.conf.all.forwarding=1,

Next up configure the LXC container:

lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.ipv4.address = IPv4-alias-range.container/XY
lxc.net.0.ipv4.gateway = IPv4-alias-range.host
lxc.net.0.ipv6.address = IPv6-subnet.container/XYZ
lxc.net.0.ipv6.gateway = IPv6-subnet.host

Dependinging on the system in the container a network management deamon might overwrite the manual network configuration done by LXC. In Debian 12 container I had to disable systemd-resolved (not sure if that is the best way) and reboot it. Afterwards the manual IP configuration worked.

Note: IPv4 alias range is not automatically NATed by Google. This is generally fine for hosting services since there is usually a load balancer in between us and the internet. However we might want to install/update packages within the container. My current solution for that is to just use IPv6. IPv4 NATs be damned :smiley: