Unable to assign secondary ip to lxc container

LXC
1

We are trying to assign a static IP address to the unprivileged lxc container. My host IP is 172.31.25.120/20 and from the same network we want to assign the static IPs to the containers. There is only one physical network interface on the host machine. With LXD, we were able to achieve the same using the network type routed, but with lxc we are facing some issues.

In LXC we tried simulating the routed mode of LXD using network type veth with router mode. In this configuration the veth peer is assigned to the container but the static ip was not assigned to it and no error was thrown in the logs.

Can you provide any suggestions on how we can achieve that?

HostOS: Ubuntu 20.04
LXC Version: 4.0.6

LXC Network Configuration:

lxc.net.0.type = veth
lxc.net.0.veth.mode = router
lxc.net.0.l2proxy = 1
lxc.net.0.link = ens5
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.ipv4.gateway = 169.254.0.1
lxc.net.0.ipv4.address = 172.31.16.243/32 0.0.0.0

Host : ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:b3:2a:d8:aa:9c brd ff:ff:ff:ff:ff:ff
    inet 172.31.25.120/20 brd 172.31.31.255 scope global dynamic ens5
       valid_lft 2025sec preferred_lft 2025sec
    inet6 fe80::4b3:2aff:fed8:aa9c/64 scope link 
       valid_lft forever preferred_lft forever
3: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
       valid_lft forever preferred_lft forever
4: lxdbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:37:12:c6 brd ff:ff:ff:ff:ff:ff
    inet 10.3.116.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:ef8b:160e:5a7::1/64 scope global 
       valid_lft forever preferred_lft forever
8: vethP0pFFz@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether fe:5a:c3:02:db:60 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::fc5a:c3ff:fe02:db60/64 scope link 
       valid_lft forever preferred_lft forever

Host: ip r

default via 172.31.16.1 dev ens5 proto dhcp src 172.31.25.120 metric 100 
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown 
10.3.116.0/24 dev lxdbr0 proto kernel scope link src 10.3.116.1 linkdown 
172.31.16.0/20 dev ens5 proto kernel scope link src 172.31.25.120 
172.31.16.1 dev ens5 proto dhcp scope link src 172.31.25.120 metric 100 
172.31.16.243 dev vethP0pFFz scope link

Container: ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether 62:80:9a:9a:11:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::6080:9aff:fe9a:11e2/64 scope link 
       valid_lft forever preferred_lft forever
2

You probably need to disable netplan or what ever network configuration system is running inside the container as it is likely trying to do DHCP and wiping out the config that LXC applies.

3

Thanks for the reply it is now working, but why did we have to do it explicitly for lxc?

4

Because those same images are used for all network types and all others can support DHCP and default to using DHCP.

1 Like
5

Hi Thomas,

We were able to assign the static IP to the container using router mode, but now we are facing another issue. If we start containers one by one, their networking works fine but if we start the containers in the batch mode (say 5 containers within a second), then sometimes networking didn’t work in couple of them. For these machines IP gets assigned but we are unable to ping from them. If we restart those containers(lxc-stop & lxc-start) then it starts working fine.

Any idea what might be happening ?

6

What does the output of ip a and ip r look like in the problem containers, and what does the ip r and ip neigh show proxy output on the host look like?

7

Host:
ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:eb:19:ba:77:b4 brd ff:ff:ff:ff:ff:ff
    inet 172.31.11.32/20 brd 172.31.15.255 scope global dynamic ens5
       valid_lft 3246sec preferred_lft 3246sec
    inet6 fe80::8eb:19ff:feba:77b4/64 scope link 
       valid_lft forever preferred_lft forever
3: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
       valid_lft forever preferred_lft forever
12: vethaPzv3p@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether fe:2d:02:f6:dd:c0 brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::fc2d:2ff:fef6:ddc0/64 scope link 
       valid_lft forever preferred_lft forever

ip neigh show proxy

172.31.13.66 dev ens5  proxy

ip r

default via 172.31.0.1 dev ens5 proto dhcp src 172.31.11.32 metric 100 
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown 
172.31.0.0/20 dev ens5 proto kernel scope link src 172.31.11.32 
172.31.0.1 dev ens5 proto dhcp scope link src 172.31.11.32 metric 100 
172.31.13.66 dev vethaPzv3p scope link

Container:

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether 56:6d:93:98:67:22 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.31.13.66/32 brd 255.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::546d:93ff:fe98:6722/64 scope link 
       valid_lft forever preferred_lft forever

ip r

default via 169.254.0.1 dev eth0 
169.254.0.1 dev eth0  scope link

On creating the below mentioned entry networking starts working fine inside the container:

Command: ip neigh add proxy 169.254.0.1 dev vethaPzv3p

169.254.0.1 dev vethaPzv3p proxy
8

Ah OK, I vaguely remember hitting an issue with this in the past and never got to the bottom of it, it may be a race in the kernel when using netlink to configure the interface.

The way I worked round it in LXC (and LXD has similar fixes baked in) was to use a hook to re-add the settings on the host as follows:

lxc.net.0.script.up = /usr/share/lxc/hooks/lxc-router-up

/usr/share/lxc/hooks/lxc-router-up:

#!/bin/sh
if [ -z "${LXC_NET_PEER}" ]
then
        echo "LXC_NET_PEER not set"
        exit 1
fi

sysctl net.ipv6.conf."${LXC_NET_PEER}".autoconf=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".accept_dad=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".accept_ra=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".dad_transmits=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".addr_gen_mode=1
ip a flush local dev "${LXC_NET_PEER}" scope link
ip a add fe80::1/64 dev "${LXC_NET_PEER}"
ip a add 169.254.0.1 dev "${LXC_NET_PEER}"

By adding the default gateway IP directly to each host-side interface this avoids the need for the proxy ARP entry on those interfaces and also improves reliability by making the default gateway IP in the containers respond to ICMP pings, ensuring that liveness detection for neighbours works to maintain the ARP neighbour cache.

9

Thanks, this resolved the race condition.

Now we are facing another problem, this is an intermittent issue and happens when we start multiple containers in short span of time. We ran 3 containers out of which 2 started but one failed to start and we got the following error in the lxc log file.

LXC logs

lxc-start c3 20220506140504.178 INFO     network - network.c:instantiate_veth:285 - Retrieved mtu 9001 from ens5 
lxc-start c3 20220506140504.178 ERROR    network - network.c:setup_ipv4_addr_routes:163 - Unknown error -17 - Failed to setup ipv4 address route for network device with eifindex 6
lxc-start c3 20220506140504.178 ERROR    network - network.c:instantiate_veth:422 - Unknown error -17 - Failed to setup ip address routes for network device "vethjRGV0g"
lxc-start c3 20220506140504.237 ERROR    network - network.c:lxc_create_network_priv:3068 - Unknown error -17 - Failed to create network device
lxc-start c3 20220506140504.237 ERROR    start - start.c:lxc_spawn:1786 - Failed to create the network
lxc-start c3 20220506140504.237 DEBUG    network - network.c:lxc_delete_network:3672 - Deleted network devices
lxc-start c3 20220506140504.237 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:859 - Received container state "ABORTING" instead of "RUNNING" 
lxc-start c3 20220506140504.237 TRACE    start - start.c:lxc_serve_state_socket_pair:502 - Sent container state "ABORTING" to 4 
lxc-start c3 20220506140504.237 TRACE    start - start.c:lxc_serve_state_clients:438 - Set container state to ABORTING

/var/log/syslog

May  6 19:35:02 ip-172-31-15-128 systemd-networkd[614]: vethr9qInt: Link UP
May  6 19:35:02 ip-172-31-15-128 networkd-dispatcher[673]: WARNING:Unknown index 5 seen, reloading interface list
May  6 19:35:02 ip-172-31-15-128 systemd-udevd[7342]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
May  6 19:35:02 ip-172-31-15-128 systemd-udevd[7342]: Using default interface naming scheme 'v245'.
May  6 19:35:02 ip-172-31-15-128 kernel: [  258.342684] eth0: renamed from vethaF1e1e
May  6 19:35:02 ip-172-31-15-128 systemd-networkd[614]: vethr9qInt: Gained carrier
May  6 19:35:02 ip-172-31-15-128 kernel: [  258.390156] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 
May  6 19:35:02 ip-172-31-15-128 kernel: [  258.390194] IPv6: ADDRCONF(NETDEV_CHANGE): vethr9qInt: link becomes ready 
May  6 19:35:03 ip-172-31-15-128 kernel: [  258.403713] cgroup: Unknown subsys name 'schedtune'
May  6 19:35:03 ip-172-31-15-128 kernel: [  258.405050] new mount options do not match the existing superblock, will be ignored
May  6 19:35:04 ip-172-31-15-128 systemd-networkd[614]: vethr9qInt: Gained IPv6LL
May  6 19:35:04 ip-172-31-15-128 systemd-networkd[614]: vethjRGV0g: Link UP
May  6 19:35:04 ip-172-31-15-128 systemd-networkd[614]: vethjRGV0g: Link DOWN
May  6 19:35:04 ip-172-31-15-128 networkd-dispatcher[673]: WARNING:Unknown index 6 seen, reloading interface list
May  6 19:35:04 ip-172-31-15-128 systemd-udevd[7659]: vethjRGV0g: Failed to get link config: No such device
May  6 19:35:04 ip-172-31-15-128 networkd-dispatcher[673]: ERROR:Unknown interface index 6 seen even after reload
May  6 19:35:04 ip-172-31-15-128 networkd-dispatcher[673]: WARNING:Unknown index 6 seen, reloading interface list
May  6 19:35:04 ip-172-31-15-128 networkd-dispatcher[673]: ERROR:Unknown interface index 6 seen even after reload
May  6 19:35:04 ip-172-31-15-128 networkd-dispatcher[673]: WARNING:Unknown index 6 seen, reloading interface list
May  6 19:35:04 ip-172-31-15-128 networkd-dispatcher[673]: ERROR:Unknown interface index 6 seen even after reload
May  6 19:35:04 ip-172-31-15-128 networkd-dispatcher[673]: WARNING:Unknown index 6 seen, reloading interface list
May  6 19:35:04 ip-172-31-15-128 networkd-dispatcher[673]: ERROR:Unknown interface index 6 seen even after reload

Host OS: Ubuntu 20.04.3 LTS
LXC Version: 4.0.6
systemd Version: systemd 245 (245.4-4ubuntu3.13)

Host: ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:60:37:be:26:e0 brd ff:ff:ff:ff:ff:ff
    inet 172.31.15.128/20 brd 172.31.15.255 scope global dynamic ens5
       valid_lft 3081sec preferred_lft 3081sec
    inet6 fe80::860:37ff:febe:26e0/64 scope link 
       valid_lft forever preferred_lft forever
3: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr0
       valid_lft forever preferred_lft forever
4: vethVPakWq@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether fe:06:47:aa:48:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.0.1/32 scope global vethVPakWq
       valid_lft forever preferred_lft forever
    inet6 fe80::fc06:47ff:feaa:4881/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
5: vethr9qInt@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether fe:80:18:eb:d3:94 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 169.254.0.1/32 scope global vethr9qInt
       valid_lft forever preferred_lft forever
    inet6 fe80::fc80:18ff:feeb:d394/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever

Host: ip r

default via 172.31.0.1 dev ens5 proto dhcp src 172.31.15.128 metric 100 
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown 
172.31.0.0/20 dev ens5 proto kernel scope link src 172.31.15.128 
172.31.0.1 dev ens5 proto dhcp scope link src 172.31.15.128 metric 100 
172.31.0.254 dev vethVPakWq scope link 
172.31.7.248 dev vethr9qInt scope link

Host: sudo ip neigh show proxy

169.254.0.1 dev vethr9qInt  proxy
169.254.0.1 dev vethVPakWq  proxy
172.31.7.248 dev ens5  proxy
172.31.0.254 dev ens5  proxy

Container1: ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:1c:04:ab:df:7d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.31.0.254/32 brd 172.31.0.254 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::781c:4ff:feab:df7d/64 scope link 
       valid_lft forever preferred_lft forever

Container 2: ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether 76:d3:6d:a1:da:75 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.31.7.248/32 brd 172.31.7.248 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::74d3:6dff:fea1:da75/64 scope link 
       valid_lft forever preferred_lft forever
10

Perhaps something on the host is trying to manage the veth interfaces as they are created?