LXD Binding multiple static ip via routed network cannot start. Singe ip works

LXD
1

This works

 lxc config device add c1 eth0 nic nictype=routed parent=eth0 ipv4.address=104.1.1.130

This doesn’t start.

 lxc config device add c1 eth0 nic nictype=routed parent=eth0 ipv4.address=104.1.1.130
 lxc config device add c1 eth1 nic nictype=routed parent=eth0 ipv4.address=104.1.1.131

lxd start c1 fails with:

lxc c1 20210117021743.367 ERROR    network - network.c:lxc_network_setup_in_child_namespaces_common:3464 - File exists - Failed to add ipv4 dest "169.254.0.1" for network device "eth1"
lxc c1 20210117021743.367 ERROR    network - network.c:lxc_setup_network_in_child_namespaces:3528 - File exists - Failed to setup netdev
lxc c1 20210117021743.367 ERROR    conf - conf.c:lxc_setup:3295 - Failed to setup network
lxc c1 20210117021743.367 ERROR    start - start.c:do_start:1218 - Failed to setup container "c1"
lxc c1 20210117021743.367 ERROR    sync - sync.c:__sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc c1 20210117021743.367 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:860 - Received container state "ABORTING" instead of "RUNNING"
lxc c1 20210117021743.369 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "c1"
lxc c1 20210117021743.369 WARN     start - start.c:lxc_abort:1013 - No such process - Failed to send SIGKILL via pidfd 30 for process 2884210
lxc 20210117021743.670 WARN     commands - commands.c:lxc_cmd_rsp_recv:126 - Connection reset by peer - Failed to receive response for command "get_state"

Anyone got any idea? The main error appears to be Failed to add ipv4 dest "169.254.0.1" for network device "eth1"

On LXD snap 4.10

2

You are adding multiple nic devices rather than one nic device with multiple ips. I recommend just using one nic and using a comma delimited list of ips instead if parent is the same. Otherwise the automatic default gateway route added will conflict.

1 Like
3

You can specify ipv4.gateway=none on one of the NICs to avoid multiple default routes being added although you will likely face issues with asymmetric routing in your setup.

See https://linuxcontainers.org/lxd/docs/master/instances#nic-routed

4

Thank you. Didn’t realize it actually accepted comma delimited list!. =)

5

We hit a user-case bug, not a design bug, when using the comma delimited feature.

6

I’ve added duplicate address detection to the routed NIC device. Although I was not able to reproduce the network storm you reported. However I did notice that if duplicate IPs were specified, then the container would fail to start because it could not add the duplicate static routes on the host (as expected). This then failed to clean up the IP neighbour proxy entries it added on the host, so perhaps that may have caused the issue you experienced (as it would have advertised the IPs but not routed them anywhere).

I’ve added a fix to clean up those entries on failed start now too.