Veth router mode - why no src in the static route?

I run lxc on a Gentoo host and created a container (Alpine Linux) with the following network setup: = veth = router = vl291 = vl291get = eth0 = up = 02:b6:aa:9b:87:29 = = auto = 1

The host is connected to two networks – ip -br a s up (after container start):

lo               UNKNOWN ::1/128 
vl191            UP             xxx.yyy.zzz.3/24 fe80::6e2b:59ff:feb0:2534/64 
vl291            UP    fe80::6e2b:59ff:feb0:2535/64 
vl291get@if2     UP             fe80::fc91:d2ff:fed6:1400/64 

(xxx.yyy.zzz.3 is a public IP address). A static route is created, by LXC, to the container IP address (as expected), ip r:

default via xxx.yyy.zzz.1 dev vl191 metric 2 
xxx.yyy.zzz.0/24 dev vl191 proto kernel scope link src xxx.yyy.zzz.3 dev vl291 proto kernel scope link src dev vl291get scope link 

However, the static route is missing a src to set the source address – and with my setup ip route get (from host) outputs: dev vl291get src xxx.yyy.zzz.3 uid 0

So when the host communicates with the container it will use the IP address of the other interface (i.e. not the IP address of the interface set by

In my case this makes the connection between host and container non functional – because I also use policy based routing with the following rules (ip rule):

0:	from all lookup local
32764:	from lookup vl291 realms vl291/vl291
32765:	from xxx.yyy.zzz.3 lookup vl191 realms vl191/vl191
32766:	from all lookup main
32767:	from all lookup default

So, the result is that packets from the host to the container will enter into the wrong table, vl191 (instead of vl291). To fix this I ran:

ip route change dev vl291get scope link src

and now:

$ ip r g dev vl291get src uid 0 

To be complete, I also had to add a static route in table vl291 in order to get my setup working (similar to the solution in #7152). Before this change table vl291 looked like this:

$ ip route show table vl291
default via dev vl291 metric 3 dev vl291 scope link src metric 3 

i,e, all packets would go to interface vl291 (including those to – which can also be verified by running:

$ ip r g from from dev vl291 table vl291 realms vl291/vl291 uid 0 

To fix this I added the following static route in table vl291:

ip route add table vl291 dev vl291get src

In order to permanently solve this issue I run the above two ip-route commands in lxc.hook.start-host. There are probably several other, perhaps better, ways to solve it… – but, my main question is if the lack of src address in the static route (added by LXC) is intentional? I would rather expect it to be set to the IP address of the interface defined by

Its not intentional per-se, but then neither would be always adding the source address of the linked parent interface.

However, what may be more appropriate is to set the source address of the inbound route to the same address as is used for the container’s gateway. This way when the host communicates with the container it will appear to be coming from the gateway (which it technically is).

In this case you have = auto which will use the linked parent interfaces IP as gateway, and in that case would then set the source address of the inbound route to the same IP.

In LXD we always add an IP address to the host-side interface of the veth pair (for IPv4 its the link-local and use this as the gateway, and so the source address used for host->container comms is always that IP for host originated traffic. So this issue doesn’t come up AFAIK.

It also has the benefit that the IP address of the host’s external interface doesn’t matter.

In LXC I use a start up hook like this to get a similar behaviour:


if [ -z "${LXC_NET_PEER}" ]
        echo "LXC_NET_PEER not set"
        exit 1

sysctl net.ipv6.conf."${LXC_NET_PEER}".autoconf=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".accept_dad=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".accept_ra=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".dad_transmits=0
sysctl net.ipv6.conf."${LXC_NET_PEER}".addr_gen_mode=1
ip a flush local dev "${LXC_NET_PEER}" scope link
ip a add fe80::1/64 dev "${LXC_NET_PEER}"
ip a add dev "${LXC_NET_PEER}"

And a container NIC config like this: = veth = router = 1 = eth1 = up = eth0 = /usr/share/lxc/hooks/lxc-router-up = = fe80::1 = = 2a02:nnnn:nnnn:1::15/128

You could open an issue over at to request that we add source address of the route based on the gateway setting.

Thanks for your answer and the example – it helped to understand some of the LXD example on this. I see how it could be useful to give the parent interface an IP address in some situations (as in your example). Since I am still experimenting on how to set this up, it would be interesting to hear if you see any obvious disadvantages of not doing that (as I did above)?

I opened #4037 about adding a source address to the static route.

1 Like

Well in your example the gateway inside the container maps to one of the host’s IPs so should be fine.
However I found when adding routed NIC to LXD, that the guest OS would periodically send ICMPv6 packets to the gateway IP for liveness detection, and if just using proxy NDP without the gateway IP actually responding, it would cause periodic packet loss until the NDP resolution occurred again.

1 Like