IPv6 device preventing container from starting

Maran · May 26, 2020, 11:31am

I’m unsure what I did, if anything, but from one day to the next my lxd container won’t boot as long as my routed network for IPv6 is in my container configuration.

I have a config that specifies a eth device for IPv6 traffic.

eth1:
ipv6.address: 2a0a:7000:1337:c:d282:4784:6ad9:7f9d,2a0a:7000:1337:c:be54:955a:33b:cdc2
nictype: routed
parent: enp216s0
type: nic

I restarted my container today after having it run for a week or two (I did restart it before) and all of a sudden it won’t start. It will start if I remove this configuration.

My lxc logs that seem relevant.

lxc tachikoma 20200526112458.397 DEBUG lxc tachikoma 20200526112458.402 TRACE lxc tachikoma 20200526112458.407 TRACE lxc tachikoma 20200526112458.407 TRACE lxc tachikoma 20200526112458.407 NOTICE lxc tachikoma 20200526112458.408 WARN lxc tachikoma 20200526112458.409 DEBUG lxc tachikoma 20200526112458.409 TRACE lxc tachikoma 20200526112458.411 DEBUG lxc tachikoma 20200526112458.412 INFO lxc tachikoma 20200526112458.413 ERROR lxc tachikoma 20200526112458.413 ERROR lxc tachikoma 20200526112458.458 ERROR lxc tachikoma 20200526112458.458 ERROR lxc tachikoma 20200526112458.460 TRACE lxc tachikoma 20200526112458.460 TRACE lxc tachikoma 20200526112458.460 DEBUG lxc tachikoma 20200526112458.460 TRACE lxc tachikoma 20200526112458.460 TRACE lxc tachikoma 20200526112458.460 TRACE lxc tachikoma 20200526112458.460 ERROR conf - conf.c:lxc_map_ids:2710 - Functional newuidmap and newgidmap binary found
conf - conf.c:lxc_map_ids:2780 - newuidmap wrote mapping “newuidmap 52923 65536 0 1 0 100000 65536”
conf - conf.c:lxc_map_ids:2780 - newgidmap wrote mapping “newgidmap 52923 65536 0 1 0 100000 65536”
conf - conf.c:run_userns_fn:3857 - Calling function “chown_cgroup_wrapper”
utils - utils.c:lxc_setgroups:1366 - Dropped additional groups
cgfsng - cgroups/cgfsng.c:fchowmodat:1455 - No such file or directory - Failed to fchownat(17, memory.oom.group, 65536, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
start - start.c:lxc_spawn:1737 - Preserved net namespace via fd 4
start - start.c:lxc_spawn:1744 - Allocated new network namespace id
network - network.c:instantiate_phys:828 - Instantiated phys “veth11bb0ca8” with ifindex is “136”
network - network.c:instantiate_veth:290 - Retrieved mtu 1500 from enp216s0
network - network.c:setup_ipv6_addr_routes:179 - Unknown error -17 - Failed to setup ipv6 address route for network device with eifindex 138
network - network.c:instantiate_veth:430 - Unknown error -17 - Failed to setup ip address routes for network device “vethf11de550”
network - network.c:lxc_create_network_priv:3096 - Unknown error -17 - Failed to create network device
start - start.c:lxc_spawn:1750 - Failed to create the network
network - network.c:lxc_delete_network_priv:3218 - Renamed interface with index 0 from “eth0” to its initial name “veth11bb0ca8”
network - network.c:lxc_delete_network_priv:3227 - Restored interface “veth11bb0ca8” to its initial mtu “1450”
network - network.c:lxc_delete_network:3693 - Deleted network devices
start - start.c:lxc_serve_state_socket_pair:492 - Sent container state “ABORTING” to 7
start - start.c:lxc_serve_state_clients:427 - Set container state to ABORTING
start - start.c:lxc_serve_state_clients:430 - No state clients registered
lxccontainer - lxccontainer.c:wait_on_daemonized_start:852 - Received container state “ABORTING” instead of “RUNNING”

Does anybody know what could have changed on the server that prevents this from working all of a sudden? The bridged ethernet device still does exist on the host and has the same name. Before I restarted the container IPv6 was working fine.

This is on 4.0.0.

tomp · May 26, 2020, 2:55pm

Can you show the full output of lxc config show <container> --expanded please.

The error:

lxc tachikoma 20200526112458.413 ERROR network - network.c:setup_ipv6_addr_routes:179 - Unknown error -17 - Failed to setup ipv6 address route for network device with eifindex 138

Suggests you may have a route either on the host or the container that conflicts with what its trying to setup.

Maran · May 26, 2020, 3:10pm

Sure thing: https://paste.ubuntu.com/p/Wd43Wjyw3N/

tomp · May 26, 2020, 3:11pm

Ah, there’s the likely issue, you have two NIC devices in your container, one is a fan and the other is routed. Both will be trying to add a default route.

Which one do you want to use as your default route?

Maran · May 26, 2020, 3:13pm

I think I prefer the IPv4 as the default route which is on the fanned one.

What could I have done on the host to break this all of a sudden? It’s been working fine for the past month or so.

tomp · May 26, 2020, 3:17pm

Its probably racing each other, if routed completes first its ok, if fan then routed breaks.

Try adding ipv4.gateway=none on the routed NIC, so that it only has the IPv6 default gateway setup.

See https://linuxcontainers.org/lxd/docs/master/instances#nictype-routed

Maran · May 27, 2020, 6:49am

Seems it still does not like it.

~# lxc config show tachikoma --expanded | grep eth -A 5
  eth0:
    name: eth0
    network: lxdfan0
    type: nic
  eth1:
    ipv4.gateway: none
    ipv6.address: 2a0a:7000:1337:c:d282:4784:6ad9:7f9d
    nictype: routed
    parent: enp216s0
    type: nic

When starting it

~# lxc start tachikoma
Error: Not a device
Try `lxc info --show-log tachikoma` for more info

lxc info --show-log tachikoma: https://paste.ubuntu.com/p/f38s8MDtmB/

tomp · May 27, 2020, 7:15am

The “Not a device” error is unlikely to be coming from the NICs, as that is related to Unix char devices.

If you remove the routed NIC and start the container up can you show the output of:

ip a
ip r

inside the container please.

Maran · May 27, 2020, 7:24am

Here it is

root@tachikoma:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:26:0a:39:60 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
152: eth0@if153: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:bc:e8:58 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 240.8.0.98/8 brd 240.255.255.255 scope global dynamic eth0
       valid_lft 3594sec preferred_lft 3594sec
    inet6 fe80::216:3eff:febc:e858/64 scope link
       valid_lft forever preferred_lft forever
root@tachikoma:~# ip r
default via 240.8.0.1 dev eth0 proto dhcp src 240.8.0.98 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
240.0.0.0/8 dev eth0 proto kernel scope link src 240.8.0.98
240.8.0.1 dev eth0 proto dhcp scope link src 240.8.0.98 metric 100

tomp · May 27, 2020, 7:42am

Apologies I meant ip -6 r

Maran · May 27, 2020, 7:43am

root@tachikoma:~# ip -6 r
fe80::/64 dev eth0 proto kernel metric 256 pref medium

tomp · May 27, 2020, 7:58am

Thanks, so no default ipv6 gateway, thats good.

Can you show the output of ip -6 r on your host OS too please.

I suspect the error “Failed to setup ipv6 address route…” is caused by an existing conflicting route existing on your host.

Maran · May 27, 2020, 8:06am

Sure thing.

procyon:~# ip -6 r
::1 dev lo proto kernel metric 256 pref medium
2a0a:7000:1337::/63 dev enp216s0 proto kernel metric 256 pref medium
2a0a:7000:1337:b:2189:a94e:f533:ea2 dev veth481cf7f7 metric 1024 pref medium
2a0a:7000:1337:c:d282:4784:6ad9:7f9d dev vetha83778e7 metric 1024 pref medium
2a0a:7000:1337:d:bf7:286:f356:4445 dev veth6516c916 metric 1024 pref medium
2a0a:7000:1337:e:9766:e149:5912:3d7c dev veth4f13cc40 metric 1024 pref medium
2a0a:7000:1337:f:e191:d142:69ad:5fa5 dev veth810f7c6b metric 1024 pref medium
2a0a:7000:1337:10:5a0b:da57:e312:a48f dev veth723c5e0f metric 1024 pref medium
2a0a:7000:1337:12:436c:feee:7c80:f887 dev vethea3c41f0 metric 1024 pref medium
2a0a:7000:1337:13:3f09:aa03:f556:6fde dev veth31cd5ba1 metric 1024 pref medium
2a0a:7000:1337:15:7cf1:fecb:ade:e6b8 dev veth6ecd25c5 metric 1024 pref medium
2a0a:7000::/29 dev enp216s0 proto ra metric 1024 pref medium
fe80::1 dev veth481cf7f7 proto kernel metric 256 pref medium
fe80::1 dev vetha83778e7 proto kernel metric 256 pref medium
fe80::1 dev veth6516c916 proto kernel metric 256 pref medium
fe80::1 dev veth4f13cc40 proto kernel metric 256 pref medium
fe80::1 dev veth810f7c6b proto kernel metric 256 pref medium
fe80::1 dev veth723c5e0f proto kernel metric 256 pref medium
fe80::1 dev vethea3c41f0 proto kernel metric 256 pref medium
fe80::1 dev veth31cd5ba1 proto kernel metric 256 pref medium
fe80::1 dev veth6ecd25c5 proto kernel metric 256 pref medium
fe80::/64 dev enp216s0 proto kernel metric 256 pref medium
fe80::/64 dev lxdfan0 proto kernel metric 256 pref medium
fe80::/64 dev lxdfan0-mtu proto kernel metric 256 pref medium
fe80::/64 dev lxdfan0-fan proto kernel metric 256 pref medium
fe80::/64 dev veth481cf7f7 proto kernel metric 256 pref medium
fe80::/64 dev vetha83778e7 proto kernel metric 256 pref medium
fe80::/64 dev veth6516c916 proto kernel metric 256 pref medium
fe80::/64 dev veth4f13cc40 proto kernel metric 256 pref medium
fe80::/64 dev veth810f7c6b proto kernel metric 256 pref medium
fe80::/64 dev veth723c5e0f proto kernel metric 256 pref medium
fe80::/64 dev vethea3c41f0 proto kernel metric 256 pref medium
fe80::/64 dev veth31cd5ba1 proto kernel metric 256 pref medium
fe80::/64 dev veth6ecd25c5 proto kernel metric 256 pref medium
default via fe80::21c:73ff:fe00:99 dev enp216s0 proto ra metric 1024 mtu 1500 pref medium

tomp · May 27, 2020, 8:12am

Yep there is the conflicting route:

2a0a:7000:1337:c:d282:4784:6ad9:7f9d dev vetha83778e7 metric 1024 pref medium

Maran · May 27, 2020, 8:15am

Can I figure which container that veth belongs to? Is it possible it’s the veth device from the tachikoma container that just was not cleaned up?

tomp · May 27, 2020, 8:21am

The host-side veth interface should be removed by the kernel when the container is stopped, which should also remove any static routes pointing to it. Its possible there is some sort of orphaned veth pair that is preventing the route from being removed.

Do you have any other containers running?

tomp · May 27, 2020, 8:21am

Can you show output of ip a on the host please

Maran · May 27, 2020, 8:23am

There are a bunch of other containers running but all of IPv6 host bits are randomly generated so a collision should be near impossible.

root@procyon:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ac:1f:6b:b0:97:d8 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ac:1f:6b:b0:97:d9 brd ff:ff:ff:ff:ff:ff
4: enp216s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 7c:fe:90:a2:45:c1 brd ff:ff:ff:ff:ff:ff
    inet 185.162.186.8/22 brd 185.162.187.255 scope global enp216s0
       valid_lft forever preferred_lft forever
    inet6 2a0a:7000:1337::2/63 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::7efe:90ff:fea2:45c1/64 scope link
       valid_lft forever preferred_lft forever
5: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 06:e7:a2:fe:ed:9b brd ff:ff:ff:ff:ff:ff
    inet 240.8.0.1/8 scope global lxdfan0
       valid_lft forever preferred_lft forever
    inet6 fe80::5c8f:69ff:feac:b40e/64 scope link
       valid_lft forever preferred_lft forever
8: veth7781a0b0@veth6db9a519: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:16:3e:02:e2:ae brd ff:ff:ff:ff:ff:ff
9: veth6db9a519@veth7781a0b0: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1450 qdisc noqueue master lxdfan0 state LOWERLAYERDOWN group default qlen 1000
    link/ether fe:61:5f:55:65:24 brd ff:ff:ff:ff:ff:ff
45: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether 0a:b9:5b:61:39:06 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8b9:5bff:fe61:3906/64 scope link
       valid_lft forever preferred_lft forever
46: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether c6:a2:be:64:69:10 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c4a2:beff:fe64:6910/64 scope link
       valid_lft forever preferred_lft forever
66: veth568b9f2a@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether a6:b5:6b:92:7a:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
67: veth481cf7f7@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:d6:32:8d:56:5a brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fcd6:32ff:fe8d:565a/64 scope link
       valid_lft forever preferred_lft forever
70: vetha83778e7@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:b4:18:c6:e7:6a brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fcb4:18ff:fec6:e76a/64 scope link
       valid_lft forever preferred_lft forever
72: vethe982f5ea@if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether 8e:e6:89:ae:c6:4a brd ff:ff:ff:ff:ff:ff link-netnsid 3
73: veth6516c916@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:ff:52:78:ed:5d brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fcff:52ff:fe78:ed5d/64 scope link
       valid_lft forever preferred_lft forever
75: veth4afd73ff@if74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether ea:2e:8e:f6:06:aa brd ff:ff:ff:ff:ff:ff link-netnsid 4
76: veth4f13cc40@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:bf:cb:18:95:e9 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fcbf:cbff:fe18:95e9/64 scope link
       valid_lft forever preferred_lft forever
79: veth810f7c6b@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:aa:a9:db:ad:18 brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fcaa:a9ff:fedb:ad18/64 scope link
       valid_lft forever preferred_lft forever
84: veth12db08f3@if83: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether 06:e7:a2:fe:ed:9b brd ff:ff:ff:ff:ff:ff link-netnsid 6
85: veth723c5e0f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:20:85:8d:9e:7c brd ff:ff:ff:ff:ff:ff link-netnsid 6
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fc20:85ff:fe8d:9e7c/64 scope link
       valid_lft forever preferred_lft forever
90: veth85d80687@if89: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether aa:3a:2e:2c:ef:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 7
91: vethea3c41f0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:9e:96:48:8d:01 brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fc9e:96ff:fe48:8d01/64 scope link
       valid_lft forever preferred_lft forever
93: veth39bff917@if92: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether c6:ba:07:b8:78:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 8
94: veth31cd5ba1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:52:37:5d:1a:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 8
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fc52:37ff:fe5d:1aa3/64 scope link
       valid_lft forever preferred_lft forever
114: veth113764e8@if113: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether 4a:2f:43:2a:14:33 brd ff:ff:ff:ff:ff:ff link-netnsid 9
115: veth6ecd25c5@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:17:bf:2c:bd:94 brd ff:ff:ff:ff:ff:ff link-netnsid 9
    inet6 fe80::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::fc17:bfff:fe2c:bd94/64 scope link
       valid_lft forever preferred_lft forever
153: veth7bbc4bd2@if152: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether 16:6c:57:c4:14:3b brd ff:ff:ff:ff:ff:ff link-netnsid 10

tomp · May 27, 2020, 8:27am

Try doing ip link delete vetha83778e7

Maran · May 27, 2020, 8:29am

Awesome that solved it. Glad I know how to handle this next time. Although I hope to figure out what is causing this to begin with. Thank you for your assistance.