OVN with LXD cluster flood log with multiple assigned dynamic IPv6 address

LXD
, , ,
1

Hi,

I run into several problems when set up ovn with lxd cluster.

My current environment:

  • 2 Raspberry Pi 4 / 8 GB
  • Debain 11
  • LXD Version 5.3
  • Open vSwitch Version 2.15
  • OVN Version 21.06

The LXD Cluster has yet two server instances and run’s smoothly.

I follow this tutorial to set up OVN: How to set up OVN with LXD - LXD documentation. As parent=<uplink_interface> I use the physical ethernet bridge interface.

What works well:

  • add the ovn network to containers
  • ping container on several server instances
  • reach a website over a reverse proxy inside a second container over IP

What’s curious:

After adding the ovn network to a container and start the container, a flood of log messages appear in /var/log/ovn/ovn-northd.log.

  • ovn_northd|INFO|Assigned dynamic IPv6 address 'fd42:306f:4495:374a:216::d7fd' to port 'lxd-net15-instance-48221a04-36f0--eth0' nearly every second
  • ipam|WARN|16f52932-704d--e4ca3f3694d4: Duplicate IP set: 10.17.192.2
  • ipam|WARN|Dropped 52 log messages in last 60 seconds (most recently, 4 seconds ago) due to excessive rate

Is this related to LXD? Anybody an idea how to resolve?

Thanks!

2

Hi,

Yes we have seen that ourselves too.
It doesn’t appear to cause any problems, but seems like OVN bugs.

I’ve not been able to find out why OVN thinks there is a duplicate IP, as looking in the OVN DB there isn’t a duplicate IP for multiple ports defined.

I see this when an instance starts on Ubuntu 22.04:

2022-07-21T07:24:24.685Z|00039|northd|INFO|Assigned dynamic IPv4 address '10.173.122.2' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.685Z|00040|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.689Z|00041|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.689Z|00042|ipam|WARN|Dropped 18 log messages in last 67 seconds (most recently, 8 seconds ago) due to excessive rate
2022-07-21T07:24:24.689Z|00043|ipam|WARN|071b0d1d-4234-4f4e-809c-81c8659264aa: Duplicate IP set: 10.173.122.2
2022-07-21T07:24:24.690Z|00044|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.702Z|00045|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.704Z|00046|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.705Z|00047|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.708Z|00048|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.710Z|00049|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.713Z|00050|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.716Z|00051|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.718Z|00052|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.719Z|00053|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.733Z|00054|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.737Z|00055|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:24.738Z|00056|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'
2022-07-21T07:24:33.567Z|00057|northd|INFO|Assigned dynamic IPv6 address 'fd42:1907:ee48:b90f:216:3eff:fe50:ce86' to port 'lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0'

But after that it stops.

And the OVN processes are not using high CPU, so doesn’t look like its stuck in a loop.

And as you can see there are no duplicate port IP assignments (as there is only one NIC port):

ovn-nbctl list logical_switch_port
_uuid               : d7053ac8-20e0-4de7-88b5-6e02638f6691
addresses           : ["00:16:3e:50:ce:86 dynamic"]
dhcpv4_options      : 53891d15-3d46-4f86-8985-1b3c0a785e5f
dhcpv6_options      : 58e2c126-df56-48c9-94ec-dd869aa9d2a6
dynamic_addresses   : "00:16:3e:50:ce:86 10.173.122.2 fd42:1907:ee48:b90f:216:3eff:fe50:ce86"
enabled             : []
external_ids        : {}
ha_chassis_group    : []
name                : lxd-net39-instance-0135fe8d-fbf2-4f36-933b-469f09aaad0a-eth0
options             : {}
parent_name         : []
port_security       : []
tag                 : []
tag_request         : []
type                : ""
up                  : true

_uuid               : 397ce96c-ada9-4855-836f-724f3afe4f17
addresses           : [router]
dhcpv4_options      : []
dhcpv6_options      : []
dynamic_addresses   : []
enabled             : []
external_ids        : {}
ha_chassis_group    : []
name                : lxd-net39-ls-int-lsp-router
options             : {nat-addresses=router, router-port=lxd-net39-lr-lrp-int}
parent_name         : []
port_security       : []
tag                 : []
tag_request         : []
type                : router
up                  : true

_uuid               : f039d8d2-7cc1-4912-9e23-2b2678d9a375
addresses           : [router]
dhcpv4_options      : []
dhcpv6_options      : []
dynamic_addresses   : []
enabled             : []
external_ids        : {}
ha_chassis_group    : []
name                : lxd-net39-ls-ext-lsp-router
options             : {nat-addresses=router, router-port=lxd-net39-lr-lrp-ext}
parent_name         : []
port_security       : []
tag                 : []
tag_request         : []
type                : router
up                  : true

_uuid               : 79c23cf7-5b15-4f7a-841c-2f09ec5427f7
addresses           : [unknown]
dhcpv4_options      : []
dhcpv6_options      : []
dynamic_addresses   : []
enabled             : []
external_ids        : {}
ha_chassis_group    : []
name                : lxd-net39-ls-ext-lsp-provider
options             : {network_name=lxdbr0}
parent_name         : []
port_security       : []
tag                 : []
tag_request         : []
type                : localnet
up                  : false

For the IPv6 log messages I would like to change our implementation to just statically assign the EUI64 address to the port (as this is all the dynamic assignment does anyway), which gets rid of the log message. However there is a limitation in OVN that you cannot specify a port has a dynamic IPv4 address assigned and a static IPv6 address - its both dynamic or both static. And if we want both static then we would need to implement IPAM in LXD. This isn’t something we are fundamentally adverse to (we are already doing it somewhat for bridged networks when NICs use security.ipv{n}_filtering=true) but its not something we’ve gotten around to doing yet.

3

Hi,

thanks for the fast reply. To just statically assign the EUI64 address to the port, I add a static IPv4 address for each related container with:

lxc config device set strand2 eth0 ipv4.address 10.17.192.2

For me it seems to be the solution. Or is there a overall solution, not to assign a static ip for each container manually?

Thanks

:heart: lxd

4

You can set the IP manually like that and its fine. :slight_smile:
Thanks for confirming it resolves the log messages.