Does OVN work on debian?

bodleytunes · August 31, 2020, 12:39pm

Hi,

I thought I’d have a go with the OVN networking, but I’m testing on a debian proxmox 6 server which is running LXD 4.5 on snap.

I’ve gone through the motions of setting up an ovn network but getting an error:

Error: Failed to run: ovn-nbctl --db unix:/var/lib/snapd/hostfs/run/ovn/ovnnb_db.sock ha-chassis-group-add lxd-net22: ovn-nbctl: unix:/var/lib/snapd/hostfs/run/ovn/ovnnb_db.sock: database connection failed (No such file or directory)

This is what I did to get to this point:

apt install ovn-host ovn-central
apt install openvswitch-switch
ovs-vsctl set open_vswitch .   external_ids:ovn-remote=unix:/var/run/ovn/ovnsb_db.sock   external_ids:ovn-encap-type=geneve   external_ids:ovn-encap-ip=10.55.0.20
lxc network create lxdbr200 ipv4.address=10.200.0.1/24 bridge.driver=openvswitch
lxc network set lxdbr200 ipv4.dhcp.ranges=10.200.0.10-10.200.0.200
lxc network set lxdbr200 ipv4.ovn.ranges=10.200.0.201-10.200.0.254
lxc network create br-ovn1 network=lxdbr200 --type=ovn

Many thanks,

Jon.

stgraber · September 1, 2020, 4:33am

Does /run/ovn/ovnnb_db.sock exist on your system?
The error suggests it may not, or it may be a symlink?

@tomp in the event where it’s a symlink, we may need to replace an invocation from shared.HostPath to shared.HostPathFollow to deal with this.

tomp · September 1, 2020, 7:30am

Will take a look.

bodleytunes · September 1, 2020, 8:10am

Hi
so I was looking around this area it looks like its in a different location… I was messing around with symlinks but couldn’t get working.

root@p20:/etc/openvswitch# ls -lahst
total 56K
4.0K drwxr-xr-x 118 root root 4.0K Sep  1 07:00 ..
4.0K drwxr-xr-x   2 root root 4.0K Aug 31 15:02 .
 16K -rw-r--r--   1 root root  16K Aug 31 15:02 conf.db
 16K -rw-r--r--   1 root root  14K Aug 31 12:43 ovnsb_db.db
 12K -rw-r--r--   1 root root  10K Aug 31 12:11 ovnnb_db.db
   0 -rw-------   1 root root    0 Aug 31 12:11 .ovnsb_db.db.~lock~
   0 -rw-------   1 root root    0 Aug 31 12:11 .ovnnb_db.db.~lock~
4.0K -rw-r--r--   1 root root   37 May 18 09:39 system-id.conf
   0 lrwxrwxrwx   1 root root   36 May 18 09:39 .conf.db.~lock~ -> /var/lib/openvswitch/.conf.db.~lock~

bodleytunes · September 1, 2020, 8:54am

bodleytunes · September 1, 2020, 8:55am

looks like /run/openvswitch/ovnnb_db.sock

tomp · September 1, 2020, 9:34am

Yes the Debian packages setup the unix socket at /var/run/openvswitch/ovnnb_db.sock

You can specify LXD use this by setting:

lxc config set network.ovn.northbound_connection=unix:/var/run/openvswitch/ovnnb_db.sock

bodleytunes · September 1, 2020, 9:35am

Great, will give it a try! thanks

bodleytunes · September 1, 2020, 9:38am

Got further!
Now its saying the database needs upgrade?

This is possibly because proxmox uses an older openvswitch I’ll look into this.

root@p20:/home/debian# ovs-vsctl --version
ovs-vsctl (Open vSwitch) 2.12.0
DB Schema 8.0.0
root@p20:/home/debian# dpkg -l | grep -i openv
ii  openvswitch-common                   2.12.0-1                        amd64        Open vSwitch common components
ii  openvswitch-switch                   2.12.0-1                        amd64        Open vSwitch switch implementations

tomp · September 1, 2020, 9:42am

Please paste the error you are getting. Although its likely the OVN controller provided in Debian is too old (we actually need a very recent version for full functionality we need, even currently beyond that which is in Ubuntu Focal, although the one in Groovy is sufficient - mostly this is to do with limited IPv6 support in older versions of OVN).

bodleytunes · September 1, 2020, 9:43am

Error: Failed to run: ovn-nbctl --db unix:/var/lib/snapd/hostfs/run/openvswitch/ovnnb_db.sock lr-nat-add lxd-net27-lr snat 10.200.0.201 10.71.251.0/24: 2020-09-01T09:40:03Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks Forwarding_Group table (database needs upgrade?)
2020-09-01T09:40:03Z|00003|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks health_check column (database needs upgrade?)
2020-09-01T09:40:03Z|00004|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks ip_port_mappings column (database needs upgrade?)
2020-09-01T09:40:03Z|00005|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks selection_fields column (database needs upgrade?)
2020-09-01T09:40:03Z|00006|ovsdb_idl|WARN|OVN_Northbound database lacks Load_Balancer_Health_Check table (database needs upgrade?)
2020-09-01T09:40:03Z|00007|ovsdb_idl|WARN|Logical_Router_Policy table in OVN_Northbound database lacks external_ids column (database needs upgrade?)
2020-09-01T09:40:03Z|00008|ovsdb_idl|WARN|Logical_Router_Port table in OVN_Northbound database lacks ipv6_prefix column (database needs upgrade?)
2020-09-01T09:40:03Z|00009|ovsdb_idl|WARN|Logical_Switch table in OVN_Northbound database lacks forwarding_groups column (database needs upgrade?)
2020-09-01T09:40:03Z|00010|ovsdb_idl|WARN|NAT table in OVN_Northbound database lacks external_port_range column (database needs upgrade?)
2020-09-01T09:40:03Z|00011|ovsdb_idl|WARN|NAT table in OVN_Northbound database lacks options column (database needs upgrade?)
2020-09-01T09:40:03Z|00012|ovsdb_idl|WARN|NB_Global table in OVN_Northbound database lacks name column (database needs upgrade?)
2020-09-01T09:40:03Z|00013|ovsdb_idl|WARN|transaction error: {"details":"No column options in table NAT.","error":"unknown column","syntax":"{\"external_ip\":\"10.200.0.201\",\"logical_ip\":\"10.71.251.0/24\",\"options\":[\"map\",[[\"stateless\",\"false\"]]],\"type\":\"snat\"}"}
ovn-nbctl: transaction error: {"details":"No column options in table NAT.","error":"unknown column","syntax":"{\"external_ip\":\"10.200.0.201\",\"logical_ip\":\"10.71.251.0/24\",\"options\":[\"map\",[[\"stateless\",\"false\"]]],\"type\":\"snat\"}"}

tomp · September 1, 2020, 9:49am

Yes so that is the ovn-nbctl command inside the LXD snap trying to configure the outbound SNAT rules to allow the private OVN networks to have external internet access using the designated parent network.

In this case your OVN controller database looks like it is missing the NAT functionality.

At this time you cannot turn off the NAT functionality so this looks like a limitation. The OVN NAT functionality was added in OVN 2.9, and looking at LXD’s debian/buster images it includes OVN 2.10, so should be sufficient.

What version of OVN do you have:

dpkg -l | grep ovn

bodleytunes · September 1, 2020, 10:06am

root@p20:/home/debian# dpkg -l | grep ovn
ii  novnc-pve                            1.1.0-1                         all          HTML5 VNC client
ii  ovn-central                          2.12.0-1                        amd64        OVN central components
ii  ovn-host                             2.12.0-1                        amd64        OVN host components

tomp · September 1, 2020, 10:14am

Well it looks recent enough, I’m not sure, will have to spend some time trying it on Debian, but it looks like the OVN northbound database is missing tables for some reason. Certainly not something I’ve seen before.

bodleytunes · September 1, 2020, 10:15am

The only other spanner in the works is that its proxmox and they often use their own versions of packages, but I think in this case they don’t touch ovn as its not part of their usual network toolset.

I’ll also have a go on a vanilla debian and see if any diffferent.

tomp · September 1, 2020, 10:17am

Running ovn-nbctl list nat should show an empty output with no errors, meaning the table exists but is empty (which it does on an empty install of debian buster).

bodleytunes · September 1, 2020, 10:17am

empty output no errors.

tomp · September 1, 2020, 10:19am

So perhaps the issue is with the OVN southbound databaseor something to do with OVS itself (the specific error logs you pasted suggest this too). I’ll try and find some time to try it on Debian.

tomp · September 1, 2020, 1:37pm

I’ve tried it on Debian Buster and get similar error:

root@v1:~# lxc network create ovn --type=ovn network=lxdbr0
Error: Failed to run: ovn-nbctl --db unix:/var/lib/snapd/hostfs/run/openvswitch/ovnnb_db.sock ha-chassis-group-add lxd-net4: 2020-09-01T13:25:27Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks Forwarding_Group table (database needs upgrade?)
2020-09-01T13:25:27Z|00003|ovsdb_idl|WARN|OVN_Northbound database lacks HA_Chassis table (database needs upgrade?)
2020-09-01T13:25:27Z|00004|ovsdb_idl|WARN|OVN_Northbound database lacks HA_Chassis_Group table (database needs upgrade?)
2020-09-01T13:25:27Z|00005|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks health_check column (database needs upgrade?)
2020-09-01T13:25:27Z|00006|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks ip_port_mappings column (database needs upgrade?)
2020-09-01T13:25:27Z|00007|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks selection_fields column (database needs upgrade?)
2020-09-01T13:25:27Z|00008|ovsdb_idl|WARN|OVN_Northbound database lacks Load_Balancer_Health_Check table (database needs upgrade?)
2020-09-01T13:25:27Z|00009|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2020-09-01T13:25:27Z|00010|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2020-09-01T13:25:27Z|00011|ovsdb_idl|WARN|Logical_Router_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
2020-09-01T13:25:27Z|00012|ovsdb_idl|WARN|Logical_Router_Port table in OVN_Northbound database lacks ipv6_prefix column (database needs upgrade?)
2020-09-01T13:25:27Z|00013|ovsdb_idl|WARN|Logical_Switch table in OVN_Northbound database lacks forwarding_groups column (database needs upgrade?)
2020-09-01T13:25:27Z|00014|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
2020-09-01T13:25:27Z|00015|ovsdb_idl|WARN|NAT table in OVN_Northbound database lacks external_port_range column (database needs upgrade?)
2020-09-01T13:25:27Z|00016|ovsdb_idl|WARN|NAT table in OVN_Northbound database lacks options column (database needs upgrade?)
2020-09-01T13:25:27Z|00017|ovsdb_idl|WARN|NB_Global table in OVN_Northbound database lacks ipsec column (database needs upgrade?)
2020-09-01T13:25:27Z|00018|ovsdb_idl|WARN|NB_Global table in OVN_Northbound database lacks name column (database needs upgrade?)
2020-09-01T13:25:27Z|00019|ovsdb_idl|WARN|NB_Global table in OVN_Northbound database lacks options column (database needs upgrade?)
2020-09-01T13:25:27Z|00020|ovsdb_idl|WARN|OVN_Northbound database lacks Forwarding_Group table (database needs upgrade?)
2020-09-01T13:25:27Z|00021|ovsdb_idl|WARN|OVN_Northbound database lacks HA_Chassis table (database needs upgrade?)
2020-09-01T13:25:27Z|00022|ovsdb_idl|WARN|OVN_Northbound database lacks HA_Chassis_Group table (database needs upgrade?)
2020-09-01T13:25:27Z|00023|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks health_check column (database needs upgrade?)
2020-09-01T13:25:27Z|00024|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks ip_port_mappings column (database needs upgrade?)
2020-09-01T13:25:27Z|00025|ovsdb_idl|WARN|Load_Balancer table in OVN_Northbound database lacks selection_fields column (database needs upgrade?)
2020-09-01T13:25:27Z|00026|ovsdb_idl|WARN|OVN_Northbound database lacks Load_Balancer_Health_Check table (database needs upgrade?)
2020-09-01T13:25:27Z|00027|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2020-09-01T13:25:27Z|00028|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2020-09-01T13:25:27Z|00029|ovsdb_idl|WARN|Logical_Router_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
2020-09-01T13:25:27Z|00030|ovsdb_idl|WARN|Logical_Router_Port table in OVN_Northbound database lacks ipv6_prefix column (database needs upgrade?)
2020-09-01T13:25:27Z|00031|ovsdb_idl|WARN|Logical_Switch table in OVN_Northbound database lacks forwarding_groups column (database needs upgrade?)
2020-09-01T13:25:27Z|00032|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
2020-09-01T13:25:27Z|00033|ovsdb_idl|WARN|NAT table in OVN_Northbound database lacks external_port_range column (database needs upgrade?)
2020-09-01T13:25:27Z|00034|ovsdb_idl|WARN|NAT table in OVN_Northbound database lacks options column (database needs upgrade?)
2020-09-01T13:25:27Z|00035|ovsdb_idl|WARN|NB_Global table in OVN_Northbound database lacks ipsec column (database needs upgrade?)
2020-09-01T13:25:27Z|00036|ovsdb_idl|WARN|NB_Global table in OVN_Northbound database lacks name column (database needs upgrade?)
2020-09-01T13:25:27Z|00037|ovsdb_idl|WARN|NB_Global table in OVN_Northbound database lacks options column (database needs upgrade?)
2020-09-01T13:25:27Z|00038|ovsdb_idl|WARN|transaction error: {"details":"Parsing ovsdb operation 1 of 2 failed: No table named HA_Chassis_Group.","error":"syntax error","syntax":"{\"op\":\"insert\",\"row\":{\"name\":\"lxd-net4\"},\"table\":\"HA_Chassis_Group\",\"uuid-name\":\"row26e6c85e_d968_4c6b_8418_aac9e9c6cae7\"}"}
ovn-nbctl: transaction error: {"details":"Parsing ovsdb operation 1 of 2 failed: No table named HA_Chassis_Group.","error":"syntax error","syntax":"{\"op\":\"insert\",\"row\":{\"name\":\"lxd-net4\"},\"table\":\"HA_Chassis_Group\",\"uuid-name\":\"row26e6c85e_d968_4c6b_8418_aac9e9c6cae7\"}"}

It looks like the schemas are too old:

Debian Buster:

ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.10.1
DB Schema 5.13.0

ovn-sbctl -ovn-sbctl --version
ovn-sbctl (Open vSwitch) 2.10.1
DB Schema 1.16.0

Compare to Ubuntu Focal:

ovn-nbctl --version
ovn-nbctl 20.03.0
Open vSwitch Library 2.13.0
DB Schema 5.20.0

ovn-sbctl --version
ovn-sbctl 20.03.0
Open vSwitch Library 2.13.0
DB Schema 2.7.0

It looks like the Debian versions are before OVN split out of the OVS package and became its own code base too.

bodleytunes · September 1, 2020, 1:54pm

Would it be worth me trying to compile and install later packages to get it to work?