Serving DNS over OVN networks and accessing the instances from the hosts

LXD
, ,
21

Part of the confusion might come from here too, since usually NAT’ing (routers) can reach the nodes behind the NAT. I’m just saying that probably my networking background here might have derailed me from fully understanding all this.

22

The LXD host is considered external to the virtual OVN router that provides the NAT to the virtual OVN network. Which is why it provides the uplink network to the OVN router.

HTTP requests go over TCP/IP, so for this scenario they are both the same.

There are various ways round it:

  1. Don’t use NAT on the OVN networks. This will require you to manage subnet allocations so that you don’t end up with overlapping subnets for your OVN networks. It may also require you to perform SNAT on the LXD hosts, so they can still access the external internet (I’m presuming you still don’t want external incoming connections to the OVN networks). This would still only work on one of the LXD hosts at a time, because OVN only uses one LXD host at a time for the active chassis connected to the uplink network.
  2. Create instance(s) with a NIC inside each OVN network to provide local ansible services.
  3. Use a (temporary?) proxy device attached to each instance to allow ansible to connect to the instance’s SSH port from the local LXD host.
  4. Use lxc exec with ansible that doesn’t use networking.
23

On the DNS point, @stgraber do we plan on adding custom internal DNS record support to OVN networks?

24

I definitely need a graphic explaining all this, it’s going over my head, and reading OVS/OVN’s docs is surprisingly harder than I expected (in part, again, for the lack of graphics explaining the high level view. I’m getting old).

25

Can I connect the host to the OVN router?

26

I think the fundamental issue here is that you are looking for private throw away distributed networks that are also not private and are reachable from the outside :slight_smile:

If you didn’t need them distributed you could use throw away bridge networks.
If you didn’t need them to be throw away you could use bridge on vlans on the physical network to get it distributed.

27

I can live with a SPOF node (the host running Ansible + Terraform is such a thing). And I was already using throwaway bridges:

28

In this case, I wasn’t referring to highly available, but distributed over multiple LXD servers in a cluster, which I seem to recall you needed also.

29

They are private, what do you mean? I just didn’t know the host would be outside.

I fact, I need most networks that way. I still need a single distributed network that it’s not private for the host; the one I use for general inet and ssh/ansible access. I need it distributed (across all hosts) so I can reach all the instances from a single (static) host for that.

30

Aren’t NAT and SNAT usually synonyms?

Let’s put some numbers here:

  • host: lan0 is a bridge with the 10.130.40.81/22 IP. It also has the host’s single NIC enp2s0.
  • instance: has 4 NICS: eth0, eth1, provision and ipmi. provision and ipmi are connected to lxd-provision network. eth0 and eth1 are connected to a private_to_cluster network.
    • private_to_cluster used to be a bridge network on a single host, now it’s an OVN network.
      • no routing is expected on this.
      • there used to be a dnsmasq providing private DNS.
        • You tell me I can still create the brigde network with dnsmasq and use it as network for the OVN network.
    • lxd-provision used to also be a bridge network. I switched to a OVN network in the hope I could still use it to reach the instances from a single host via TCP/IP.
      • lxd magically provided a dnsmasq for upstream DNS
      • lxd magically provided (S)NAT/masquerading and forwarding through enp2s0, now would be lan0.

The ovn-overlay OVN network seems to be comprised of:

  • something on each host connecting to enp2s0 by ‘guessing it’ from the lan0 bridge. lxc network create ovn-overlay --type=physical --target $host parent=lan0. Can you explain what’s behind here?
  • a cluster wide defined someotherthing lxc network create ovn-overlay --type=physical. Can you also explain what’s going on here?

And then any LXD-type-OVN network on top of that is… what? An OVN/OVS router + switch, it seems:

switch bed7c1c4-8bbe-471f-a7f6-346080351020 (lxd-net10-ls-int)
    port lxd-net10-instance-63cd677d-1c53-48e8-87cf-85864d462509-ipmi
        addresses: ["00:16:3e:19:1b:ff dynamic"]
    port lxd-net10-instance-9bc6da20-62f8-490d-badd-ae4c6b690dde-ipmi
        addresses: ["00:16:3e:e0:6e:ec dynamic"]
    port lxd-net10-ls-int-lsp-router
        type: router
        router-port: lxd-net10-lr-lrp-int
    port lxd-net10-instance-63cd677d-1c53-48e8-87cf-85864d462509-provision
        addresses: ["00:16:3e:62:fb:24 dynamic"]
    port lxd-net10-instance-9bc6da20-62f8-490d-badd-ae4c6b690dde-provision
        addresses: ["00:16:3e:6a:83:42 dynamic"]

router 9418cef1-10fc-4705-82a1-450a98e33a7f (lxd-net10-lr)
    port lxd-net10-lr-lrp-int
        mac: "00:16:3e:66:43:76"
        networks: ["10.127.117.1/24", "fd42:2ff9:9655:63c9::1/64"]

Can’t I just add my host to that switch?

31

Or how can I build an L3? gateway + NAT to it from the host?

32

Like this?

lxc network create lxd-provision --type=ovn ipv4.dhcp=true ipv4.nat=false network=ovn-overlay
Error: Uplink network doesn't contain "10.31.98.0/24" in its routes

Not sure what should I do there…

33

From OVN and DHCP: A minimal example · The Odd Bit and thanks to numans in #openvshitch:

# create provision network
lxc network create ovn-overlay --type=ovn ipv4.dhcp=true ipv4.nat=true network=ovn-overlay

# create a port in the switch created by that, check the output of ovn-nbctl show
sudo ovn-nbctl lsp-add bed7c1c4-8bbe-471f-a7f6-346080351020 ovn-access

# get MAC addr and IP dynamically, 
sudo ovn-nbctl lsp-set-addresses ovn-access dynamic

# get actual addresses from sudo ovn-nbctl list logical_switch_port
# use the same MAC addr here
# TODO: why OVS?
sudo ovs-vsctl add-port br-int ovn-access -- set interface ovn-access type=internal mac='["46:1e:43:7f:75:07"]' external_ids:iface-id=ovn-access

# this creates an interface ovn-access on the host
# set the SAME ip as above
sudo ip address add 10.201.159.2/24 dev ovn-access

# TODO: why same IP?
# TODO: setup NAT + forwarding + dnsmasq, etc
34

In this case you need to indicate to OVN that the uplink network being used is responsible for those routes by setting ipv4.routes on the uplink network. You will also need to add a static route to the active LXD host forwarding those routes to the OVN network’s volatile.network.ipv4.address setting.

35

You should not use the ovn/ovs tools directly with LXD as it will not know what ports/state is configured and will likely overwrite it.

36

I should not, but for the moment I have no choice as this is the only way I have found to do it, even when I have to do many things by hand.

OTOH I’m reviewing our conversations on IRC to see if I can finally understand everything you told me; now I know the tools a little bit more and at the time most of what you said didn’t compute for me :slight_smile:

37

Disabling NAT and then adding a route on your LXD host to the OVN router’s external IP on the uplink would be a supported way. Running sudo ip address add 10.201.159.2/24 dev ovn-access as above will create a static route on your LXD host to 10.201.159.9/24 anyway, so no different in that respect (in that you’ll need to avoid overlapping internal OVN subnets).

38

You keep saying “uplink”. What does it mean to you?

1 Like
39

The network property of the OVN network, the bridge in your case.
This is the terminology we use in the docs also OVN network - LXD documentation

40

Yes, I do get the route you mention, but I also get an interface that allows me to run f.i. a dnsmasq that will serve DNS on the host. AFAIK this is not provided by LXD (yet?).