Help for LXD cluster network


(Park Kyung Won) #1

I want to create LXD vxlan network over multiple hosts

and attach that network to certain profile, so I can launch container

with profile on any hosts, connected to vxlan network.

How can I achieve this? It seems like creating network on cluster requires target node for option.

If I create network with same name on all cluster machines, would LXD cluster recognize it as one?


(Stéphane Graber) #2

When manually creating a network on a LXD cluster, you indeed need to use --target NAME for the create operation, targeting each of the clustern nodes one by one, passing the key=value arguments that are node specific.

Once the node-specific stuff is done, you can run a non-targeted create which will then ensure that all nodes have their node-specific config and will make that network be usable in the cluster.


(Zack Siri) #3

Will cross host networking work if i use lxd init to join an existing cluster? Or is there something extra I have to do?

I created a cluster and tried creating containers and pinging them doesn’t seem to be working. I can ping containers on the same machine as expected.


(Park Kyung Won) #4

Seems like u r trying to achieve same thing with me

No just clustering LXD engines won’t make default network overlay one

You need to use macvlan + switch you have and setup VLAN yourself

or openvswitch would be needed


(Free Ekanayaka) #5

I think there are some changes coming in 3.1.0 that will allow you to have cross-node networking work out of the box, using a network fan. @stgraber will be able to tell more exactly.


(Park Kyung Won) #6

Is “fan” conceptual thing like VxLAN

of actual name of package to be used like “openvswitch” ?


(Free Ekanayaka) #7

It’s an overlay network which basically works out of the box:

https://wiki.ubuntu.com/FanNetworking


(Zack Siri) #8

Having a Fan networking out of the box would be awesome, I just broke my lxd config trying to configure the Fan networking myself


(Stéphane Graber) #9

Right, lxd init in LXD 3.1 (and in fact in the current snap) will setup a Fan overlay bridge for you if you don’t pick a physical bridge or NIC.

The Fan can either use ipip or vxlan for tunneling, LXD defaults to using vxlan.
The difference between it and regular vxlan is that it’s setup automatically for you. Each host gets its own /24 subnet with LXD doing normal DHCP/DNS on it and the host acting as the gateway. The Fan then effectively sets up route to all the Fan subnets of your other hosts.

That way, there’s no need for some kind of HA gateway that does DNS/DHCP/gateway for all your hosts, yet all containers can still contact each other.


(Zack Siri) #10

Wow that would be awesome, I’m using lxd from Ubuntu 18.04, any chance I can get that without snap?


(Stéphane Graber) #11

Yeah, that’ll make it to Ubuntu 18.04 when we release LXD 3.0.1, hopefully in the next week or so (+1 week delay for it actually making it to 18.04 due to Ubuntu update gating).


(Zack Siri) #13

OK I just tried it out this is the error I’m getting

Error: Failed to create network 'lxdfan0': Only /16 or /24 underlays are supported at this time

I’m guessing because my node doesn’t have private networking that’s why it’s failing?

My node is a VM with only public IP which obviously has /32 ‘underlay’

edit 2: I tried the samething on a digitalocean node (has private networking) I’m getting

Error: Failed to create network 'lxdfan0': More than one IPv4 subnet on default interface

(Stéphane Graber) #14

Yeah, so LXD makes a good guess as to what interface and subnet to use, but that doesn’t always work.

If it doesn’t, just answer “no” to the question during init and configure it by hand afterwards, this will let you override the underlay network which should fix the problem.

lxc network create fanbr0 bridge.mode=fan fan.underlay_subnet=a.b.c.d/24
lxc profile device add default eth0 nic nictype=bridged parent=fanbr0 name=eth0

More information about the various config keys available can be found in doc/networks.md or at https://lxd.readthedocs.org


(Zack Siri) #15

Thx I’ll give this a whirl. btw snapcraft just went down :grinning:


(Zack Siri) #16

So I just managed to get zerotier to work as a fan underlay for lxd. Works beautifully. I think I learned a bunch about networking / linux / containers. It’s all much clearer in my head.

I have one question thought when I use the hostname to ping another container on the same machine they can find each other, but when they’re remote they can’t seem to find each other via hostname i have to use the IP

Thank You


(Park Kyung Won) #17

This is exactly what I’m trying to do but

for now I am trying with macvlan + switch + VLAN setup

Ping to containers with hostname part, I am trying to setup

isc-dhcp with bind9 and DDNS update. Would it be possible?


(Brian Mullan) #19

If you have anything written up about your install of ZeroTier on multiple LXD hosts could you post it so others benefit from your lessons learned?


(Zack Siri) #20

Indeed! I will make a video showing exactly how to do it. I have a website that teaches people about devops and building web apps https://www.codemy.net in fact I’m hosting the new version on this lxd cluster I’m setting up.

I’ve also figured out how to route request based on domain name to the various lxd containers.


(ROKA Admin) #21

I tried using --target it gives different error every time.

lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv6.nat'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.nat'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.address'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.address'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.address'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv6.address'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.nat'

(Stéphane Graber) #22

@freeekanayaka this looks like a bug for you :slight_smile: