I want to create LXD vxlan network over multiple hosts
and attach that network to certain profile, so I can launch container
with profile on any hosts, connected to vxlan network.
How can I achieve this? It seems like creating network on cluster requires target node for option.
If I create network with same name on all cluster machines, would LXD cluster recognize it as one?
When manually creating a network on a LXD cluster, you indeed need to use --target NAME for the create operation, targeting each of the clustern nodes one by one, passing the key=value arguments that are node specific.
Once the node-specific stuff is done, you can run a non-targeted create which will then ensure that all nodes have their node-specific config and will make that network be usable in the cluster.
Will cross host networking work if i use lxd init to join an existing cluster? Or is there something extra I have to do?
I created a cluster and tried creating containers and pinging them doesnât seem to be working. I can ping containers on the same machine as expected.
Seems like u r trying to achieve same thing with me
No just clustering LXD engines wonât make default network overlay one
You need to use macvlan + switch you have and setup VLAN yourself
or openvswitch would be needed
I think there are some changes coming in 3.1.0 that will allow you to have cross-node networking work out of the box, using a network fan. @stgraber will be able to tell more exactly.
Is âfanâ conceptual thing like VxLAN
of actual name of package to be used like âopenvswitchâ ?
Itâs an overlay network which basically works out of the box:
Having a Fan networking out of the box would be awesome, I just broke my lxd config trying to configure the Fan networking myself
Right, lxd init in LXD 3.1 (and in fact in the current snap) will setup a Fan overlay bridge for you if you donât pick a physical bridge or NIC.
The Fan can either use ipip or vxlan for tunneling, LXD defaults to using vxlan.
The difference between it and regular vxlan is that itâs setup automatically for you. Each host gets its own /24 subnet with LXD doing normal DHCP/DNS on it and the host acting as the gateway. The Fan then effectively sets up route to all the Fan subnets of your other hosts.
That way, thereâs no need for some kind of HA gateway that does DNS/DHCP/gateway for all your hosts, yet all containers can still contact each other.
Wow that would be awesome, Iâm using lxd from Ubuntu 18.04, any chance I can get that without snap?
Yeah, thatâll make it to Ubuntu 18.04 when we release LXD 3.0.1, hopefully in the next week or so (+1 week delay for it actually making it to 18.04 due to Ubuntu update gating).
OK I just tried it out this is the error Iâm getting
Error: Failed to create network 'lxdfan0': Only /16 or /24 underlays are supported at this time
Iâm guessing because my node doesnât have private networking thatâs why itâs failing?
My node is a VM with only public IP which obviously has /32 âunderlayâ
edit 2: I tried the samething on a digitalocean node (has private networking) Iâm getting
Error: Failed to create network 'lxdfan0': More than one IPv4 subnet on default interfaceYeah, so LXD makes a good guess as to what interface and subnet to use, but that doesnât always work.
If it doesnât, just answer ânoâ to the question during init and configure it by hand afterwards, this will let you override the underlay network which should fix the problem.
lxc network create fanbr0 bridge.mode=fan fan.underlay_subnet=a.b.c.d/24
lxc profile device add default eth0 nic nictype=bridged parent=fanbr0 name=eth0
More information about the various config keys available can be found in doc/networks.md or at https://lxd.readthedocs.org
Thx Iâll give this a whirl. btw snapcraft just went down 
So I just managed to get zerotier to work as a fan underlay for lxd. Works beautifully. I think I learned a bunch about networking / linux / containers. Itâs all much clearer in my head.
I have one question thought when I use the hostname to ping another container on the same machine they can find each other, but when theyâre remote they canât seem to find each other via hostname i have to use the IP
Thank You
This is exactly what Iâm trying to do but
for now I am trying with macvlan + switch + VLAN setup
Ping to containers with hostname part, I am trying to setup
isc-dhcp with bind9 and DDNS update. Would it be possible?
If you have anything written up about your install of ZeroTier on multiple LXD hosts could you post it so others benefit from your lessons learned?
Indeed! I will make a video showing exactly how to do it. I have a website that teaches people about devops and building web apps https://www.codemy.net in fact Iâm hosting the new version on this lxd cluster Iâm setting up.
Iâve also figured out how to route request based on domain name to the various lxd containers.
I tried using --target it gives different error every time.
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv6.nat'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.nat'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.address'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.address'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.address'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv6.address'
lxdtest@lxdtest8:~$ sudo lxc network create lxdbr0 ipv4.address=10.8.8.1/24 ipv4.nat=true ipv6.address=fd68:3604:ad86:9::8/64 ipv6.nat=true --target lxdtest8
Error: Invalid config key 'ipv4.nat'