LXD demo on AWS with Ubuntu fan networking

LXD
1

Hello,

I have been experimenting with LXD clustering on AWS. The idea is to use AWS as a simple way to spin up demos and proof of concept environments.

A few days I ago I spun up a cluster like in the YouTube video about the Ubuntu fan.

https://youtu.be/5cwd0vZJ5bw

I was not able to get the containers to see each other.

So, I am thinking I will write some Terraform to create the VPC, subnets, security groups, instances, etc. If I can get the Ubuntu fan to work, I will write a tutorial about it and post here with the howto tag.

Before I start working on this, are there any details about running an LXD cluster on AWS I should know about? Has anyone seen a tutorial like this anywhere else?

Thanks for the help.

2

Hi,

Have you checked that the intra-EC2 network is allowing VXLAN?

Saw a similar issue on Oracle cloud the other day, see:

3

Oh, cool. I bet that is it.

I will rebuild the cluster and check. If I remember correctly, I allowed all TCP traffic between nodes within the VPC. That sounds like the problem. I should have also allowed UDP.

When I get it rebuilt, I will let you know.

1 Like
4

As it turns out, I was allowing all traffic.

This might be the issue.

Would you like to create a new Fan overlay network? (yes/no) [default=yes]: 
What subnet should be used as the Fan underlay? [default=auto]: 
Invalid input: The auto-detected underlay (172.31.0.0/20) isn't a /16 or /24, please specify manually

What subnet should be used as the Fan underlay? [default=auto]: 172.31.0.0/16

I just guessed when I put in 172.31.0.0/16 and then tested to see if it worked.

Now I have three containers on three different nodes.

u1 and u2 can communicate but u3 can’t communicate with u1 or with u2.

I bet it has to do with the CIDR ranges. I will try again later in the week. I am going to tear the cluster down again.

Any idea why I could not use 173.21.0.0/20? That is the CIDR for the subnet that all of the instances are in.

5

The fan network provides an overlay subnet, and it requires a /16 or /24 underlay to operate.

See FanNetworking - Ubuntu Wiki

6

This might help too.

Screenshot from 2022-07-18 18-10-28
Screenshot from 2022-07-18 18-10-282061×1831 497 KB

Thanks for your help so far.

7

Quick update.

I created a new VPC with a CIDR of 10.0.0.0/16 and single public subnet with the same CIDR. I then created a three node cluster with a fan network.

This time everything worked. For demos, this could work. It is not likely to work for a production setup since only one availability zone is being used and there are no private subnets.

I will try again some time soon with a more realistic VPC.

1 Like