Connect LXD instances across AWS EC2 Instances

LXD
1

Two AWS EC2 VM instances, both VM instances in same VPC and both VM instances on same subnet.

Both EC2 VM instances have LXD installed and both EC2 VM instances have a single LXD instance running.

What is the way to get the LXD instances talking to each other, to the VM’s, and to other VM’s in the same VPC and subnet?

Thanks

2

I can’t remember what kind of weird limits VPCs have these days, but in theory, good old routing should be fine.

For example, use 10.10.1.0/24 as the container subnet on VM1 and 10.10.2.0/24 as the container subnet on VM2. You can then add routes to both VMs to reach the other.

Some like ip -4 route add 10.10.2.0/24 via 172.x.x.x on VM1 and ip -4 route add 10.10.1.0/24 via 172.x.x.x on VM2 where the “via” refers to the VPC address of the other VM.

This obviously gets more and more annoying as you add more and more VMs.
You could use one of the dynamic routing protocols to do that for you if you grow much bigger, or you could switch to using some kind of SDN that’d provide you with a bridge usable by LXD on every host.

3

Thanks Stéphane!

I have an AWS EC2 Windows 2012 R2 instance that is in the same VPC as an Ubuntu 16.04 EC2 LXC host on which there are LXC containers running on OpenvSwitch network 10.209.53.0/24.

I can ping the Windows EC2 adapters from the LXC container OK but cannot ping the LXC containers from the Windows EC2 instance.

4

In case this helps anyone, here is how this is solved. Sincere thanks to Sai at AWS and the team at AWS who worked on this ticket over several day period.

Caveat this has been tested only when all EC2 instances are in the SAME VPC.

The setup is LXC but it should work for LXD as well.

The goal here is to make the LXC containers on the Ubuntu 16.04 EC2 host generally available on any selected set of ports from the Windows 2012 R2 EC2 instance. It’s a generic solution so it should work with any EC2 instance calling to an EC2 LXC host.

The reference configuration is:

(1)

Ubuntu 16.04 EC2 instance on which LXC containers that have been deployed by Orabuntu-LXC are running. This means the LXC containers are running on an OpenvSwitch network. However this solution should be general for any network solution, not just OpenvSwitch. The containers are on the 10.209.53.0/24 network.

(2)

Windows 2012 R2 EC2 instance.

The LXC containers running on the Ubuntu 16.04 EC2 instance can ping the Windows 2012 R2 EC2 instance, but NOT vice versa.

The problem therefore is to establish communication from the Windows 2012 R2 EC2 instance to the LXC containers on the Ubuntu 16.04 EC2 host on any arbitrary ports.

The solution is as follows:

  1. On the AWS VPC that the EC2 instances are in, go to VPC then to ROUTE TABLES.
  2. Choose the VPC which is explicitly associated with the subnets to which the EC2 instances belong. Be sure to choose the VPC that is actually associated with the subnets.
  3. Go to Routes tab and click on big blue “Edit” button to edit routes
  4. Add a route as follows:
  • In the “Destination” field put the CIDR format of your LXC network, e.g. 10.209.53.0/24
  • In the “Target” field put the ENI of the adapter of the LXC EC2 host (the Ubuntu16.04 EC2 host)
    You can open another window to your EC2 hosts to get this.
    Just be sure it’s the ENI for the LXC EC2 host.
    The ENI’s have names like this: eni-4da3dbca

routes
routes.png1198×731 73.1 KB

When you put the ENI in the “Target” field, it will also grab the instance id and save both into the field, and you will see both of those in the “Target” field. You can double-check that the instance id that it saved is the one that is the LXC host.

Now you have to do one more setting.

  1. Go to the EC2 console
  2. Go to the “Description” tab of the Ubuntu 16.04 EC2 LXC host
  3. Click on the “eth0” link
  4. Click on the actual ENI link in the black pop-up dialog (takes you to interfaces page)
  5. Select the ENI (it should already be selected)
  6. Go to “Actions”
  7. Select “Change Source/Dest. Check” from the drop-down menu
  8. DISABLE source/dest checking.

disabled

Now your calling instance, in this case the Windows 2012 R2 EC2 instance, will be able to ping and ssh and pass traffic on any other ports you choose to allow to the LXC containers on the destination EC2 host instance (in this case Ubuntu 16.04 EC2 instance) and vice versa.

Again, sincere thanks to Sai at AWS and the team at AWS who worked on this ticket over several day period.

2 Likes
5

AFAIK this same approach cannot be done with IPv6 subnets in AWS as they do not allow more specific prefixes from the /56 allocated to the VPC to be routed inside the VPC (i.e to an ENI).

I was at reInvent this week and spoke to one of the VPC engineers about this, they said they couldnt remember why more specific prefix routing was blocked, but that it has come up a few times as an issue.

Hopefully this will be something that is fixed, as otherwise getting fully routed IPv6 addresses into containers on EC2 is tricky without having to resort to managing IP aliases on EC2 instances.

2 Likes