Containers on EC2

LXD
1

Hi,

I’m starting to get a bit lost here, and AWS support has been unable to answer my questions.

I’m trying to run several EC2 instances with LXC/LXD. Each instance is hosted in a private subnet, with the primary network adapter (ens5). In the VPC, I also create additional private subnets, which would contain the containers running on those instances. There’s also a (correctly) configured NAT gateway (I know it’s working because I can directly connect from the instance running on the private subnet).

What works:

  • The EC2 instance itself can ping the containers on that instance
  • It can ping the EC2 instance

What doesn’t work:

  • Can’t ping the containers from other EC2 instances
  • Outbound Internet is not working using the NAT gateway

This is all with a default managed lxdbr0 which is set to the ip range of one of the private subnets. All of this works fine (with IP alias ranges) on Google Cloud Platform for example, but I am completely unable to replicate this on AWS.

Anyone have any experience with such a setup and might have some tips? End goal is: private EC2 instances, each running a private subnet which can contain containers, each container should be able to reach everything else and the internet.

2

Hi
You need to enable ip routing in the ec2 LXD host.
You need a route in the VPC or subnet “main” routing table to point to the LXD network to go via the ec2 lxd host.
You probably also need to disable “reverse path” verification check in AWS which will drop traffic from a subnet it doesn’t know about.
Those are usually the main gotchas that spring to mind.
Re: nat gateway that could be an AWS general thing, do you have an internet gateway also setup? from recollection the nat gateway needs to point to an internet gateway.
Cheers,
Jon.

3

I’ll check those things out, thanks!

Wrt the NAT Gateway, that’s working just fine directly from hosts on the private subnet, so it’s basically Private Subnet has a default gw pointing to the NAT gateway, which is hosted inside a public subnet with an Internet gateway.

1 Like