AWS: best way to give container its own IP from attached ENI/NIC

LXD
1

I have a 3 node LXD cluster, and I’d like to have ENIs with public, stable Elastic IP addresses (EIPs) plugged into each node.

The use case is one additional virtual NIC per node.

I’d like to dedicate the NIC entirely to a container running on each node. Inside the containers I’ll run L7 proxies.

Here is the output of ip addr on one of the nodes. The extra NIC (the AWS ENI) is ens6.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:3c:1a:a2:91:fb brd ff:ff:ff:ff:ff:ff
    inet 10.111.10.194/24 brd 10.111.10.255 scope global dynamic ens5
       valid_lft 2089sec preferred_lft 2089sec
    inet6 2600:1f18:474:9d01:e7a9:bb22:8d85:a979/128 scope global dynamic noprefixroute 
       valid_lft 407sec preferred_lft 97sec
    inet6 fe80::83c:1aff:fea2:91fb/64 scope link 
       valid_lft forever preferred_lft forever
3: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 0a:66:3d:5d:6a:eb brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    inet 10.111.10.142/24 brd 10.111.10.255 scope global ens6
       valid_lft forever preferred_lft forever
    inet6 2600:1f18:474:9d01:e9b3:6059:dd57:663c/128 scope global dynamic noprefixroute 
       valid_lft 447sec preferred_lft 137sec
    inet6 fe80::866:3dff:fe5d:6aeb/64 scope link 
       valid_lft forever preferred_lft forever
4: lxdfan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:cc:fb:92 brd ff:ff:ff:ff:ff:ff
    inet 240.194.0.1/8 scope global lxdfan0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fecc:fb92/64 scope link 
       valid_lft forever preferred_lft forever
5: lxdfan0-mtu: <BROADCAST,NOARP,UP,LOWER_UP> mtu 8951 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether 42:80:fa:66:95:25 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::4080:faff:fe66:9525/64 scope link 
       valid_lft forever preferred_lft forever
6: lxdfan0-fan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue master lxdfan0 state UNKNOWN group default qlen 1000
    link/ether ba:5e:6e:ca:84:45 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b85e:6eff:feca:8445/64 scope link 
       valid_lft forever preferred_lft forever
9: veth9c5789ba@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue master lxdfan0 state UP group default qlen 1000
    link/ether 26:08:6c:56:87:2c brd ff:ff:ff:ff:ff:ff link-netnsid 0

As you can see, the cluster is set up with a managed bridge and Fan Networking.

The one container running is veth9c5789ba@if8.

On the same node, this is lxc network list

+---------+----------+---------+-------------+---------+---------+
|  NAME   |   TYPE   | MANAGED | DESCRIPTION | USED BY |  STATE  |
+---------+----------+---------+-------------+---------+---------+
| ens5    | physical | NO      |             | 0       |         |
+---------+----------+---------+-------------+---------+---------+
| ens6    | physical | NO      |             | 1       |         |
+---------+----------+---------+-------------+---------+---------+
| lxdfan0 | bridge   | YES     |             | 2       | CREATED |
+---------+----------+---------+-------------+---------+---------+

I have attached the ens6 NIC to the container.

What is the best way to dedicate the entire NIC to forward everything to this container?

  • Should I set up nftables rules to forward all traffic (v6 and v4) to the veth interface?
  • If I’m going to be setting up nftable rules at all, what’s the point of adding the NIC to the container? Can I just rewrite the IP and forward packets to the LXD bridge?

Does anyone have any advice to accomplish my goal? To summarize, my goal is dedicate an IP - attached to an ENI/NIC - to each node, forwarded to one container on each node.

2

Hi Coleman - I’m interested in the same solution. Were you able to get it setup properly?

3

Unfortunately, no. It was more expedient for me to rely on port-forwarding from the container to the main host’s IP.

I am really interested in a solution here, though. Being able to move the ENI/NIC from node to node would be very convenient. I just don’t know how to do the configuration in LXD or nftables yet.

4

I agree. I’m looking for the exact same solution but haven’t been able to get it to work yet. It seems a reasonable use case so I’m not sure why it’s proving so difficult to crack!