Using host gateway for DHCP for LXC containers

LXD
1

Hello,

I’m a FreeBSD user playing around with LXD and LXC as an alternative to jails. They seem to be very similar but more flexible.

One thing I’m used to with jails is that upon creating one you can enable vnet, a virtual network stack for each jail. By default they are exposed to the LAN like any other machine, I can assign them an IP address and it gets leased by the host’s gateway. For example: If my FreeBSD host 10.110.110.5 is connected to my router 10.110.110.1 I can create a jail with vnet and assigned IP of for example 10.110.110.6 and I will see it in my LAN. If I host a web server in it I can access it from any machine connected to the same network. Very simple.

I’m having problems doing the same thing with LXC and I’m not sure what I’m doing wrong. I’ve been looking at the Linux Containers documentation but I’m not that good at network things. I saw one interesting bit in the Arch Wiki that says:

There are several main setups to consider:

1) A host bridge
2) A NAT bridge

The host bridge requires the host’s network manager to manage a shared bridge interface. The host and any lxc will be assigned an IP address in the same network (for example 192.168.1.x). This might be more simplistic in cases where the goal is to containerize some network-exposed service like a webserver, or VPN server. The user can think of the lxc as just another PC on the physical LAN, and forward the needed ports in the router accordingly. The added simplicity can also be thought of as an added threat vector, again, if WAN traffic is being forwarded to the lxc, having it running on a separate range presents a smaller threat surface.

I created a bridge interface when I initalized LXD:

Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]: no
Would you like to connect to a MAAS server? (yes/no) [default=no]:
[ins]Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:[/ins]
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like the LXD server to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

When I list them I have a bridge interface as well as my host’s ethernet and WLAN interfaces.

[parski@mango ~]$ lxc network list
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
|  NAME  |   TYPE   | MANAGED |     IPV4      |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
| enp3s0 | physical | NO      |               |                           |             | 0       |         |
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
| lxdbr0 | bridge   | YES     | 10.52.83.1/24 | fd42:806b:fbe4:ccd7::1/64 |             | 2       | CREATED |
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+
| wlan0  | physical | NO      |               |                           |             | 0       |         |
+--------+----------+---------+---------------+---------------------------+-------------+---------+---------+

If I launch a container it get’s an IP address in the same subnet as the bridge which is different from my LAN that the host is connected to. My host can access the containers via their IP addresses but another machine on my LAN can’t because my router can’t route to the subnet used by LXD. Does that make sense?

This is different from what I’m reading in the Arch Wiki, quoted above, in particular the part “lxc as just another PC on the physical LAN”. How can I let my router assign IP addresses to the LXC containers so that they are just like another PC on the physical LAN?

If I run a web server in a container I am able to proxy that to a port on my host, like so:

lxc config device add <container-name> <proxy-name> proxy "listen=tcp:<host-ip-address>:<destination-port>" connect=tcp:127.0.0.1:<source-port>

But if I want to run five different web applications on the host in different containers I’d have to 10.110.110.5:1111, 10.110.110.5:2222, 10.110.110.5:3333, and so on. With my jails I can just let them have their own dedicated IP addresses on my network like 10.110.110.6, 10.110.110.7, 10.110.110.8. I much prefer this.

As a final note this is a competely fresh install. No Docker to interfere or anything like that. I installed git, yay, vim and make I think. Then I installed lxd/lxc. When I installed lxd it replaced the iptables package with iptables-nft or something like that. That’s it.

2

Hi,

This is similar to a request the other day, see my response:

1 Like
3

I did not think macvlan worked (I’ve tried it before) since I could not see an IP address when listing the containers:

[parski@mango documentation]$ lxc list
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+
|     NAME      |  STATE  |        IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| capital-hare  | RUNNING |                     |                                               | CONTAINER | 0         |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| popular-molly | RUNNING | 10.52.83.197 (eth0) | fd42:806b:fbe4:ccd7:216:3eff:fe11:38e1 (eth0) | CONTAINER | 0         |
+---------------+---------+---------------------+-----------------------------------------------+-----------+-----------+

However, I noticed your note in the thread you linked just now:

Note: With macvlan NICs the LXD host and the instance will not be able to communicate with each other.

When I try to connect to the container from the machine I’m using (not the host) it works perfectly.

I don’t really understand why the host cannot communicate with the guest but it’s not necessary for my use case anyway so I’m okay with this.

Thank you!

4

Macvlan does work, in that it will use the external network’s DHCP server, and should get an IP address. If it doesn’t you’ve probably got a firewall on the host thats blocking outbound requests or an external network that is restricting what MAC addresses the external port can use.

Please show ip a and ip r on the host and lxc config show <instance> --expanded?

As for why macvlan devices cannot communicate with the host, this is a limitation/design feature of the macvlan interface type in the kernel and not something instigated by LXD.

5

I was unclear. Macvlan works great! I was just previously fooled by the output I cited above when I’d tried it before but your note made me try to access the container from another machine on the LAN and that worked fine.

Again, thanks!

1 Like