Containers can't start with unmanaged, standalone bridge

Hello there,
so the other day I started to set up a new homeserver for myself and thought about using LXD/LXC to manage my services (as I already do on my webhost) and to experiment with things without messing up the host installation.

Unfortunately, I am now at a point where i’m honestly thinking about reverting the whole machine back to a clean install as I can’t figure out how to use a manually configured network-bridge with LXD, which should have the same functionality as the one LXD uses and manages, but with dnsmasq running on the host. I need this setup, because I need dnsmasq to also serve my local network (DHCP and local DNS server). Yes, I read that you could run both your own and the lxc-bridge-dnsmasq in parallel, but it was nowhere to be found how much that would complicate things.

What I did was that I removed netplan (following this procedure) and reverted back to /etc/network/interfaces as I wanted to create a bridge without attached hardware-interface, which is afaik not possible with netplan. I tested various things, followed a dozen or so tutorials and guides, but nothing was really able to give me a hint on how to do what I want. I want the LXD-Containers in their own, private network. For this to work, dnsmasq needs seperate interfaces to operate the DHCP server on. So I wanted something simillar to the bridge that LXD creates. In the interfaces file I then created a bridge with bridge_ports none - that’s the “no attached hardware-interface”-part of the story. This works, concerning the hosts network interfaces, and my whole configuration is something like this:

auto lo
iface lo inet loopback

auto enp3s0
iface enp3s0 inet static
  address 192.168.178.103
  netmask 255.255.255.0
  broadcast 192.168.178.255
  gateway 192.168.178.1
  dns-nameservers 192.168.178.1 1.1.1.1

# lxd bridge
auto bridge0
iface bridge0 inet static
  bridge_ports none
  bridge_fd 0
  bridge_maxwait 0
  bridge_stp no
  address 10.254.116.1
  netmask 255.255.255.0
  up iptables -t nat -A POSTROUTING -o enp3s0 -j MASQUERADE

After configuring the interfaces, rebooting and all that, lxc network list shows the bridge:

+---------+----------+---------+-------------+---------+
|  NAME   |   TYPE   | MANAGED | DESCRIPTION | USED BY |
+---------+----------+---------+-------------+---------+
| bridge0 | bridge   | NO      |             | 3       |
+---------+----------+---------+-------------+---------+
| enp3s0  | physical | NO      |             | 0       |
+---------+----------+---------+-------------+---------+

So I added it to the default profile and the containers, thinking that i configure DHCP later (I also need to find out how to make the host-dnsmasq recognize the ipv4.address-part of the container config). But then, desaster. All of my containers with this bridge attached won’t start up anymore. For example, the config for a test-container is as follows:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20200317)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20200317"
  image.type: squashfs
  image.version: "18.04"
  volatile.base_image: 98e43d99d83ef1e4d0b28a31fc98e01dd98a2dbace3870e51c5cb03ce908144b
  volatile.bridge0.hwaddr: 00:16:3e:a8:61:b3
  volatile.bridge0.name: eth0
  volatile.eth0.hwaddr: 00:16:3e:29:7a:97
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: bridge0
    type: nic
ephemeral: false
profiles:
- default
stateful: false
description: ""

before i attached bridge0 on eth0, it was able to start. Now if I try lxc start test it gives me an error, if I open the log i get this:

Name: test
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/04/08 00:23 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc test 20200408162411.148 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.test"
lxc test 20200408162411.148 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1143 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.test"
lxc test 20200408162411.149 ERROR    utils - utils.c:lxc_can_use_pidfd:1834 - Kernel does not support pidfds
lxc test 20200408162411.231 ERROR    network - network.c:lxc_network_move_created_netdev_priv:3129 - File exists - Failed to move network device "veth79e2d470" with ifindex 6 to network namespace 6911
lxc test 20200408162411.231 ERROR    start - start.c:lxc_spawn:1751 - Failed to create the network
lxc test 20200408162411.234 WARN     network - network.c:lxc_delete_network_priv:3213 - Failed to rename interface with index 0 from "eth0" to its initial name "vethfa99e2b8"
lxc test 20200408162411.237 WARN     network - network.c:lxc_delete_network_priv:3213 - Failed to rename interface with index 0 from "eth0" to its initial name "veth79e2d470"
lxc test 20200408162411.237 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:852 - Received container state "ABORTING" instead of "RUNNING"
lxc test 20200408162411.238 ERROR    start - start.c:__lxc_start:1953 - Failed to spawn container "test"
lxc test 20200408162411.238 WARN     start - start.c:lxc_abort:1030 - No such process - Failed to send SIGKILL to 6911
lxc 20200408162411.613 WARN     commands - commands.c:lxc_cmd_rsp_recv:122 - Connection reset by peer - Failed to receive response for command "get_state"

This happens to every container as soon as I attach the bridge. If I detach/remove it, then the container is able to start again.
I searched here and in the git repos for the network-related erros, but I really could not find anything usefull regarding these. I’m also wondering why it’s listing eth0 two times after the failed move. I have a feeling that I did something horribly wrong regarding the bridge, but I have absolutely no clue how to fix this error, get my containers starting again and keep going.

Hi!

You can have a setup where some containers are configured by a LXD-managed network, and others are configured by an independent DHCP/DNS server on the LAN.

The first requirement is that you need this host it connect to the LAN over a Ethernet interface. WiFi does not work due to port security.

The second requirement is that, since you are installing the independent dnsmasq on the host, you need to configure it to only listen/bind on the bridge interface, not any other interfaces.
From the dnsmasq documentation:

-z, --bind-interfaces
On systems which support it, dnsmasq binds the wildcard address, even when it is listening on only some interfaces. It then discards requests that it shouldn’t reply to. This has the advantage of working even when interfaces come and go and change address. This option forces dnsmasq to really bind only the interfaces it is listening on. About the only time when this is useful is when running another nameserver (or another instance of dnsmasq) on the same machine. Setting this option also enables multiple instances of dnsmasq which provide DHCP service to run in the same machine.

You can verify your actual setup when you run the following command. It shows what is listening to port 53 (DNS), and you take notice on the COMMAND and USER columns to identify which services are related. And you need to make sure that your own dnsmasq is not binding/listening to LXD’s managed interfaces.

sudo lsof -ni :53  

If you new host is a desktop computer, you can use NetworkManager (GUI and all) to create the bridge. See, for example, https://blog.simos.info/how-to-make-your-lxd-containers-get-ip-addresses-from-your-lan-using-a-bridge/
Personally, I do not like to make invasive changes to a host like removing netplan unless it is really really required.

Hi simos,
first, thanks for your answer.

I think i missed a few points:
The host is a Ubuntu 18.04 Server installation.

DNS would not be the problem (as you said, port 53 behaves as you describe it). But my real problem is the DHCP server. I don’t want the containers on my LAN. But the managed-bridge dnsmasq defaults to binding to 0.0.0.0:67.
As i want to also use DHCP in my network (and be it only to always assign my other known network devices their reserved IP via MAC, which adds some sort of “security” as devices not known to my dnsmasq won’t automatically get an IP on their own) I absolutely need that port to be free.

My thinking was that if I let the dnsmasq server run on the host, i then can

  • assign enp3s0 to my local network range (192.168.178.XXX)
  • assign bridge0 to the containers network range (10.254.116.XXX)

and have them separated, just like with the managed bridge. You have to use different interfaces for different address ranges in dnsmasq for DHCP. From what I found out, there is no other way to differentiate. And as long as the lxd managed bridge is running, it’s not possible to bind anything else to port 67.
Because of this, I created the bridge manually without binding it to the hardware interface, which netplan doesn’t support at the moment. DNS would not be the problem - DHCP is where everything falls apart at the moment.

Ok… I found out what I did wrong:
lxc network attach bridge0 <container> **eth0** **eth0** is bad. In my usecase it’s enough to do lxc network attach bridge0 <container> - then the bridge is attached to the container and can be used by it. A quick tcpdump -i bridge0 confirms - we have DHCP traffic from the containers using the bridge:

root@ascalon:/# tcpdump -i bridge0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:10:56.098111 IP6 fe80::216:3eff:fe99:d8e0 > ip6-allrouters: ICMP6, router solicitation, length 16
23:10:58.665941 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:99:d8:e0 (oui Unknown), length 287
23:11:03.835310 IP6 fe80::216:3eff:fe99:d8e0 > ip6-allrouters: ICMP6, router solicitation, length 16
23:11:07.183590 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:99:d8:e0 (oui Unknown), length 287
23:11:19.525348 IP6 fe80::216:3eff:fe99:d8e0 > ip6-allrouters: ICMP6, router solicitation, length 16
23:11:23.045649 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:99:d8:e0 (oui Unknown), length 287
23:11:34.368962 IP6 fe80::216:3eff:fe1d:1a91 > ip6-allrouters: ICMP6, router solicitation, length 16
23:11:36.503070 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:16:3e:1d:1a:91 (oui Unknown), length 286

Now if I install dnsmasq, it should do it’s job on both interfaces.

E: And if you want your interface to have a normal name in your container, use lxc network attach bridge0 <container> eth0 or something.