Lxd + Netplan + Static IP's in same subnet HOW-TO

LXD
1

I have 17.10 server with LXD 2.21 installed via snap (2 containers).
I’ve configured host to have static IP and let containers be accessible from LAN.
My 01-netcfg.yaml in /etc/netplan:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      dhcp4: no
      dhcp6: no
  bridges:
    lxdbr0:
      dhcp4: no
      interfaces:
        - enp1s0
      addresses: [192.168.0.10/24]
      gateway4: 192.168.0.1
      nameservers:
        addresses: [192.168.0.1,8.8.8.8,8.8.4.4]
      parameters:
        stp: false
        forward-delay: 0

After running sudo netplan apply everything works, I have internet access from host.
ifconfig gives the following:

enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 2c:4d:54:4f:33:30  txqueuelen 1000  (Ethernet)
        RX packets 5412  bytes 5196030 (5.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4483  bytes 624400 (624.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 222  bytes 19478 (19.4 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 222  bytes 19478 (19.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lxdbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.10  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::60c5:c9ff:fee3:a28f  prefixlen 64  scopeid 0x20<link>
        ether 62:c5:c9:e3:a2:8f  txqueuelen 1000  (Ethernet)
        RX packets 2646  bytes 1973322 (1.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2640  bytes 332354 (332.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethAUK196: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fca4:95ff:fed5:a908  prefixlen 64  scopeid 0x20<link>
        ether fe:a4:95:d5:a9:08  txqueuelen 1000  (Ethernet)
        RX packets 1787  bytes 291994 (291.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2916  bytes 3204729 (3.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethBY974H: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fccc:7dff:fe51:3ae6  prefixlen 64  scopeid 0x20<link>
        ether fe:cc:7d:51:3a:e6  txqueuelen 1000  (Ethernet)
        RX packets 23  bytes 1958 (1.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 326  bytes 26711 (26.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethP62W14: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::fce2:c7ff:fe4d:d7da  prefixlen 64  scopeid 0x20<link>
        ether fe:e2:c7:4d:d7:da  txqueuelen 1000  (Ethernet)
        RX packets 46  bytes 3508 (3.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 317  bytes 26407 (26.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

After reboot there’s no internet in host (though, ping inside LAN subnet works and internet in containers works).
ifconfig shows exactly the same, except lxdbr0 has broadcast 0.0.0.0 instead of original value broadcast 192.168.0.255.
After running sudo netplan apply after reboot it starts working again and broadcast for lxdbr0 again fills with working value.
Any ideas?

I’ll provide any additional info/logs/configs, just tell me which one may help.

2

I have no idea on how netplan works, so I am doing a wild guess here.
Is netplan maybe running before the host boots and all the settings are overwritten by the network configuration of Ubuntu? Perhaps double checking the configuration of /etc/network/interfaces and /etc/network/interfaces.d/* might give you a hint on what might be going wrong on the initial boot.

3

/etc/network/interfaces is abandoned in 17.10

I figured out that Netplan doesn’t run/generate anything on each boot.
It just generates 3 networkd config files, but 1 of them gets overwritten by LXD networking, I guess

So what I really need then, is to set lxdbr0 broadcast address in LXD networking settings. Trying to find out how to do that.

4

Well, I don’t see settings for bridge broadcast in a list of available bridge settings…

5

Netplan is (IMO) another “systemd” approach (read as “fuckup”) to next linux component. Maybe it is ok for desktops, but after tinkering with it on 17.10 server i throw it away. Multiple network devices (mix of 10ge, 1ge) bonds, tunnels, vlans - it is pain in the ass and when i netplan apply - everything stopped for a while - i dont know how to manipulate only with one interface without affecting others.
If you want “old” eni behaviour back you must install server with
kernel option netcfg/do_not_use_netplan=true
and preseed
d-i netcfg/do_not_use_netplan true
With this - installer install ifupdown same way as previous versions of ubuntu server and netplan isn’t in action at all.
Or try to install ifupdown to your existing “netplaned” server…

6

Ok, I will ask the following then, as my initial task is to have host and containers in same subnet with static IP addresses, all having access to internet and host having access to containers.
Can I achieve it with LXD network config and networkd/Netplan?

7

Ok, this is the solution:

Install LXD (host only)

sudo apt remove --purge lxd lxd-client && apt install bridge-utils
sudo snap install lxd
sudo reboot
sudo lxd init

While installing, don’t create network bridge.

Setup network for host

sudo nano /etc/netplan/01-netcfg.yaml

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp1s0:
      dhcp4: no
      dhcp6: no
  bridges:
    br0:
      dhcp4: no
      dhcp6: no
      interfaces:
        - enp1s0
      addresses: [ 192.168.0.2/24 ]
      gateway4: 192.168.0.1
      nameservers:
          addresses:
              - 192.168.0.1
              - 8.8.8.8
              - 8.8.4.4
      parameters:
          stp: false
          forward-delay: 0

sudo netplan --debug apply

Reboot and confirm that config is fine:

sudo reboot
ifconfig -a
ping google.com

Edit default profile to make sure only bridge is present in config, with no extra nic’s

sudo lxc profile show default // add '> out.yaml' to output to file
sudo lxc profile edit default // add '< out.yaml' to read from file

### This is a yaml representation of the profile.
### Any line starting with a '# will be ignored.
###
### A profile consists of a set of configuration items followed by a set of
### devices.
###
### An example would look like:
### name: onenic
### config:
###   raw.lxc: lxc.aa_profile=unconfined
### devices:
###   eth0:
###     nictype: bridged
###     parent: lxdbr0
###     type: nic
###
### Note that the name is shown but cannot be changed

config: {}
description: Default LXD profile
devices:
  br0:
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/containers/apache

Setup network for container

sudo lxc exec <container> bash
nano /etc/netplan/50-cloud-init.yaml

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: no
      dhcp6: no
      addresses:
        - 192.168.0.5/24
      gateway4: 192.168.0.1
      nameservers:
        addresses:
          - 192.168.0.1
          - 8.8.8.8

netplan --debug apply

Now everything should work fine.

4 Likes
8

I figured out that Netplan doesn’t run/generate anything on each boot.

9

Yes. And at some point I was trying to run it with cron at each boot, but failed :smile:

10

@wociscz, one way to login and keep your shell while screwing with netplan/etc. is to ssh using the IPv6 Link Local address. So

ip -6 addr ls | grep fe80
....
inet6 fe80::56b2:3ff:fe0b:d84/64

And then from another host on the same subnet (“LAN”):
ssh root@fe80::56b2:3ff:fe0b:d84%eth0

now, this does not always work if the interface pushed to be “ifconfig down”, but it will survive most everything else including IP address changes. Otherwise, use "

lxc exec container -- /bin/bash

from the container host, if you have access to that.

I’m not a big systemd fan myself, but I can see why putting all the network configuration in a very machine manipulable format like netplan does is a good thing. I am trying to figure out how to live only on netplan so that my container can be transported elsewhere, while still having static IPv4 and IPv6 when they are in my data center. And everything bridged. (And make the same image work in docker on Windows.)

11

This was very helpful. I tweaked this procedure significantly simplifying it. Instead of editing the default profile and setting up bridges I simply attached the physical interface directly to the container.

Assuming the host has physical interface eno1 and the desired container interface eth0 the following command achieves the goal and gets the container running on the same network as the host

lxc network attach eno1 container_name eth0

The command even conveniently creates the container netplan configuration file that uses DHCP by default but can be changed.

2 Likes
12

Thank you for this! this worked perfectly!