How to setup LXD on Ubuntu 20.04 with whole rootfs already in ZFS?

#1

I eventually have a question.

I have installed Ubuntu 20.04 on ZFS, using the new experimental feature to setup the disk with ZFS, and boot on ZFS.

When a user installs LXD and attempts to lxd init, they can select the defaults and get created a loop file at /var/snap/lxd/common/lxd/disks/default.img, default pool name is default.

What is the recommended configuration for LXD to reuse the existing ZFS setup?
lxd init does not detect the presence of ZFS, and does not suggest to create a pool in there.
You are asked whether to specify a block device.

Ubuntu uses the pool names bpool (boot) and rpool (root). While LXD uses default.
Wouldn’t it be more descriptive if the pool name was something specific to LXD?

#2 
Name of the storage backend to use (zfs, ceph, btrfs, dir) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: no
Name of the existing ZFS pool or dataset: rpool/lxd
#3

When I select not to create a new storage pool, the wizard does not ask for an existing ZFS pool. On LXD 3.23:

$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: no
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
...

When I specify instead to create a new storage pool, I get the following.

$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: yes
Name of the new storage pool [default=default]: rpool/lxd 
Name of the storage backend to use (btrfs, dir, lvm, zfs, ceph) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: no
Name of the existing ZFS pool or dataset: rpool/lxd
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
...
Error: Failed to create storage pool 'rpool/lxd': Storage pool names may not contain slashes

When I repeat the reinstallation of LXD, I take care to sudo snap remove lxd --purge. I do not think there are remnants on the system that may confuse LXD and the lxc init process.

I ended up configuring the storage pool with

lxc storage create default zfs source=rpool/LXD
#4

I got a few more issues.

I create manually the default storage pool and configure the default LXD profile to use it.
I launched a container, and it started successfully according to lxc list. Got an IP address.
But when I lxc exec mycontainer -- /bin/sh, this command gets stuck.
There is not output, and does not respond to Ctrl+C either. I can only kill -9 the process from another terminal.

I can stop the container. But when I try to delete it, I get the following.

Error: Error deleting storage volume: Failed to run: zfs destroy -r rpool/LXD/containers/mycontainer: cannot destroy 'rpool/LXD/containers/mycontainer': dataset is busy

The logs did not show something interesting, apart from /var/snap/lxd/common/lxd/logs/mycontainer/lxc.log:

lxc mycontainer 20200327025357.268 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1136 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.mycontainer"
lxc mycontainer 20200327025357.269 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1136 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.mycontainer"
lxc mycontainer 20200327025357.274 WARN     cgfsng - cgroups/cgfsng.c:fchowmodat:1448 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
#5

Yeah, the lxc exec issue is being worked on now.

#6

Hi,

I’m also working with LXD, ZFS and Ubuntu 20.04 and your answer question me.
Are we really allow to do that? (create a rpool/lxd)

Note that LXD will assume it has full control over the ZFS pool or dataset. It is recommended to not maintain any non-LXD owned filesystem entities in a LXD zfs pool or dataset since LXD might delete them.

So, I manually (i.e. without the installer) installed my Ubuntu system to create an additionally partition to store my LXD ZFS pool.

#7

Note that it says zfs pool or dataset. rpool/lxd is a dataset. If you tell LXD to use that, it will assume full control of whatever is below rpool/lxd but it will not mess with anything outside of it.

That’s how I have my own laptop setup (though my setup predates the automated Ubuntu option).

1 Like
#8

Here is a sudo lxd init for this,

$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: no
Do you want to configure a new storage pool? (yes/no) [default=yes]: yes
Name of the new storage pool [default=default]: default
Name of the storage backend to use (btrfs, dir, lvm, zfs, ceph) [default=zfs]: zfs
Create a new ZFS pool? (yes/no) [default=yes]: no
Name of the existing ZFS pool or dataset: rpool/LXD
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like LXD to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
1 Like