[LXD] ZFS block mode

Project LXD
Status Implemented
Author(s) @monstermunchkin
Approver(s) @stgraber @tomp
Release 5.11
Internal ID LX036

Abstract

This adds support for ZFS block mode.

Rationale

LXD currently doesn’t support ZFS block mode. Supporting this will allow the use of btrfs, ext4, and xfs filesystems on top of ZFS.

Specification

Design

The following new configuration keys will be added for ZFS storage pools:

  • volume.zfs.block_mode
  • volume.block.mount_options
  • volume.block.filesystem

Setting volume.zfs.block_mode to true on the storage pool will turn on ZFS block mode. This setting can be changed per-volume, setting zfs.block_mode on the volume itself. This will use ZFS zvols instead of ZFS datasets. Zvols need to be formatted with a filesystem. The default FS will be ext4 but can be configured using the volume.block.filesystem configuration key.

Once ZFS block mode is turned on, new instances will be created as zvols. Existing instances which use datasets will not be converted to zvols. Also, copying dataset based instances will not create zvols but instead create new datasets.

Zvols will follow the same naming pattern as the existing datasets except for images. These will be suffixed with the filesystem they were formatted with.

Example:

default/images/999c7ff6585ab0bcc6ff21736407cfd494dcdde2c68907512fef2b307a25c36b_ext4 

If there are both a zvol and a dataset for the same image, this will not create an additional database record.

When deleting image storage volumes, it will attempt to delete both datasets and zvols.

API changes

No API changes.

CLI changes

No CLI changes.

Database changes

No database changes.

Upgrade handling

No upgrading handling.

Further information

No further information.

Hi

Just a bit of food for thought since i don’t have many instances with a requirement for this because Docker Swarm doesn’t like being in an unprivileged container.

If possible, it would be good to have these settings available at the profile/instance level because not all instances will need zvols on a storage pool.

Thanks

As this allows for the coexistence of filesystem zvols (volumes created after zfs.block_mode was enabled) as well as filesystem datasets in a single pool (volumes create before zfs.block_mode was enabled), would it be more consistent to have this as a volume.zfs.block_mode setting?

Please could you provide some detail on storage DB record changes and ZFS zvol naming/layout?

I think I agree but rather than special case this one volume setting, I think I’d prefer we think through a way to apply storage volume setting through the root disk device definition.

The same could then be used to set a different filesystem, quota mode, …

So out of scope for this work, but something that I think would make sense in general.

Done.

Apologize for jumping into an ongoing team development.
This new feature is promising, yet when I create a container with a block device as storage backend, I also eventually would want to attach another block device as second hdd …

Can in this context, the below feature request be revisited?

Add lxc config device add disk --type=block for containers · Issue #10077 · lxc/lxd (github.com)

Hello,

I have a lxd server with 2 storages:

The old one: with LVM controller
The new one: with ZFS driver, zfs.block_mode = true

When I copy or move containers from LVM storage to ZFS storage, the zfs.block_mode flag is not honored and the containers are created on ZFS storage as a dataset.
Also if I export containers from the LVM storage and I import them into ZFS storage they are created as dataset instead of zvols.

I understand that “existing instances using datasets will not be converted to zvols” when they are copied or moved between zfs data set, but why zfs.block_mode = true can’t be honored in previous described cases?

I found no way to migrate my db/docker containers (that are not compatible with zfs datasets) from LVM to ZFS and create all of them from scratch is not an option, is there any solution?

Regards