Mapping storage volume as block device

seffyroff · October 3, 2018, 7:01pm

Hey folks! I have a quick question that probably has a quick ‘no’ answer

Can I map an LXD storage volume into a container as a block device?

I’m setting up an LXD-hosted Ceph cluster, and deploying Ceph inside each container using Docker (the official ceph/daemon docker images are really great for rapid iteration). I expect there’s more efficient and LXD-aligned ways of accomplishing this, and I’m open to helpful suggestions. Right now I was hoping to create a volume inside my LXD created ZFS storage pool, and attach that to the container as a unix-block device, then hand that block device to ceph-osd.

simos · October 3, 2018, 7:15pm

Hi!

If I understand correctly, you can instead mount directly the block device to a container.
See the lxc config device xxx command, where you use the disk device.

github.com

lxc/lxd/blob/master/doc/containers.md#type-disk

# Container configuration
## Properties
The following are direct container properties and can't be part of a profile:

 - `name`
 - `architecture`

Name is the container name and can only be changed by renaming the container.

Valid container names must:

 - Be between 1 and 63 characters long
 - Be made up exclusively of letters, numbers and dashes from the ASCII table
 - Not start with a digit or a dash
 - Not end with a dash

This requirement is so that the container name may properly be used in
DNS records, on the filesystem, in various security profiles as well as
the hostname of the container itself.

This file has been truncated. show original

seffyroff · October 3, 2018, 7:27pm

Thanks for your reply @simos.
If I understand what you’re referencing correctly, then I had considered this as a ‘Plan B’ with the following caveats:

I currently have the host block device allocated completely to the LXD ZFS storage pool. I assume I’d need to free up some space on the physical disk and have a new, unformatted partition as a block device to pass in this case - I was hoping I could avoid doing this at the physical level.
Could I instead use zpool on the host to add a ZVOL to the existing pool, and map that as the block device to the container? Would that upset LXD? (I.e. is LXD managing the entire ZFS pool and unaware of changes made by external tools?, in which case I’d need a second ZVOL for the LXD pool and pass that during lxd init?).

(edit: I meant ZVOL, not vdev, i think)

simos · October 3, 2018, 7:38pm

In your case, you can use volumes. See https://blog.ubuntu.com/2017/07/12/storage-management-in-lxd-2-15

Specifically, you can create a volume in some storage pool, then attach to a container.
It’s the lxc storage volume --help command.

seffyroff · October 3, 2018, 8:44pm

So, I already tried this. The issue being that volumes get mapped to filesystems rather than block devices in the container, as far as I can tell. Ceph wants a block device (logical partition or device) to setup as OSD storage. I guess ‘Plan C’ would be to run Ceph inside KVM inside LXD and create virtual block devices with KVM.