How to move lxd zfs-pool to another zfs dataset on the same host

Hi,

I am rebuilding my workstation, including a reinstall of Ubuntu 16.04.x LTS.

Currently my lxd zfs storage uses a whole disk/pool. I would like to:

  • move to lxd zfs pool to a dataset within another pool.
  • tell lxd/c to use the new copy of the original pool.
  • not have to individually export and import each container

I have got this far, but find it unclear how I should proceed. I’ve spent a few hours with google, but nothing clearly covers the topic of telling lxd to use an existing populated zfs dataset - in my interpretation.

sudo -i
lxd shutdown
zfs snapshot -r lxd-virt@migrate
zfs send -R lxd-virt@migrate | zfs receive -F dev-pool/lxd-storage 

Is there a way to progress from here? Or do I need to take a different approach. I’m happy to document any clear solution.

Cheers,
–Peter G

2 Likes

It’s certainly possible, but it’s not exactly supported so is slightly involved.

You’ll indeed want to transfer all the datasets to the new zpool, which the above will do fine.
Then you’ll need to remove that @migrate snapshot from the target so that it won’t confuse LXD. Then you should mark down all the mountpoint properties of the source datasets, then unset them all and set the original value on the target datasets.

Lastly you’ll need to do a tiny bit of DB surgery to have LXD look at the new place.
That bit depends on whether you’re on LXD 2.0.x (default on 16.04) or if you’re on 2.x (likely 2.21 if using backports).

In the former case, something like this should do the trick:

sudo sqlite3 /var/lib/lxd/lxd.db "UPDATE config SET value='POOL/DATASET' WHERE value='POOL';"

In the latter case, it’ll instead be:

sudo sqlite3 /var/lib/lxd/lxd.db "UPDATE storage_pools_config SET value='POOL/DATASET' WHERE value='POOL';"

Once that’s done, you can start LXD again and things should be fine.

1 Like

Hello,
Here we are in 2022, and I have roughly the same question. I’d like to know if there is any update on the means to achieve my goal.

When I originally did lxd init on my current Ubuntu 20.04 workstation, I believe I accepted the default location, which placed into rpool/lxd which unfortunately shares space with zsysctl.

lxc containers are filling that pool, causing problems with zsysctl.

I would like to move my lxc default container to a new zfs dataset on a different device - which has no impending space shortages.

I can still do the zfs send| zfs receive operation. Is there now a way of changing the source of the default pool in an API-friendly way?

Cheers…

You can move instances between storage pools now.
So create a new ZFS storage pool where you want using lxc storage create myzfs zfs source=... and then move the instances using:

lxc launch images:ubuntu/jammy c1 -s zfs
sudo zfs list | grep containers
zfs/containers                                                                              894K  39.0G       24K  legacy
zfs/containers/c1                                                                           870K  39.0G      232M  legacy
zfs/deleted/containers                                                                       24K  39.0G       24K  legacy

lxc storage create zfs2 zfs
sudo zfs list | grep containers
zfs/containers                                                                              894K  39.0G       24K  legacy
zfs/containers/c1                                                                           870K  39.0G      232M  legacy
zfs/deleted/containers                                                                       24K  39.0G       24K  legacy
zfs2/containers                                                                              24K  26.6G       24K  legacy
zfs2/deleted/containers                                                                      24K  26.6G       24K  legacy

lxc stop c1
lxc move c1 -s zfs2

sudo zfs list | grep containers
zfs/containers                                                                               24K  39.0G       24K  legacy
zfs/deleted/containers                                                                       24K  39.0G       24K  legacy
zfs2/containers                                                                             232M  26.4G       24K  legacy
zfs2/containers/c1                                                                          232M  26.4G      232M  legacy
zfs2/deleted/containers                                                                      24K  26.4G       24K  legacy

lxc storage delete zfs

Be aware there are some issues with moving instances between ZFS pools on LXD 5.0.0 which will be fixed in LXD 5.0.1 soon to be released. They are already fixed in LXD 5.4.

Thanks Thomas,
Excellent!
How do I change the default pool for container creation to zfs2?

You can do it via lxc profile edit default and change the root disk’s pool property to the name of the new storage pool. Or you can do lxc profile device set default root pool=<pool name>.

Note: This will modify the settings for all instances using the default profile, so if there are other instances on a different pool the operation will fail.

I’d like to revisit the sense of my original question.

I would like:

  • to move my default storage pool (not the individual containers) - which is in a zfs dataset rpool/lxd, to another zfs pool, which is not competing for space with my Ubuntu system configuration on rpool, and giving me hell with zsysctl.

  • example destination to be the zfs dataset MyWorkstation/lxd

  • to do this directly with zfs send -R rpool/lxd | zfs receive MyWorkstation

  • delete the old storage rpool\lxd

zfs send/receive is much faster than moving individual containers, and seems more in the spirit of using zfs.

I am also concerned about the semantics of lxc move containers which are copied from container snapshots. What happens in the destination if I copy two clones copied from the same container snapshot one-at-a-time.

would what I want be something like zfs snapshot, zfs send | sfs receive, lxd recover?

Thanks…