Migrating LXD 5.20 -> Incus 0.5 on Ubuntu 22.04 LTS (ZFS 2.1.5) and shiftfs support?

Incus
1

Hi

I’ve just been upgraded by snapd from LXD 5.19 to 5.20 on Ubuntu 22.04 LTS which has broken my LXD setup, such that I now have some containers that won’t start due to:

$ lxc start backups
Error: Failed to setup device mount "paperlessng": idmapping abilities are required but aren't supported on system
Try `lxc info --show-log backups` for more info

This is due to LXD 5.20 removing support for shiftfs, but 22.04 only having ZFS 2.1.5 which doesn’t support idmapped mounts. Hence, my containers that share volumes now won’t start.

So, first question: Does incus at 0.5 still have shiftfs in the packages for 22.04?

If so, now that I have LXD 5.20 installed, am I still able to migrate to Incus? I was planning to, but I hadn’t noticed that a) I was on latest/stable, and b) was surprised that Canonical would remove the feature from 5.20 that the LTS (that is in support for the next 8 years).

Yes, I’m probably supposed to be pinned to 5.0/stable, but snapped LXD latest/stable seems to be a be user-hostile in not detecting that it’s not actually compatible with the ZFS that’s shipped with the LTS. Maybe that’s a limitation of the snap ecosystem?

Anyway, what are my options here? Migrate to Incus? Switch all the containers to privileged?

Thanks very much for any help/advice you can offer!

2

Incus also removed shiftfs support. You can try VFS idmap shifting that is handled by kernel itself.

Read those two topics: one and two. You simply add shift=true to the device config.

If you’re using LXD, try disabling shiftfs first:

sudo snap set lxd shiftfs.enable=false
sudo systemctl reload snap.lxd.daemon

If VFS idmap shifting doesn’t work for you, you can try setting raw.idmap by hand. Check your users’ UID and GID on host and in container (id -u and id -u commands), then use command:

printf "uid 1000 1000\ngid 1000 1000" | incus config set <container_name> raw.idmap -

Where:

printf "uid <host_user_uid> <container_user_uid>\ngid <host_user_gid> <container_user_gid>" | incus config set <container_name> raw.idmap -
3

Incus doesn’t have shiftfs support, that was an Ubuntu-only feature which we removed with the initial Incus release.

An alternative may be for you to use my ZFS packages at GitHub - zabbly/zfs: OpenZFS builds which would then get you ZFS 2.2 on Ubuntu 22.04. Note that you will need to turn off secureboot on your system in case you have that currently enabled.

Once you are on ZFS 2.2 your current LXD should start behaving again (hopefully) and so will a migration to Incus.

4

Hi @qkiel @stgraber ; thanks for your replies. I didn’t realise that shiftfs was an Ubuntu only thing!

@stgraber I’m guessing to use ZFS 2.2 I’ll also need the zabbly kernel as well? What’s the risk associated with upgrading ZFS 2.1.5 to 2.2?

@qkiel I’ll have a go at the manual idmap stuff; I’m not quite sure how to get the id in the container (i’m guessing it’s in the config somewhere), as I can’t start the container! I could change the profile to drop the problematic volume and then try to add it back in.

Thanks again, both.

5

I would expect my ZFS packages to work fine with the Ubuntu kernel.
In general DKMS is supposed to have a higher priority than anything that comes with your local kernel.

The difficulty with maintaining the ZFS repository tends to be around building on newer kernels (6.7 currently) rather than on the older ones, so I wouldn’t expect any issues with using those packages on the older Ubuntu kernel.

And yeah, shiftfs only ever shipped with Ubuntu as a kernel sauce patch. It’s also never been stable enough for us to feel confident flipping the switch and having it be enabled for everyone by default. It was a useful learning experience though which was then used to shape the proper solution to the problem, VFS idmap support that @brauner then implemented and which has now seen native support by most filesystems.

6

Thanks @stgraber . I appear to be running a 5.15 kernel on my 22.04 LTS

$ uname -a
Linux bigstore 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# and

$ lsb_release -d
Description:    Ubuntu 22.04.4 LTS

I think I need a 6.x kernel for VFS idmap? Also will ZFS 2.2.x work on my kernel. I guess I need the HWE kernel to get 6.x?

7

Ah yeah, I think you’d need HWE just to have VFS idmap support.

While I’d expect ZFS 2.2 to build and load on your 5.15 kernel, it likely would not provide you with the VFS idmap bits.

8

Regular user in containers usually have UID and GID 1000, so after disabling shiftfs like mentioned above, try:

printf "uid $(id -u) 1000\ngid $(id -g) 1000" | lxc config set <container_name> raw.idmap -

The $(id -u) part is for the host user UID and GID. But if you can switch to HWE kernel, using VFS idmap is a better option.

9

So I set off on the path to upgrade the kernel to the hwe (6.5) and then was going to do ZFS. However, once I’d rebooted the server, I thought, on a whim, to try starting the containers and see what happens. Ands it’s a bit odd; everything has come up. I’m not sure about the idmaps, and there may be some gotchas, but it is working. This is with LXD 5.20, which I’ve now pinned, as I think it’s still possible to migrate to incus from that version.

I’ll do some more testing, and see if anything is still not correct, but this is an interesting development!

10

Well it finally clicked for me. Upgrading the ubuntu kernel to 6.5 hwe on 22.04 has brought in ZFS 2.2, but my zfsutils-linux (the userland ZFS components) is still at 2.1.5. However, LXD 5.20 has ZFS 2.2 userland components as part of the snap, and so idmap works out-of-the-box.

I’m guessing with incus, as it’s not snapped, that it’ll require a zfs 2.2 version of zfsutils-linux as otherwize incus won’t have a version of the userland tools that has the VFS idmap support. I probably need to update the zfsutils-linux to 2.2 anyway as thata mismatch may bite me in the future.

Thanks again for both of your help!

11

Yeah, my ZFS package repo includes both the updated DKMS package as well as the ZFS tools.

So you probably could just use it to get you the newer tools.

1 Like
12

Yeah, my ZFS package repo includes both the updated DKMS package as well as the ZFS tools.

So you probably could just use it to get you the newer tools.

Yes, I think I will do so. Thanks again.