What’s shiftfs
shiftfs is a kernel filesystem which was initially developed by James Bottomley, then stabilized and extended by @sforshee and @brauner.
That filesystem acts like an overlay on top of an existing mount on the host and then allows mounting inside the container, shifting uid/gid on all filesystem operations. This allows for instant creation and startup of unprivileged containers as no costly filesystem remapping is needed on creation or startup.
In tests, on fast systems backed by NVME storage, we’ve seen the typical lxc launch time for an Ubuntu 18.04 image go from around 2.5s down to just 500ms. For systems using hard disks, the difference should be far more noticeable, possibly saving tens of seconds.
shiftfs will also allow for multiple containers that use non-overlapping maps (security.idmap.isolated=true) to share custom storage volumes, which is currently impossible.
Where can it be found
At this time, shiftfs isn’t mainline. It’s included in the Ubuntu kernel starting with 5.0 and some other Linux distributions may pick it up. It should also be reasonably easy to package as a DKMS package for distributions that ship a suitably recent kernel.
Ubuntu users have it in their kernel out of the box, this can be confirmed with modinfo shiftfs.
As we had to fix a few early issues, you should make sure that your kernel is fully up to date and you are booted onto it.
Limitations
None that we’re aware of at present for Ubuntu users.
Note that in addition to shiftfs, the Ubuntu kernel also carries changes to overlayfs which allows its use by unprivileged containers and make it cooperate with shiftfs. Users of shiftfs on other Linux distributions may therefore be unable to use overlayfs on top of `shiftfs.
How can it be used
The easiest is with the LXD snap on version 3.12 or higher. We currently have it disabled out of the box because of the above limitations but you can opt into it if those don’t apply to you.
To turn it on, do:
- sudo snap set lxd shiftfs.enable=true
- sudo systemctl reload snap.lxd.daemon
You can then run:
- lxc info
And should see:
shiftfs: "true"
Under the kernel_features header. If you see it, then you’ve got it enabled.
After that, any newly created container will use shiftfs.
To convert an existing container, the easiest is to temporarily convert it to privileged:
- lxc config set NAME security.privileged true
- lxc restart NAME
- lxc config unset NAME security.privileged
- lxc restart NAME
Feedback
Any feedback on shiftfs would be appreciated and we will be updating the limitations section above as we become aware of new issues.
Once all of them are addressed, we will slowly start rolling it out by default to our users as this is a significant performance improvement and completely avoids the rather complex and risky remapping logic that we have to use today.