Can't create unprivilleged containers on zfs

LXC
1

Hello, I have the following problem:

root@kapnet:/home/pi# zfs list
NAME USED AVAIL REFER MOUNTPOINT
kapnet-raidz 124G 1,67T 124G /kapnet-raidz

root@kapnet:/home/pi# lxc-create -t download -n lxc-test --dir /kapnet-raidz/lxc-test

I select ubuntu bionic armhf and I get the following errors:

lxc-create: lxccontainer.c: create_run_template: 1297 container creation template for lxc-test failed
lxc-create: tools/lxc_create.c: main: 318 Error creating lxc-create: lxccontainer.c: create_run_template: 1297 container creation template for lxc-test failed
lxc-create: tools/lxc_create.c: main: 318 Error creating container lxc-test
ile or directory
tar: ./root: Cannot mkdir: Permission denied
tar: ./root/.profile: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd: Cannot mkdir: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd/systemd-rfkill: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd/systemd-cryptsetup: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission deniedcontainer lxc-test
ile or directory
tar: ./root: Cannot mkdir: Permission denied
tar: ./root/.profile: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd: Cannot mkdir: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd/systemd-rfkill: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd/systemd-cryptsetup: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd/systemd-sulogin-shell: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd/systemd-fsckd: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd/systemd-hostnamed: Cannot open: No such file or directory
tar: ./lib: Cannot mkdir: Permission denied
tar: ./lib/systemd/systemd-sleep: Cannot open: No such file or directory

This error is repeated for all files and finally the creation fails.

The creation with the same command works if i change --dir to something on an ext4 filesystem.

2

You need to add the -B parameter to lxc-create to indicate your backing store (ZFS).

If you are just starting to use system containers, I suggest to try instead with LXD.
With plain LXC, the commands look like lxc-create. With LXD, the commands are of the form lxc launch ubuntu:18.04 mycontainer. For LXC, you need to complete an initial setup by running sudo lxd init, and then you can create containers right away.

3

Hello @simos ,
thanks for replying.

I am on Raspbian (raspberry pi3) which does not have LXD. I tried with -B zfs but I have almost the same problem:

root@kapnet:/home/pi# lxc-create -t download -n lxc-test -B zfs

cannot create 'lxc/lxc-test': no such pool 'lxc'
lxc-create: lxccontainer.c: do_bdev_create: 1042 Failed to create backing store type zfs
lxc-create: lxccontainer.c: do_lxcapi_create: 1518 Error creating backing store type zfs for lxc-test
lxc-create: tools/lxc_create.c: main: 318 Error creating container lxc-test

I dont know why expects me to have a “lxc” pool… Anyway, I renamed my pool to lxc and ran the command again:

root@kapnet:/home/pi# lxc-create -t download -n lxc-test -B zfs

Setting up the GPG keyring
Downloading the image index

---
DIST	RELEASE	ARCH	VARIANT	BUILD
---
alpine	3.6	amd64	default	20190531_17:39
alpine	3.6	arm64	default	20190531_17:39
alpine	3.6	armhf	default	20190531_17:39
alpine	3.6	i386	default	20190531_17:39
alpine	3.7	amd64	default	20190531_17:38
alpine	3.7	arm64	default	20190531_17:39
alpine	3.7	armhf	default	20190531_17:39
alpine	3.7	i386	default	20190531_17:39
alpine	3.8	amd64	default	20190531_17:38
alpine	3.8	arm64	default	20190531_17:41
alpine	3.8	armhf	default	20190531_17:39
alpine	3.8	i386	default	20190531_17:39
alpine	3.8	ppc64el	default	20190531_17:39
alpine	3.8	s390x	default	20190531_17:39
alpine	3.9	amd64	default	20190531_17:39
alpine	3.9	arm64	default	20190531_17:41
alpine	3.9	armhf	default	20190531_17:39
alpine	3.9	i386	default	20190531_17:39
alpine	3.9	ppc64el	default	20190531_17:39
alpine	3.9	s390x	default	20190531_17:39
alpine	edge	amd64	default	20190531_17:38
alpine	edge	arm64	default	20190531_17:39
alpine	edge	armhf	default	20190531_17:39
alpine	edge	i386	default	20190531_17:39
alpine	edge	ppc64el	default	20190531_17:39
alpine	edge	s390x	default	20190531_17:39
alt	Sisyphus	amd64	default	20190601_01:17
alt	Sisyphus	arm64	default	20190601_01:17
alt	Sisyphus	i386	default	20190601_01:17
alt	p8	amd64	default	20190601_01:17
alt	p8	i386	default	20190601_01:18
alt	p9	amd64	default	20190601_01:17
alt	p9	arm64	default	20190601_01:17
alt	p9	i386	default	20190601_01:18
archlinux	current	amd64	default	20190601_04:18
archlinux	current	arm64	default	20190601_04:18
archlinux	current	armhf	default	20190601_04:18
centos	6	amd64	default	20190601_07:08
centos	6	i386	default	20190601_07:08
centos	7	amd64	default	20190601_07:08
centos	7	arm64	default	20190601_07:08
centos	7	armhf	default	20190601_07:08
centos	7	i386	default	20190601_07:08
centos	7	ppc64el	default	20190601_07:08
debian	buster	amd64	default	20190601_05:24
debian	buster	arm64	default	20190601_05:24
debian	buster	armel	default	20190601_05:41
debian	buster	armhf	default	20190601_05:24
debian	buster	i386	default	20190601_05:24
debian	buster	ppc64el	default	20190601_05:24
debian	buster	s390x	default	20190601_05:24
debian	jessie	amd64	default	20190601_05:24
debian	jessie	arm64	default	20180626_05:25
debian	jessie	armel	default	20190601_05:40
debian	jessie	armhf	default	20190601_05:24
debian	jessie	i386	default	20190601_05:24
debian	jessie	powerpc	default	20180626_05:25
debian	jessie	ppc64el	default	20180626_05:25
debian	jessie	s390x	default	20180626_05:25
debian	sid	amd64	default	20190601_05:24
debian	sid	arm64	default	20190601_05:24
debian	sid	armel	default	20190601_05:39
debian	sid	armhf	default	20190601_05:24
debian	sid	i386	default	20190601_05:24
debian	sid	powerpc	default	20180708_05:25
debian	sid	ppc64el	default	20190601_05:24
debian	sid	s390x	default	20190601_05:24
debian	stretch	amd64	default	20190601_05:24
debian	stretch	arm64	default	20190601_05:27
debian	stretch	armel	default	20190601_05:40
debian	stretch	armhf	default	20190601_05:24
debian	stretch	i386	default	20190601_05:24
debian	stretch	ppc64el	default	20190601_05:24
debian	stretch	s390x	default	20190601_05:24
fedora	28	amd64	default	20190531_20:55
fedora	28	arm64	default	20190531_20:33
fedora	28	armhf	default	20190531_20:33
fedora	28	ppc64el	default	20190227_20:33
fedora	28	s390x	default	20190531_20:33
fedora	29	amd64	default	20190531_20:55
fedora	29	arm64	default	20190531_20:36
fedora	29	armhf	default	20190531_20:33
fedora	29	ppc64el	default	20190531_20:33
fedora	29	s390x	default	20190531_20:33
fedora	30	amd64	default	20190531_20:33
fedora	30	arm64	default	20190531_20:33
gentoo	current	amd64	default	20190531_16:07
gentoo	current	armhf	default	20190531_16:07
gentoo	current	i386	default	20190531_16:07
gentoo	current	ppc64el	default	20190531_16:07
gentoo	current	s390x	default	20190531_16:07
opensuse	15.0	amd64	default	20190601_04:20
opensuse	15.0	arm64	default	20190601_04:20
opensuse	15.0	ppc64el	default	20190322_04:20
opensuse	42.3	amd64	default	20190601_04:20
opensuse	tumbleweed	amd64	default	20190601_04:20
opensuse	tumbleweed	arm64	default	20190601_04:23
opensuse	tumbleweed	i386	default	20190601_04:20
opensuse	tumbleweed	ppc64el	default	20190601_04:20
oracle	6	amd64	default	20190601_07:46
oracle	6	i386	default	20190601_07:46
oracle	7	amd64	default	20190601_08:20
plamo	6.x	amd64	default	20190601_01:33
plamo	6.x	i386	default	20190601_01:33
plamo	7.x	amd64	default	20190601_01:33
ubuntu	bionic	amd64	default	20190601_07:42
ubuntu	bionic	arm64	default	20190601_07:42
ubuntu	bionic	armhf	default	20190601_08:03
ubuntu	bionic	i386	default	20190601_07:42
ubuntu	bionic	ppc64el	default	20190601_07:42
ubuntu	bionic	s390x	default	20190601_07:42
ubuntu	cosmic	amd64	default	20190601_07:42
ubuntu	cosmic	arm64	default	20190601_07:57
ubuntu	cosmic	armhf	default	20190601_08:39
ubuntu	cosmic	i386	default	20190601_07:42
ubuntu	cosmic	ppc64el	default	20190601_07:55
ubuntu	cosmic	s390x	default	20190601_07:42
ubuntu	disco	amd64	default	20190601_07:42
ubuntu	disco	arm64	default	20190601_07:53
ubuntu	disco	armhf	default	20190601_07:42
ubuntu	disco	i386	default	20190601_07:42
ubuntu	disco	ppc64el	default	20190601_07:42
ubuntu	disco	s390x	default	20190601_07:52
ubuntu	eoan	amd64	default	20190601_07:42
ubuntu	eoan	arm64	default	20190601_07:45
ubuntu	eoan	armhf	default	20190601_07:42
ubuntu	eoan	i386	default	20190601_07:42
ubuntu	eoan	ppc64el	default	20190601_07:57
ubuntu	eoan	s390x	default	20190601_07:42
ubuntu	trusty	amd64	default	20190601_07:42
ubuntu	trusty	arm64	default	20190601_07:45
ubuntu	trusty	armhf	default	20190601_07:42
ubuntu	trusty	i386	default	20190601_07:42
ubuntu	trusty	powerpc	default	20180824_07:43
ubuntu	trusty	ppc64el	default	20190601_07:42
ubuntu	xenial	amd64	default	20190601_07:42
ubuntu	xenial	arm64	default	20190601_07:42
ubuntu	xenial	armhf	default	20190601_07:42
ubuntu	xenial	i386	default	20190601_07:42
ubuntu	xenial	powerpc	default	20180824_07:44
ubuntu	xenial	ppc64el	default	20190601_07:42
ubuntu	xenial	s390x	default	20190601_07:42
---

Distribution: ubuntu 
Release: bionic
Architecture: armhf

Using image from local cache
Unpacking the rootfs
tar: ./usr/lib/dbus-1.0/dbus-daemon-launch-helper: Cannot change ownership to uid 0, gid 107: Operation not permitted
tar: ./usr/local/lib/python3.6/dist-packages: Cannot change ownership to uid 0, gid 50: Operation not permitted
tar: ./usr/local/lib/python3.6: Cannot change ownership to uid 0, gid 50: Operation not permitted
tar: ./usr/bin/chage: Cannot change ownership to uid 0, gid 42: Operation not permitted
tar: ./usr/bin/crontab: Cannot change ownership to uid 0, gid 105: Operation not permitted
tar: ./usr/bin/expiry: Cannot change ownership to uid 0, gid 42: Operation not permitted
tar: ./usr/bin/ssh-agent: Cannot change ownership to uid 0, gid 108: Operation not permitted
tar: ./usr/bin/wall: Cannot change ownership to uid 0, gid 5: Operation not permitted
tar: ./etc/gshadow: Cannot change ownership to uid 0, gid 42: Operation not permitted
tar: ./etc/shadow: Cannot change ownership to uid 0, gid 42: Operation not permitted
tar: ./var/lib/apt/lists/partial: Cannot change ownership to uid 104, gid 0: Operation not permitted
tar: ./var/lib/apt/lists/auxfiles: Cannot change ownership to uid 104, gid 0: Operation not permitted
tar: ./var/spool/rsyslog: Cannot change ownership to uid 102, gid 4: Operation not permitted
tar: ./var/spool/cron/crontabs: Cannot change ownership to uid 0, gid 105: Operation not permitted
tar: ./var/cache/apt/archives/partial: Cannot change ownership to uid 104, gid 0: Operation not permitted
tar: ./var/log/journal: Cannot change ownership to uid 0, gid 101: Operation not permitted
tar: ./var/log/wtmp: Cannot change ownership to uid 0, gid 43: Operation not permitted
tar: ./var/log/btmp: Cannot change ownership to uid 0, gid 43: Operation not permitted
tar: ./var/log/lastlog: Cannot change ownership to uid 0, gid 43: Operation not permitted
tar: ./var/log/apt/term.log: Cannot change ownership to uid 0, gid 4: Operation not permitted
tar: ./var/log: Cannot change ownership to uid 0, gid 106: Operation not permitted
tar: ./var/local: Cannot change ownership to uid 0, gid 50: Operation not permitted
tar: ./var/mail: Cannot change ownership to uid 0, gid 8: Operation not permitted
tar: ./sbin/unix_chkpwd: Cannot change ownership to uid 0, gid 42: Operation not permitted
tar: ./sbin/pam_extrausers_chkpwd: Cannot change ownership to uid 0, gid 42: Operation not permitted
tar: ./home/ubuntu/.bash_logout: Cannot change ownership to uid 1000, gid 1000: Operation not permitted
tar: ./home/ubuntu/.bashrc: Cannot change ownership to uid 1000, gid 1000: Operation not permitted
tar: ./home/ubuntu/.profile: Cannot change ownership to uid 1000, gid 1000: Operation not permitted
tar: ./home/ubuntu: Cannot change ownership to uid 1000, gid 1000: Operation not permitted
tar: Exiting with failure status due to previous errors
lxc-create: lxccontainer.c: create_run_template: 1297 container creation template for lxc-test failed
cannot destroy 'lxc/lxc-test': dataset is busy
lxc-create: lxccontainer.c: container_destroy: 2395 Error destroying rootfs for lxc-test
lxc-create: tools/lxc_create.c: main: 318 Error creating container lxc-test

This time it created the files that are owned by root user and group (which is id 100000 because im using unprevileged containers) but not the ones that have a different user id or group id than 100000.

# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
lxc            124G  1,67T   124G  /lxc
lxc/lxc-test   430M  1,67T   430M  /var/lib/lxc/lxc-test/rootfs

# ls -l /var/lib/lxc/
drwxrwx--- 3 100000 100000 4096 Ιούν  1 12:33 lxc-test
4

I use LXD with snap on raspberry on an Ubuntu 18.04 LTS derivative. Not with ZFS though.

5

@gpatel-fr thanks, I will have that in mind. But first I would like to see if I can avoid changing OS. On Raspbian I tried installing LXD through snap and it didn’t even have an option for zfs…

$ sudo /snap/bin/lxd init
...
Name of the storage backend to use (btrfs, ceph, dir, lvm) [default=btrfs]:
6

I would expect that LXD would not show ZFS in the list of storage backends, if the kernel does not have ZFS support. Or, the snap package does not have built-in the corresponding version of ZFS utilities for the version of ZFS in the kernel.

7

@simos Actually I don’t have ZFS in a kernel module but instead I use ZFS through FUSE with the package zfs-fuse.

8

If you installed zfs-fuse after you have installed LXD, then you need to restart the LXD server.
Reference: "lxd init" fails to find zfs tool

9

@simos No, I installed the lxd snap after zfs-fuse.
The command:
sudo systemctl reload snap.lxd.daemon
did not change anything.

10

For anyone interested in the solution that worked for me:

  1. The zfs-fuse did not support acltype=posixacls so I had to install zfs-dkms (as I describe in this issue).

  2. Raspbian had a bug (probably inhereted from Debian) and sudo apt install zfs-dkms failed.

  3. I worked around this using this guide for zfs 0.7.13 (read my comment).

Now I have working LXC (unprivileged) containers in ZFS on a Raspberry Pi 3 with Raspbian Stretch 9.9 :smile:

11

Ah, good to hear.

And yeah, I’d strongly urge everyone to stay away from zfs-fuse, it’s certainly a nice trick but the performance is going to be bad, it’s a bit behind on features and we’ve also seen some report of odd behaviors even during what should be normal zfs operations in the past.