Docker with overlay driver in LXD cluster not working

harsh-vishwakarma · October 20, 2020, 10:19am

I have been trying to work docker inside lxc container to work with overlay driver. LXD cluster is setup on Ubuntu 18.04 VM with zfs pool. Each time I install docker it starts with vfs storage driver, but when I change with overlay (already added as linux module in lxd profile ) docker fails to start. I even tried setting docker storage driver to zfs which fails too. I am new with LXD technology and can’t figure out if it is possible to run docker with overlay storage driver on LXD cluster containers on ZFS pool.

stgraber · October 20, 2020, 2:06pm

The overlay kernel filesystem cannot be used on top of zfs.
That’s not a LXD limitation but a hard kernel one as the two filesystems are just incompatible.

For similar environments in the past, some have created a separate storage pool which is backed by something compatible with overlay (dir on top of ext4 is the most solid) and then assigned one storage volume per container, mounting that volume at /var/lib/docker.

harsh-vishwakarma · October 20, 2020, 2:07pm

That’s exactly what I needed to know thanks.

kassius · July 15, 2021, 11:43pm

Hello! I got what you said but have no idea on how to do it, can you give an idea on how to do this? Thank you!

tomp · July 19, 2021, 8:48am

Also might be worth pointing out that BTRFS backed storage also doesn’t play completely nicely with docker because docker then uses BTRFS’ native snapshotting functionality instead of overlay, which isn’t fully supported by LXD when transferring the container between hosts/pools.

See

github.com/lxc/lxd

LXD does not send BTRFS subvolume snapshots efficiently in relation to parent snapshots

opened 10:54AM - 01 Feb 21 UTC

tomponline

Bug

Originally discussed here https://discuss.linuxcontainers.org/t/no-space-left-on…-device-while-changing-the-btrfs-storage-pool-of-an-lxd-container-with-docker/10040/34?u=tomp While LXD correctly provides the parent subvolume to the BTRFS send tool in relation to LXD created snapshots when performing an optimized migration or export, it does not appear to use the relationship when transferring subvolumes inside the container. This causes size amplification on the target when using a tool that creates many subvolumes (such as docker) inside the container.

trumee · May 3, 2022, 4:42pm

I took your advice and created a dir based container.

# lxc storage list         
+----------+--------+--------------------------+-------------+---------+---------+
|   NAME   | DRIVER |          SOURCE          | DESCRIPTION | USED BY |  STATE  |
+----------+--------+--------------------------+-------------+---------+---------+
| ext4pool | dir    | /mnt/dockerssd/ext4pool/ |             | 1       | CREATED |

However, i still get errors when running docker instance.

[11817.859223] overlayfs: upper fs does not support file handles, falling back to index=off.
[11817.859229] overlayfs: fs on '/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/userxattr-check/1099599710/lower2' does not support file handles, falling back to xino=off.
[11817.882061] overlayfs: upper fs does not support xattr, falling back to index=off,metacopy=off.
[11817.882064] overlayfs: upper fs does not support xattr, falling back to xino=off.

Any idea how to improve this?

Thanks