Is it possible to use overlay2 storage driver in docker installed in a container with zvol-ext4 volume?

LXD
1

I tried to achieve the thing in the title.

root@mailcow-lxc:~# cat /proc/mounts
/dev/zvol/lxdpool/containers/mailcow-lxc / ext4 rw,relatime,idmapped,discard,stripe=2 0 0
(...)

It looks like zvol is correctly set up in the container. However, docker runs on vfs storage driver by default and its daemon refuses to start if I try to manually change it to overlay2.

failed to start daemon: error initializing graphdriver: driver not supported: overlay2

Does anyone here have any experience with that? Should it work, and I missed something, or I just overestimated the capabilities of zvols?

2

This works fine on my side. I use a specific profile to configure the container properly and install Docker with Cloud-Init.

root@c1:~# docker info | grep Storage
 Storage Driver: overlay2

root@c1:~# findmnt /
TARGET SOURCE                                       FSTYPE OPTIONS
/      /dev/zvol/lxd/containers/sandbox_c1[/rootfs] ext4   rw,relatime,idmapped,discard,stripe=8

Here is my LXD profile (the security.* options are important) :

config:
  cloud-init.user-data: <script to install Docker, nothing special done here>
  security.nesting: "true"
  security.syscalls.intercept.mknod: "true"
  security.syscalls.intercept.setxattr: "true"
description: Profile to configure instances for running Docker
devices: {}
name: docker
used_by:
- /1.0/instances/c1?project=sandbox

Hope it helps :slight_smile:

3

That’s super weird as I have the same things in my profile. :flushed: I wonder if there is any way to debug this.

Part of the config straight from my profile:

$ lxc profile show mailcow-lxc
config:
  limits.cpu: "1"
  limits.memory: 6GB
  security.nesting: "true"
  security.syscalls.intercept.mknod: "true"
  security.syscalls.intercept.setxattr: "true"
(...)
4

Looks weird, did you setup these options live or put them during container initialization (or through a profile) ?

5

Through the profile. I just recreated the container and ended in the same place. :confused:

root@mailcow-lxc:~# findmnt /.
TARGET SOURCE                                            FSTYPE OPTIONS
/      /dev/zvol/lxdpool/containers/mailcow-lxc[/rootfs] ext4   rw,relatime,idmapped,discard,stripe=2
root@mailcow-lxc:~# docker info | grep Storage
 Storage Driver: vfs
root@mailcow-lxc:~# docker --version
Docker version 24.0.2, build cb74dfc
6

Which distribution do you use as host OS ? On my side, I use Ubuntu 22.04.

On other thing I configure on LXD side is shiftfs, I disable it explicitly, I had many issues in the past with it regarding storage. Now, idmapped mounts (the official implementation of it in the kernel) is working fine with ext4 (even inside a zvol)

$ lxc info | grep shiftfs
    shiftfs: "false"
$ lxc info | grep idmapped
    idmapped_mounts: "true"
    idmapped_mounts_v2: "true"
7

I also use Ubuntu 22.04 (for both: host and guest), and have the same configuration of these things:

$ lxc info | grep shiftfs
    shiftfs: "false"
$ lxc info | grep idmapped
    idmapped_mounts: "true"
    idmapped_mounts_v2: "true"

And also:

$ lxc --version
5.13

I wonder if there is anything else that could make a difference.

8

I’m starting to be bit dry on this one :thinking:
The last thing that could be different is that I use the HWE kernel on my server :

$ uname -a
Linux proton 5.19.0-43-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
9

That’s something that’s different.

$ uname -a
Linux hypervisor 5.15.0-73-generic #80-Ubuntu SMP Mon May 15 15:18:26 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

But that would mean that the answer to my question would be negative on a standard Ubuntu 22.04 installation, which would be pretty important, as I guess that’s the main use case for zvols in LXD/LXC. I wonder if that’s the case.

10

I probably found the root cause, I forgot that idmapped mount for overlayfs is not possible with a kernel older than 5.19.

image
image1256×207 30.8 KB

https://kernelnewbies.org/Linux_5.19

1 Like
11

And that’s why Docker is complaining IMHO

12

Wow, that’s significant info that should be mentioned somewhere in the docs next to the zvol documentation, as it impacts the latest Ubuntu LTS version and the main use case for zvols. I guess I’ll be upgrading the kernel and we’ll see.

Thank you so much!

1 Like
13

Keep us posted when you will upgrade your kernel to share results then :wink:

14

Yep! That helped!

root@mailcow-lxc:~# docker info | grep Storage
Storage Driver: overlay2

Thank you so much, I’m pretty sure this info will be helpful for many people who will stumble on this issue. :smile:

1 Like
15

Great, glad it helps ! :slight_smile: