Change in overlayfs usage?

I used lxc a few years back and had written an app that ran LXC containers from within a docker container (there are reasons for this but not relevant). The docker container was running with privileged and had volume mappings to the host for folders like /sys/fs/cgroup. Everything worked fine.

When inside the docker container the base filesystem was overlay so I when I ran the container I mounted a directory from the host, example “/vol/work” which showed up in the docker container as “/vol/work” and of the type used on the host (ext4, xfs etc). This was required since you can’t stack certain filesystems, some fs recursion check in the kernel if I recall correctly.

Within /vol/work I had a container filesystem “/vol/work/container-fs” which was a normal container FS and I would use the python lxc libraries to make many containers from the base filesystem via a jinja template config with this option:

# Lower being /vol/work/contianer-fs and upper being /vol/work/containers/NAME/local
lxc.rootfs=overlayfs:{{lower}}:{{upper}}  

This approach allowed me to make an arbitrary number of containers running within a docker container but using the host filesystem.

Fast forward to today (I updated my configs): Now when I try to use the same approach I can’t create the containers. I tried with the lxc cli and received:

lxc-create: test-container: tools/lxc_create.c: main: 319 Failed to create container test-container

for config

lxc.uts.name=test-container
lxc.init.cmd=/bin/pause
lxc.net.0.type=empty
lxc.rootfs.path=overlayfs:/vol/work/container-fs:/vol/work/containers/test-container/local
lxc.mount.auto=proc sys
lxc.pty.max=1
lxc.tty.max=1
lxc.autodev=1
lxc.apparmor.profile = unconfined

If I create the container using a normal dir for rootfs the first container is created fine

lxc.rootfs.path = dir:/vol/work/containers/test-container/container-fs

so then I tried using lxc-copy which cave me this config

lxc-copy -n test-container -N test-container2 -B overlayfs -a -s -p /vol/work/containers

lxc.init.cmd=/bin/pause
lxc.net.0.type=empty
lxc.mount.auto=proc sys
lxc.pty.max=1
lxc.tty.max=1
lxc.autodev=1
lxc.apparmor.profile = unconfined
lxc.rootfs.path = overlay:/vol/work/containers/test-container/container-fs:/vol/work/containers/test-container2/overlay/delta
lxc.uts.name = test-container2

but this resulted in the same error of

tools/lxc_create.c: main: 319 Failed to create container

I am using LXC compiled from source which is happening when I create the docker container.

I noticed if -p isn’t used for lxc-copy the path is set to the libdir default, in my case /usr/local/lib/lxc/rootfs/ which will of course not work since that’s in / which is overlay. Is llxc-create still using /usr/local/lib/lxc even if the upper and lower of the overlay are on a different device?

Does anyone know what might have changed or know how I can work around this?

Tested this on my host (not inside docker container) and I noticed the same thing. When I tried to lxc-copy a base container but provide a path via -p the container would not start. The the only diff is where the container is defined

diff /var/lib/test-container2/config /vol/work/containers/test-container2/config 
9c9
< lxc.rootfs.path = overlay:/vol/work/test-container/container-fs:/var/lib/test-container2/overlay/delta
---
> lxc.rootfs.path = overlay:/vol/work/test-container/container-fs:/vol/work/containers/test-container2/overlay/delta

Trying to create using the config lxc-copy created when custom path was provided

lxc-create -t none -n test-container2 -f /vol/work/containers/test-container2/config
lxc-create: test-container2: tools/lxc_create.c: main: 319 Failed to create container test-container2

No issue here. Maybe I was misremembering how the LXC pathing works or I was previously getting lucky with incorrect usage. I was able to do this by properly using -P when issuing ALL lxc commands.