I used lxc a few years back and had written an app that ran LXC containers from within a docker container (there are reasons for this but not relevant). The docker container was running with privileged and had volume mappings to the host for folders like /sys/fs/cgroup. Everything worked fine.
When inside the docker container the base filesystem was overlay so I when I ran the container I mounted a directory from the host, example “/vol/work” which showed up in the docker container as “/vol/work” and of the type used on the host (ext4, xfs etc). This was required since you can’t stack certain filesystems, some fs recursion check in the kernel if I recall correctly.
Within /vol/work I had a container filesystem “/vol/work/container-fs” which was a normal container FS and I would use the python lxc libraries to make many containers from the base filesystem via a jinja template config with this option:
# Lower being /vol/work/contianer-fs and upper being /vol/work/containers/NAME/local
lxc.rootfs=overlayfs:{{lower}}:{{upper}}
This approach allowed me to make an arbitrary number of containers running within a docker container but using the host filesystem.
Fast forward to today (I updated my configs): Now when I try to use the same approach I can’t create the containers. I tried with the lxc cli and received:
lxc-create: test-container: tools/lxc_create.c: main: 319 Failed to create container test-container
for config
lxc.uts.name=test-container
lxc.init.cmd=/bin/pause
lxc.net.0.type=empty
lxc.rootfs.path=overlayfs:/vol/work/container-fs:/vol/work/containers/test-container/local
lxc.mount.auto=proc sys
lxc.pty.max=1
lxc.tty.max=1
lxc.autodev=1
lxc.apparmor.profile = unconfined
If I create the container using a normal dir for rootfs the first container is created fine
lxc.rootfs.path = dir:/vol/work/containers/test-container/container-fs
so then I tried using lxc-copy which cave me this config
lxc-copy -n test-container -N test-container2 -B overlayfs -a -s -p /vol/work/containers
lxc.init.cmd=/bin/pause
lxc.net.0.type=empty
lxc.mount.auto=proc sys
lxc.pty.max=1
lxc.tty.max=1
lxc.autodev=1
lxc.apparmor.profile = unconfined
lxc.rootfs.path = overlay:/vol/work/containers/test-container/container-fs:/vol/work/containers/test-container2/overlay/delta
lxc.uts.name = test-container2
but this resulted in the same error of
tools/lxc_create.c: main: 319 Failed to create container
I am using LXC compiled from source which is happening when I create the docker container.
I noticed if -p isn’t used for lxc-copy the path is set to the libdir default, in my case /usr/local/lib/lxc/rootfs/ which will of course not work since that’s in / which is overlay. Is llxc-create still using /usr/local/lib/lxc even if the upper and lower of the overlay are on a different device?
Does anyone know what might have changed or know how I can work around this?