LXD nesting containers with Docker

LXD
1

Hi, Ive been using LXD for a while now and am really enjoying it.

I want to run Nginx Proxy Manager (in a production environment), but Nginx Proxy Manager only runs in Docker.

I have setup a container like so:

lxc launch ubuntu:20.04 npm
lxc config set npm security.nesting true
lxc config set npm security.syscalls.intercept.mknod=true
lxc config set npm security.syscalls.intercept.setxattr=true

I have installed docker and docker-compose within the container and Nginx Proxy Manager is up and running.

Nesting containers is new to me and I always run my other containers in unprivileged mode.

I guess I’d like to know is this a safe practice to use in production as I was to have Nginx Proxy Manager public facing?
or should I be locking down the container in other ways?

Thanks.

1 Like
2

Nesting with unprivileged containers is considered AFAIK.

Is this true @stgraber ?

My understanding is that nesting with privileged containers can be a security problem.

3

The config above should be fine, security.nesting=true on an unprivileged LXD container is safe and both the mknod and setxattr syscall interception features are also considered safe to use. They could be abused to cause a denial of service attack on the host (using up all the CPU) but as your container doesn’t have limits in place in the first place, that’s unlikely to be an issue here.

1 Like
4

I’ve read many times here in the forums recommendations about setting security.nesting=true in order to run docker inside nested LXD containers.

lxc config set npm security.syscalls.intercept.mknod=true
lxc config set npm security.syscalls.intercept.setxattr=true

Are these two other options also necessary for running docker in containers? Or only in certain situations?

I’ve read about these settings at Linux Containers - LXD - Has been moved to Canonical but am still not quite sure about them. With @stgraber mentioning that they could be abused, I’d rather only set them if necessary.

The Ubuntu tutorial at https://ubuntu.com/tutorials/how-to-run-docker-inside-lxd-containers#2-create-lxd-container also recommends these settings on page 2. In that tutorial, are these two additional settings used only in relation to enabling the host OS storage pool (detailed higher up on that same tutorial page) for use with docker inside the container?

1 Like
5

They should be set when using the overlay2 VFS backend in Docker which is usually the default storage backend. It may work for a while without them, but eventually you’ll hit a layer which will require those options.

Basically those options are needed to handle layers that remove a file or directory from a parent layer. You don’t need them so long as all your layers add stuff, but the moment one of them needs to delete something, if you’re using overlay2, you’ll need them.

2 Likes
6

Seems then that Frequently asked questions - Canonical LXD documentation should be updated to mention them.

7

Apparently running docker within a LXD container (nested mode) makes the docker daemon run with a driver which is not overlay2 automatically. This ends up with taxing performance inevitably. So my new rule is that if I’m going to use docker, I will use a VM.

8 
root@docker01:~# docker info 2>&1 | grep -i Storage
 Storage Driver: overlay2
root@docker01:~#

I do have it forced to overlay2 in daemon.json though and I can’t recall if that was needed or not, but overlay2 certainly does work fine inside of an Incus container.

9

Obviously you are not using a ZFS storage pool

[ 5819.126450] overlayfs: upper fs does not support RENAME_WHITEOUT.
[ 5819.126485] overlayfs: upper fs missing required features.
[ 5821.478328] overlayfs: upper fs does not support RENAME_WHITEOUT.
[ 5821.478425] overlayfs: upper fs missing required features.
[ 5823.737438] overlayfs: upper fs does not support RENAME_WHITEOUT.
[ 5823.737541] overlayfs: upper fs missing required features.

EDIT: I’m not married to ZFS, I can change it, but I need disk quotas to work. so no dirpool for me AFAIU

10

Right, I’m not because ZFS is specifically incompatible with overlay2, that’s not a container issue, that’s really a ZFS issue. If your VM had ZFS as a filesystem, you’d have the same issue in there.

But that’s also why we added ZFS block mode to our storage volumes a while back.
So you can do:

incus storage volume create default docker zfs.block_mode=true size=50GiB
incus config device add my-container docker disk pool=default source=docker path=/var/lib/docker

And you’ll now have your /var/lib/docker on an ext4 filesystem but still stored within your zpool.

2 Likes