[solved] Docker on ubuntu: can't prune container/volume

1

Hi there,

situation: lxd 3.12, kernel 5.1.4, distro: archlinux
i follow this instruction: How to run Docker in a LXD container – Mi blog lah! for a working docker on top of ubuntu 18.04.

All working great (version18.04.2 LTS) but the docker daemon can’t remove the btrfs subvolume created from himself.

“docker system prune” and “docker container prune” claim 0 bytes of space

Sometime in “docker build” i have error like this:

Error removing intermediate container 3c59aff4f48e: container 3c59aff4f48ebc9943cbdc36d7b4360911292f9441b5436199bf5405edbe6b24: driver “btrfs” failed to remove root filesystem: Failed to destroy btrfs snapshot /var/lib/docker/btrfs/subvolumes for 823fd1caaf73c70623a9e1446854c81a303a799729b4f007c33fa682d752384d: operation not permitted

docker info:

docker info
Containers: 89
 Running: 2
 Paused: 0
 Stopped: 87
Images: 31
Server Version: 18.09.5
Storage Driver: btrfs
 Build Version: Btrfs v4.15.1
 Library Version: 102
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 
runc version: N/A
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 seccomp
  Profile: default
Kernel Version: 5.1.4-arch1-1-ARCH
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.789GiB
Name: docker-bot
ID: S4OT:PQ3V:ZC2K:XEEW:EWEWB:PZS5:XPGP:UYTE:VVL6:6FWP:JP7P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

sorry for bad english

2

How’s your storage pool configured on the LXD side?

Can you show lxc storage show default?

The error suggests that the btrfs filesystem isn’t mounted with the required option for unprivileged subvolume deletion.

1 Like
3

Hi Stephane,

thanks for you kind reply.

i don’t have default storage, the only one is “btrfs” created by me with “lxd init” command, but the btrfs subvolume was created by me before the initialization

$ lxc storage ls
±------±-------------------------±-------±----------±--------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
±------±-------------------------±-------±----------±--------+
| btrfs | ----------redacted | btrfs | /DATA/lxd | 10 |
±------±-------------------------±-------±----------±--------+

$ lxc storage show btrfs
config:
source: /DATA/lxd
volatile.initial_source: /DATA/lxd
description: ----------redacted
name: btrfs
driver: btrfs
used_by:

  • /1.0/containers/one
  • /1.0/containers/two
  • /1.0/containers/docker
  • /1.0/containers/test
  • /1.0/containers/test2
  • /1.0/containers/test3
  • /1.0/containers/test3/snapshots/test66
  • /1.0/containers/test-bot
  • /1.0/images/3c09483ccd69f33a4819532c103f482f219ae4591cc0d860dfb94193e97a2627
  • /1.0/profiles/default
    status: Created
    locations:
  • none

the only things i changed is the name of my lxc containers

Thanks!

4

Excellent, so that makes sense as btrfs pools that are mounted by LXD will have the right mount options set automatically, but those that rely on a filesystem which you mounted yourself cannot.

In your case, you’re missing the user_subvol_rm_allowed mount option. Add this to your /etc/fstab or whatever mounts your btrfs filesystem in the first place and once that’s applied and visible in /proc/self/mountinfo you should be good to go as far as containers deleting subvolumes.

1 Like
5

Thanks Stephane for pointing out the problem.
Now i have another one: i can’t set user_subvol_rm_allowed to mount option, but this is my distro problem, not lxd one (i asked for help in arch forum: https://bbs.archlinux.org/viewtopic.php?id=246800)

meanwhile i can destroy the docker lxc container and start a new one with overlayfs as docker storage driver, is a good option in your opinion?

6

Nevermind: found solution with help of archlinux forum: i must put " user_subvol_rm_allowed" in root subvolume, not only the lxd one.

Thanks again

7

On a similar issue;
How do I add the mount option user_subvol_rm_allowed for a disk with a block-device as a source, to a container?