Docker: Can't Write Volumes

Incus
1

I’m running Incus 6.5 on Ubuntu Server 22.04, using zfsutils-linux / zfs --version == 2.1.5. I’d like to get docker working on a 22.04 guest (lxc or VM), using this docker-compose file:

--- 
services:
  grafan: 
    image: grafana/grafana-oss
    user: "1000"
    restart: unless-stopped
    volumes: 
      - ~/var/pdk_blob/grafana:/var/lib/grafana 
    ports: 
      - 3000:3000

I test on both lxc container and VM. I get the same failure message on both:

[+] Running 1/0
 ✔ Container basehost-grafan-1  Created                                                                                                                                                                                                    0.0s 
Attaching to grafan-1
grafan-1  | mkdir: can't create directory '/var/lib/grafana/plugins': Permission denied
grafan-1  | GF_PATHS_DATA='/var/lib/grafana' is not writable.
grafan-1  | You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
grafan-1 exited with code 0

The docker process is able to create the root volume directory (~/var/pdk_blob/grafana) , but can’t write files or directories below that.

I think something in incus prevents file writing.

I’m using ZFS. From this post I see that there is a configuration option “zfs.delegate=true”, but it doesn’t seem to work with my ZFS 2.1.5. (I think it requires ZFS 2.2, only available on Ubuntu 24.04).

Note: running a docker ‘hello world’ web service, one that doesn’t use volumes, works great on both lxc and VM.

Questions:

  • should docker volumes ‘just work’ on an Ubuntu VM?
  • if so, what OS versions are required on host and guest?
  • should I use a non-zfs storage pool? (dir?)
  • am I missing something else?

Thanks in Advance!

2

Have you tried making this an absolute path? The ~ may be translated wrongly.

3

Just tried with absolute path and get the same error. Thanks for the suggestion!

4

In that case, the next suspected issue would be that the user that the container is running as is not privileged to write to the directory.

Can you verify the permissions of the directory? What does stat ~/var/pdk_blob/grafana give you?

5

When I run “docker compose up” the first time, it creates ~/var/pdk_blob/grafana which is owned by root:root with permissions rwxr-xr-x. When I chown -R $(whoami):$(whoami) ~/var, I get the same failure message. When I sudo rm -r var, then rerun docker compose up, the ~/var directory structure is re-created with root:root ownership.

Now interestingly: I just tried to sudo chmod var/pdk_data/grafana 777, re-ran docker compose up and it seems to have worked!

But it gives an odd error message.

[+] Running 1/0
 ✔ Container basehost-grafan-1  Created                                                                           0.0s 
Attaching to grafan-1
grafan-1  | Error checking server process execution privilege. error: could not get current OS user to detect process privileges

And when I run cd ~/var/pdk_blob/grafana && ls -al, I see that all the newly created files and directories are owned by ubuntu:root. Wow!

So I seem to have a workaround!!!

But wonder what the heck is going on with the error message and the ubuntu user!! Is there a way to correct this? On an Incus guest, I need docker to behave identically to other platforms.

On other platforms, the whole directory under ~/var is owned by myuser:myuser…

@abiosoft thanks a lot for raising the issue!!

6

From what I can see, the issue is due to the fact that the directory does not exist and is being created by docker.

Docker runs as root user, hence the initial owner of the directory being root. But you specified the container to run as user 1000 who is without the privilege to write to the root owned directory.

The proper fix would be to ensure that the directory exists and is owned by the intended user before starting docker compose.

This is actually how docker behaves and I would expect you to have similar experience outside of Incus if the same scenario is repeated.

7

But of course. I should have spotted that. I changed to user 1001 and things behave as expected. I’ll continue to tweak, but I think I can see my way now to a working solution. @abiosoft thanks for pointing the way. :wink:

1 Like