Mounting a volume over existing data

Incus
,
1

Hi!

I have a YAML file for distrobuilder that sets up MariaDB in an image. Now, I need to attach a volume to the container at /var/lib/mysql, but I’ve run into an issue.

I don’t want the volume to overwrite any existing data in the container. For example, Docker has a way to avoid this behavior, as described in Docker’s documentation.

I haven’t found anything like this in the distrobuilder or Incus documentation. Is there a flag or feature that prevents a volume from being mounted over existing data? Or is this something that might be implemented in future versions?

Any guidance would be appreciated!

Thanks!

2

As far as I am aware there isn’t such a feature in Incus, might be worth to create a feature ticket for it. From my understanding of the provided link you can perform the same steps manually.

  • create a new storage volume in incus
  • attach that volume to your container under /tmp/volume
  • copy or move all data from /var/lib/mysql to /tmp/volume
  • reattach the volume to /var/lib/mysql
  • start your container

This is the only way you can recreate the pointed out Docker feature. Think having similar functionality in Incus would be a great addition.

1 Like
3

Hey, thanks a ton for the ideas!

@stgraber — do you think adding a feature like Docker’s volume mounting behavior would make sense for Incus? Manual copying doesn’t really scale with many containers.

Thanks!

4
5

Thanks for the link @stgraber !

I had a look at issue #1822, but it seems to focus on tmpfs support rather than on preserving existing container data when mounting a new volume.
Would it make sense to open a separate feature request specifically for that behavior?

Thanks again for your help!

6

@stgraber I tried a temporary solution using incus file pull and incus file push, hoping to minimize the impact, but it turned out to be much more complicated than expected.

incus create mysql mysql
incus file pull --recursive mysql/var/lib/mysql ./
incus storage volume create default mysql
incus storage volume attach default mysql mysql /var/lib/mysql
incus file push --recursive ./mysql mysql/var/lib
incus start mysql

This was an attempt to preserve /var/lib/mysql data by moving it out, attaching a volume, and restoring it back. However, I ran into a couple of problems:

1. incus file push silently fails on stopped containers
There’s no error, files “appear” to upload (even with incus file edit), but they disappear after container start.

root@mysql:~# ls /var/lib/mysql/

root@mysql:~# ls -ld /var/lib/mysql
drwx--x--x 2 root root 2 May  8 02:05 /var/lib/mysql

=> Empty. Seems incus file push only works properly on running containers, but this isn’t documented or warned about.

2. Permission denied when accessing mounted volumes
Even after setting:

incus storage volume set default data initial.uid 1000
incus storage volume set default data initial.gid 1000
incus storage volume set default data security.shifted=true

Trying to access the mount from inside the container fails:

touch: cannot touch 'grass': Permission denied

Only works if I chown the mount from inside the container after it’s started.

Environment:

  • Incus 6.12
  • ZFS 2.3.1

Can someone clarify the expected behavior or suggest a better approach?
It seems there are two issues I’m facing:

  1. Copying data into a stopped container:
    When using incus file push on a stopped container, files appear to upload but disappear after the container starts. It seems file push only works properly when containers are running.
  2. Permission issues with mounted volumes:
    Even after setting initial.uid and initial.gid, I still get “Permission denied” errors when accessing the mounted volume from inside the container. This works only if I change the ownership (chown) from inside the container after it’s started.

Any suggestions or clarifications would be greatly appreciated.

7

Do you mean “overwrite” or “hide”?

AFAIK, incus does not copy the volume into the container like docker can, it just mounts over it. So the data is not lost, but it is hidden.

What do you want to happen instead? For example, would you want incus to refuse to start the container, if the mount point is not empty?

A similar problem I’ve had in the past (with NFS mounts) is an application starting up and starting to write to /nfs (say) and find that it’s writing to local disk instead of the NFS server. The solution was to change the application to write to /nfs/data, and either it doesn’t automatically create a parent directory, or set permissions on /nfs so that it cannot. Then the application fails if the mount containing the data subdirectory isn’t present.

8

Thanks @candlerb — I mean “hide.”

Yes, Incus just mounts over the path, so the original data is hidden, not deleted. The problem is that there’s no built-in option to automatically copy that hidden data into the newly attached (empty) volume, like Docker does.

That’s exactly the behavior I’m looking for:
→ If the container has data at the mount point, and the volume is empty, then copy the data into the volume on first mount.

I’ve opened a GitHub issue for this feature here:

Another issue I ran into:
incus file push doesn’t work reliably when the container is stopped — files appear to upload without errors, but disappear after the container starts. This makes volume initialization even trickier in automated scenarios.

Thanks again for your input!

9

Are you copying to a part of the filesystem where the volume is mounted? I wonder if you’re actually copying into the container filesystem, and when the container starts, the volume is mounted (which hides the underlying mountpoint). You could easily test this by temporarily removing the volume and seeing if the “disappeared” files reappear.

There is an open issue for being able to copy files in and out of storage volumes.

10

Just confirmed this:

  • When the container is stopped, incus file push writes to the container’s rootfs.
  • When the container is running, incus file push writes inside the mounted volume (if one is attached at that path).

So the behavior is consistent with layering, but it’s very easy to run into data loss/confusion during automation if you don’t realize what’s happening. This might not be a bug, but it definitely deserves clearer documentation or better tooling.

Another related issue I noticed:
When pulling files owned by mysql:mysql from the container to the host using incus file pull, the ownership is lost — files end up as root:root on the host. So UID/GID are not preserved during the transfer.

In general, working with volumes and file transfer in Incus still feels rough around the edges — especially for automation-heavy workflows. I think there’s room for some solid improvements here.

11

For me the pain point is in the opposite direction (push), because the permissions are copied from the outer host. I end up having to add --uid, --gid and --mode flags to every single file push command.

I’d much rather root inside the container be the default - it’s a container, after all - with a flag like cp -p if you want to copy the permissions from the outside.

12

We’re essentially trying to mimic Docker’s volume initialization using incus file pull/push, but the current behavior makes it unreliable:

  1. incus file push to a stopped container writes to rootfs, not the future mount — so data silently disappears on start.
  2. incus file pull --recursive resets ownership to root:root, losing original UID/GID.
  3. file push applies host-side ownership by default, requiring --uid, --gid, and --mode manually for every file.

Together, this makes safe and automated volume workflows very difficult.

A mode like incus file push --preserve, UID/GID retention during file pull, and safeguards for writing to mount points in stopped containers could go a long way in supporting volume workflows without full custom scripting.

@stgraber — what do you think?

13

Try using incus file mount then you can deal with a filesystem tree instead.

14

incus file mount only exposes the mounted volume contents when the container is running.
If the container is stopped, it shows the underlying rootfs, and any files copied go there instead — not into the volume.

This makes safe data initialization into volumes tricky when the container is offline.

15

I explored a potential workaround to initialize a mounted volume with image data by combining incus init --empty, volume attach, and incus rebuild. The idea was

incus init mysql --empty
incus storage volume create default mysql
incus storage volume attach default mysql mysql /var/lib/mysql shift=true
incus rebuild mysql mysql

I expected incus rebuild to populate the mounted volume from the image, similar to how Docker handles empty volumes, but the data wasn’t copied — likely because the mount masks the image content.

Is this the expected behavior?

Thanks!

16

incus rebuild only works on stopped containers, and therefore the volumes aren’t mounted; so yes, that’s what I’d expect.

17

In my case, the container is stopped, but the volume is still attached. I was hoping that rebuild would write the initial image data into the attached volume. So the question is: is there a way to get that behavior—either using rebuild or some other approach?

18

The volume may be “attached” in the config, but if the container is stopped, it won’t be mounted. It’s the same as you found with incus file push on stopped containers.

In any case, incus rebuild is explicitly documented as rebuilding the rootfs of the container.

is there a way to get that behavior

As I said before: there’s currently no facility for copying files in or out of an inactive storage volume. A possible approach I can think of would be:

  • Attach the storage volume at a different mount point, say /mnt
  • Start the container
  • Copy recursively from /var/lib/mysql to /mnt (e.g. with tar)
  • Stop the container
  • Re-attach it at /var/lib/mysql
19

Ah, I see. Thanks for clarifying how rebuild works and the relationship between container state and mount points.

I’ll probably try something like OpenTofu to automate this process for now, until Incus eventually supports this kind of volume initialization natively.

Thanks for your help!