How might I mount a host directory into a container and set raw.idmap in one command?

I guess the custom Linux kernel that Proxmox uses must not support VFS idmap shifting. It’s a shame because I would have liked to use Incus and Proxmox side-by-side at least until I migrate from Proxmox to Incus, but it looks like I may have to replace Proxmox with standard Debian 12 and the original 6.5+ Linux kernel or compromise by using Incus configuration option raw.idmap instead of shift=true on directories that I mount from the host into containers.

Thank you. Not the end of the world.

Yeah, no way to do it in one shot unless you feel like manually editing both through incus config edit.

1 Like

I installed Linux Mint Debian Edition (LMDE) 6 on one of my older laptops, replaced version 6.1 of the Linux kernel with 6.5, and repeated my test of mounting a directory from the host into a container using incus config device add both with and without option shift=true. With this configuration, running incus config device add with shift=true was successful.

In my opinion, if you use a lot of containers that mount host directories into containers, option shift=true in command incus config device add is a very compelling reason to migrate from Proxmox to Debian Linux and Incus. In Proxmox, I find it very tedious to calculate and set the host to container ID mappings. Proxmox containers would benefit greatly from an option similar to option shift=true in Incus.

Command “incus config device add” without option “shift=true”

Test script:

#!/usr/bin/env bash

# Trace the script execution.
set -o xtrace

# Display kernel version.
uname -a
# Display Incus containers.
incus list
# Display host user and group names and IDs.
id
# Display the contents of the source host directory that we will mount inside the container.
ls -la /mnt/incus/share
# Display the contents of the mounted path inside the container before mounting.
incus exec careful-seahorse -- ls -al /share
# Display the user and group names and IDs of user 1000 and group 1000 inside the container. 
incus exec careful-seahorse --group 1000 --user 1000 -- id
# Mount the host directory /mnt/incus/share to /share inside the container without option shift=true.
incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share
# Display the contents of the mounted path inside the container.
incus exec careful-seahorse -- ls -al /share
# Display the contents of the mounted path inside the container replacing user and group names with their numeric IDs.
incus exec careful-seahorse -- ls -aln /share
# Attempt to create file /share/test3.txt inside the container as user 1000 and group 1000.
incus exec careful-seahorse --group 1000 --user 1000 -- touch /share/test3.txt
# Display the contents of the mounted path inside the container.
incus exec careful-seahorse -- ls -al /share
# Display the contents of the source host directory that is mounted inside the container.
ls -la /mnt/incus/share
# Remove file created during test.
rm /mnt/incus/share/test3.txt
# Remove the mounted host directory "share" from the container.
incus config device remove careful-seahorse share

Output:

derek@yoga-3-pro:~/Documents$ ./incus_test_mount_host_directory_without_shift_true 
+ uname -a
Linux yoga-3-pro 6.5.0-0.deb12.4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.10-1~bpo12+1 (2023-11-23) x86_64 GNU/Linux
+ incus list
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|       NAME       |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| careful-seahorse | RUNNING | 10.158.47.161 (eth0) | fd42:6f5d:75b3:70fc:216:3eff:fe74:222e (eth0) | CONTAINER | 0         |
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
+ id
uid=1000(derek) gid=1000(derek) groups=1000(derek),4(adm),20(dialout),21(fax),24(cdrom),25(floppy),26(tape),27(sudo),29(audio),30(dip),44(video),46(plugdev),100(users),106(netdev),110(lpadmin),115(bluetooth),123(scanner),992(incus-admin)
+ ls -la /mnt/incus/share
total 16
drwxr-xr-x 2 derek derek 4096 Feb 12 13:04 .
drwxr-xr-x 3 root  root  4096 Feb  9 13:50 ..
-rw-r--r-- 1 derek derek    7 Feb  9 14:06 test1.txt
-rw-rw-r-- 1 derek derek    7 Feb  9 14:05 test2.txt
+ incus exec careful-seahorse -- ls -al /share
ls: cannot access '/share': No such file or directory
+ incus exec careful-seahorse --group 1000 --user 1000 -- id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu)
+ incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share
Device share added to careful-seahorse
+ incus exec careful-seahorse -- ls -al /share
total 12
drwxr-xr-x 2 nobody nogroup 4096 Feb 12 18:04 .
drwxr-xr-x 1 root   root     164 Feb 12 18:12 ..
-rw-r--r-- 1 nobody nogroup    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 nobody nogroup    7 Feb  9 19:05 test2.txt
+ incus exec careful-seahorse -- ls -aln /share
total 12
drwxr-xr-x 2 65534 65534 4096 Feb 12 18:04 .
drwxr-xr-x 1     0     0  164 Feb 12 18:12 ..
-rw-r--r-- 1 65534 65534    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 65534 65534    7 Feb  9 19:05 test2.txt
+ incus exec careful-seahorse --group 1000 --user 1000 -- touch /share/test3.txt
touch: cannot touch '/share/test3.txt': Permission denied
+ incus exec careful-seahorse -- ls -al /share
total 12
drwxr-xr-x 2 nobody nogroup 4096 Feb 12 18:04 .
drwxr-xr-x 1 root   root     164 Feb 12 18:12 ..
-rw-r--r-- 1 nobody nogroup    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 nobody nogroup    7 Feb  9 19:05 test2.txt
+ ls -la /mnt/incus/share
total 16
drwxr-xr-x 2 derek derek 4096 Feb 12 13:04 .
drwxr-xr-x 3 root  root  4096 Feb  9 13:50 ..
-rw-r--r-- 1 derek derek    7 Feb  9 14:06 test1.txt
-rw-rw-r-- 1 derek derek    7 Feb  9 14:05 test2.txt
+ rm /mnt/incus/share/test3.txt
rm: cannot remove '/mnt/incus/share/test3.txt': No such file or directory
+ incus config device remove careful-seahorse share
Device share removed from careful-seahorse

Notice that on the host, user derek (ID 1000) and group derek (ID 1000) own the files in /mnt/incus/share, but inside the container, user nobody (ID 65534) and group nogroup (ID 65534) own the files in /share.

Command “incus config device add” with option “shift=true”

Test script:

#!/usr/bin/env bash

# Trace the script execution.
set -o xtrace

# Display kernel version.
uname -a
# Display Incus containers.
incus list
# Display host user and group names and IDs.
id
# Display the contents of the source host directory that we will mount inside the container.
ls -la /mnt/incus/share
# Display the contents of the mounted path inside the container before mounting.
incus exec careful-seahorse -- ls -al /share
# Display the user and group names and IDs of user 1000 and group 1000 inside the container. 
incus exec careful-seahorse --group 1000 --user 1000 -- id
# Mount the host directory /mnt/incus/share to /share inside the container with option shift=true.
incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share shift=true
# Display the contents of the mounted path inside the container.
incus exec careful-seahorse -- ls -al /share
# Attempt to create file /share/test3.txt inside the container as user 1000 and group 1000.
incus exec careful-seahorse --group 1000 --user 1000 -- touch /share/test3.txt
# Display the contents of the mounted path inside the container.
incus exec careful-seahorse -- ls -al /share
# Display the contents of the mounted path inside the container replacing user and group names with their numeric IDs.
incus exec careful-seahorse -- ls -aln /share
# Display the contents of the source host directory.
ls -la /mnt/incus/share
# Remove file created during test.
rm /mnt/incus/share/test3.txt
# Remove the mounted host directory "share" from the container.
incus config device remove careful-seahorse share

Output:

derek@yoga-3-pro:~/Documents$ ./incus_test_mount_host_directory_with_shift_true 
+ uname -a
Linux yoga-3-pro 6.5.0-0.deb12.4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.10-1~bpo12+1 (2023-11-23) x86_64 GNU/Linux
+ incus list
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|       NAME       |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| careful-seahorse | RUNNING | 10.158.47.161 (eth0) | fd42:6f5d:75b3:70fc:216:3eff:fe74:222e (eth0) | CONTAINER | 0         |
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
+ id
uid=1000(derek) gid=1000(derek) groups=1000(derek),4(adm),20(dialout),21(fax),24(cdrom),25(floppy),26(tape),27(sudo),29(audio),30(dip),44(video),46(plugdev),100(users),106(netdev),110(lpadmin),115(bluetooth),123(scanner),992(incus-admin)
+ ls -la /mnt/incus/share
total 16
drwxr-xr-x 2 derek derek 4096 Feb 12 12:46 .
drwxr-xr-x 3 root  root  4096 Feb  9 13:50 ..
-rw-r--r-- 1 derek derek    7 Feb  9 14:06 test1.txt
-rw-rw-r-- 1 derek derek    7 Feb  9 14:05 test2.txt
+ incus exec careful-seahorse -- ls -al /share
ls: cannot access '/share': No such file or directory
+ incus exec careful-seahorse --group 1000 --user 1000 -- id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu)
+ incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share shift=true
Device share added to careful-seahorse
+ incus exec careful-seahorse -- ls -al /share
total 12
drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 17:46 .
drwxr-xr-x 1 root   root    164 Feb 12 18:04 ..
-rw-r--r-- 1 ubuntu ubuntu    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 ubuntu ubuntu    7 Feb  9 19:05 test2.txt
+ incus exec careful-seahorse --group 1000 --user 1000 -- touch /share/test3.txt
+ incus exec careful-seahorse -- ls -al /share
total 12
drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 18:04 .
drwxr-xr-x 1 root   root    164 Feb 12 18:04 ..
-rw-r--r-- 1 ubuntu ubuntu    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 ubuntu ubuntu    7 Feb  9 19:05 test2.txt
-rw-r--r-- 1 ubuntu ubuntu    0 Feb 12 18:04 test3.txt
+ incus exec careful-seahorse -- ls -aln /share
total 12
drwxr-xr-x 2 1000 1000 4096 Feb 12 18:04 .
drwxr-xr-x 1    0    0  164 Feb 12 18:04 ..
-rw-r--r-- 1 1000 1000    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 1000 1000    7 Feb  9 19:05 test2.txt
-rw-r--r-- 1 1000 1000    0 Feb 12 18:04 test3.txt
+ ls -la /mnt/incus/share
total 16
drwxr-xr-x 2 derek derek 4096 Feb 12 13:04 .
drwxr-xr-x 3 root  root  4096 Feb  9 13:50 ..
-rw-r--r-- 1 derek derek    7 Feb  9 14:06 test1.txt
-rw-rw-r-- 1 derek derek    7 Feb  9 14:05 test2.txt
-rw-r--r-- 1 derek derek    0 Feb 12 13:04 test3.txt
+ rm /mnt/incus/share/test3.txt
+ incus config device remove careful-seahorse share
Device share removed from careful-seahorse

Notice that on the host, user derek (ID 1000) and group derek (ID 1000) own the files in /mnt/incus/share and inside the container, user ubuntu (ID 1000) and group ubuntu (ID 1000) own the files in /share. Inside and outside the container, the owners of the files have the same user and group IDs.

On the one line command part of your question, you can always create a profile, add your configuration to the profile and then you can add your configuration cleanly just by adding that profile to your instance.

Profiles are nice since they’ll contain the configuration you want ready to be applied so that you don’t need to remember it.

This is an interesting idea. Could I add the raw.idmap configuration to one profile and the host directory mounting device configuration to another profile using commands similar to those available in incus config or would I create each profile separately using a text editor? Does Incus have any commands to print existing container configuration as a profile?

Create the profile and then replace the incus config <instance name> part of the command with incus profile <profile name> and everything should work. Then add the profile to the instance.

To get the configuration from the instance, use incus config show <instance name> --expanded to show the full instance configuration. You can use incus profile edit <profile name> and copy the appropriate parts there as well.

1 Like

In my particular example, I substituted the incus config commands

incus config set careful-seahorse raw.idmap "both 1000 1000"
incus restart careful-seahorse
incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share

with the incus profile commands

incus profile create idmap
incus profile set idmap raw.idmap "both 1000 1000"
incus profile add careful-seahorse idmap
incus restart careful-seahorse

incus profile create share
incus profile device add share share disk source=/mnt/incus/share path=/share
incus profile add careful-seahorse share

How can I add two or more profiles to a container in one command?

Use command incus profile assign.

My first attempt at adding two profiles to a container failed because unlike incus profile add, incus profile assign does not add profiles to a container, but instead replaces all profiles of a container with the given profiles.

derek@yoga-3-pro:~/Documents$ incus profile assign careful-seahorse idmap,share
Error: Invalid expanded devices: Failed detecting root disk device: No root device could be found

In order to resolve this error, I included profile default in the list of profiles:

derek@yoga-3-pro:~/Documents$ incus profile assign careful-seahorse default,idmap,share
Profiles default,idmap,share applied to careful-seahorse

If the idmap is usually only to be used together with the share profile, you can add all configs to the same profile so they’ll come together, as they should.

1 Like

The following script combines profiles idmap and share into single profile share and adds profile share to container careful-seahorse:

incus profile create share
incus profile set share raw.idmap "both 1000 1000"
incus profile device add share share disk source=/mnt/incus/share path=/share
incus profile add careful-seahorse share
incus restart careful-seahorse
2 Likes

Looks nice.

Just a side note, many configuration changes in incus happen online, so no restart is needed. I’m not sure idmap changes requires a restart, but adding disk devices work online for me.

Btw, I used to mount host folders like you do (since I was used to Docker), but now I just create incus volumes which don’t need the idmap lines. Find the configuration to be much simpler.

According to the documentation for idmaps (and my experience), changing raw.idmap requires a container restart in order to take effect, but mounting a directory into a container with shift=true does not require a container restart.

Does Incus allow you to share volumes between containers?

In this particular case, the host directory that I mount into the container is a remote NFS, SMB, or SSHFS share that I mounted onto the host directory. Does Incus allow unprivileged containers to directly mount remote shares?

Didn’t know aboiut the idmap restart.

Since it’s a remote mount, I guess what you’re doing might be somewhat the best I can imagine. An alternative could also be to mount it the remote host directly inside the container through the usual means.

According to this article written in August 2022, an unprivileged LXD container couldn’t mount a remote SMB share. Is this still the case?

Sorry but I don’t know. It was a guess that it could be done. Maybe someone more knowledgeable can answer that.

NFS, CIFS/SMB, or SSHFS don’t appear to be available disk device types. :slightly_frowning_face:

Today I discovered that Incus idmap shifting does work on Proxmox running Linux kernel 6.5.13-3-pve, but fails when I attempt to mount into a container a host directory to which I’ve already mounted an SMB/CIFS share. I found, however, that this works on Linux Mint Debian Edition (LMDE) 6 running Linux kernel 6.5.

Any idea why mounting an SMB/CIFS share to a host directory might prevent idmap shifting for that directory? How might I instruct Incus to report a more detailed error message than “Error: Failed to start device “sftpgo”: Required idmapping abilities not available”?

This might help:

Thank you for the reference! I’ll consider smbnetfs.

Note that my test using idmap shifting to mount a host directory into a container on LMDE 6 worked only because I had mounted a normal directory, not a mounted CIFS share! When I tried the same test using a mounted CIFS share, it failed.