How might I mount a host directory into a container and set raw.idmap in one command?

derekmahar · February 4, 2024, 9:00pm

How might I combine into a single incus config device add command the following two commands, the first which sets raw.idmap on a container instance and the second which mounts a host directory into that container?

# Map group ID 1100 and user ID 995 in the host to 1100 and 995, respectively, in the container.
printf "gid 1100 1100\nuid 995 995" | incus config set faithful-colt raw.idmap -

# Mount host directory /mnt/network/truenas/SFTPGo to path /data in the container.
incus config device add faithful-colt sftpgo disk source=/mnt/network/truenas/SFTPGo path=/data

qkiel · February 5, 2024, 9:21am

Maybe:

(printf "gid 1100 1100\nuid 995 995" | incus config set faithful-colt raw.idmap -) && incus config device add faithful-colt sftpgo disk source=/mnt/network/truenas/SFTPGo path=/data

If you have kernel 6.5+ on your host, you can try skipping raw.idmap and start using shift=true option.

incus config device add faithful-colt sftpgo disk shift=true source=/mnt/network/truenas/SFTPGo path=/data

derekmahar · February 5, 2024, 4:35pm

Thank you for this suggestion. I think my original question wasn’t precise enough, though. What I meant is how might I mount the host directory into the container and set raw.idmap using a single incus config command or some other Incus command?

What does shift=true do? How do I provide a device, container, server, or host with the “required idmapping abilities” to use shift=true?

derek@proxmox:~$ incus config device add faithful-colt sftpgo disk source=/mnt/network/truenas/SFTPGo path=/data shift=true
Error: Failed to start device "sftpgo": Required idmapping abilities not available
derek@proxmox:~$ uname -a
Linux proxmox 6.5.11-8-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-8 (2024-01-30T12:27Z) x86_64 GNU/Linux

qkiel · February 5, 2024, 5:10pm

shift=true tells Incus to use VFS idmap shifting - a flexible uid/gid shifting at the kernel level. This way you don’t have to set up idmap by hand. I don’t know why that doesn’t work on proxmox.

As for a single incus config command, I don’t think it exists

derekmahar · February 5, 2024, 8:34pm

I guess the custom Linux kernel that Proxmox uses must not support VFS idmap shifting. It’s a shame because I would have liked to use Incus and Proxmox side-by-side at least until I migrate from Proxmox to Incus, but it looks like I may have to replace Proxmox with standard Debian 12 and the original 6.5+ Linux kernel or compromise by using Incus configuration option raw.idmap instead of shift=true on directories that I mount from the host into containers.

Thank you. Not the end of the world.

stgraber · February 5, 2024, 8:44pm

Yeah, no way to do it in one shot unless you feel like manually editing both through incus config edit.

derekmahar · February 12, 2024, 6:16pm

I installed Linux Mint Debian Edition (LMDE) 6 on one of my older laptops, replaced version 6.1 of the Linux kernel with 6.5, and repeated my test of mounting a directory from the host into a container using incus config device add both with and without option shift=true. With this configuration, running incus config device add with shift=true was successful.

In my opinion, if you use a lot of containers that mount host directories into containers, option shift=true in command incus config device add is a very compelling reason to migrate from Proxmox to Debian Linux and Incus. In Proxmox, I find it very tedious to calculate and set the host to container ID mappings. Proxmox containers would benefit greatly from an option similar to option shift=true in Incus.

Command “incus config device add” without option “shift=true”

Test script:

#!/usr/bin/env bash

# Trace the script execution.
set -o xtrace

# Display kernel version.
uname -a
# Display Incus containers.
incus list
# Display host user and group names and IDs.
id
# Display the contents of the source host directory that we will mount inside the container.
ls -la /mnt/incus/share
# Display the contents of the mounted path inside the container before mounting.
incus exec careful-seahorse -- ls -al /share
# Display the user and group names and IDs of user 1000 and group 1000 inside the container. 
incus exec careful-seahorse --group 1000 --user 1000 -- id
# Mount the host directory /mnt/incus/share to /share inside the container without option shift=true.
incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share
# Display the contents of the mounted path inside the container.
incus exec careful-seahorse -- ls -al /share
# Display the contents of the mounted path inside the container replacing user and group names with their numeric IDs.
incus exec careful-seahorse -- ls -aln /share
# Attempt to create file /share/test3.txt inside the container as user 1000 and group 1000.
incus exec careful-seahorse --group 1000 --user 1000 -- touch /share/test3.txt
# Display the contents of the mounted path inside the container.
incus exec careful-seahorse -- ls -al /share
# Display the contents of the source host directory that is mounted inside the container.
ls -la /mnt/incus/share
# Remove file created during test.
rm /mnt/incus/share/test3.txt
# Remove the mounted host directory "share" from the container.
incus config device remove careful-seahorse share

Output:

derek@yoga-3-pro:~/Documents$ ./incus_test_mount_host_directory_without_shift_true 
+ uname -a
Linux yoga-3-pro 6.5.0-0.deb12.4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.10-1~bpo12+1 (2023-11-23) x86_64 GNU/Linux
+ incus list
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|       NAME       |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| careful-seahorse | RUNNING | 10.158.47.161 (eth0) | fd42:6f5d:75b3:70fc:216:3eff:fe74:222e (eth0) | CONTAINER | 0         |
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
+ id
uid=1000(derek) gid=1000(derek) groups=1000(derek),4(adm),20(dialout),21(fax),24(cdrom),25(floppy),26(tape),27(sudo),29(audio),30(dip),44(video),46(plugdev),100(users),106(netdev),110(lpadmin),115(bluetooth),123(scanner),992(incus-admin)
+ ls -la /mnt/incus/share
total 16
drwxr-xr-x 2 derek derek 4096 Feb 12 13:04 .
drwxr-xr-x 3 root  root  4096 Feb  9 13:50 ..
-rw-r--r-- 1 derek derek    7 Feb  9 14:06 test1.txt
-rw-rw-r-- 1 derek derek    7 Feb  9 14:05 test2.txt
+ incus exec careful-seahorse -- ls -al /share
ls: cannot access '/share': No such file or directory
+ incus exec careful-seahorse --group 1000 --user 1000 -- id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu)
+ incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share
Device share added to careful-seahorse
+ incus exec careful-seahorse -- ls -al /share
total 12
drwxr-xr-x 2 nobody nogroup 4096 Feb 12 18:04 .
drwxr-xr-x 1 root   root     164 Feb 12 18:12 ..
-rw-r--r-- 1 nobody nogroup    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 nobody nogroup    7 Feb  9 19:05 test2.txt
+ incus exec careful-seahorse -- ls -aln /share
total 12
drwxr-xr-x 2 65534 65534 4096 Feb 12 18:04 .
drwxr-xr-x 1     0     0  164 Feb 12 18:12 ..
-rw-r--r-- 1 65534 65534    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 65534 65534    7 Feb  9 19:05 test2.txt
+ incus exec careful-seahorse --group 1000 --user 1000 -- touch /share/test3.txt
touch: cannot touch '/share/test3.txt': Permission denied
+ incus exec careful-seahorse -- ls -al /share
total 12
drwxr-xr-x 2 nobody nogroup 4096 Feb 12 18:04 .
drwxr-xr-x 1 root   root     164 Feb 12 18:12 ..
-rw-r--r-- 1 nobody nogroup    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 nobody nogroup    7 Feb  9 19:05 test2.txt
+ ls -la /mnt/incus/share
total 16
drwxr-xr-x 2 derek derek 4096 Feb 12 13:04 .
drwxr-xr-x 3 root  root  4096 Feb  9 13:50 ..
-rw-r--r-- 1 derek derek    7 Feb  9 14:06 test1.txt
-rw-rw-r-- 1 derek derek    7 Feb  9 14:05 test2.txt
+ rm /mnt/incus/share/test3.txt
rm: cannot remove '/mnt/incus/share/test3.txt': No such file or directory
+ incus config device remove careful-seahorse share
Device share removed from careful-seahorse

Notice that on the host, user derek (ID 1000) and group derek (ID 1000) own the files in /mnt/incus/share, but inside the container, user nobody (ID 65534) and group nogroup (ID 65534) own the files in /share.

Command “incus config device add” with option “shift=true”

Test script:

#!/usr/bin/env bash

# Trace the script execution.
set -o xtrace

# Display kernel version.
uname -a
# Display Incus containers.
incus list
# Display host user and group names and IDs.
id
# Display the contents of the source host directory that we will mount inside the container.
ls -la /mnt/incus/share
# Display the contents of the mounted path inside the container before mounting.
incus exec careful-seahorse -- ls -al /share
# Display the user and group names and IDs of user 1000 and group 1000 inside the container. 
incus exec careful-seahorse --group 1000 --user 1000 -- id
# Mount the host directory /mnt/incus/share to /share inside the container with option shift=true.
incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share shift=true
# Display the contents of the mounted path inside the container.
incus exec careful-seahorse -- ls -al /share
# Attempt to create file /share/test3.txt inside the container as user 1000 and group 1000.
incus exec careful-seahorse --group 1000 --user 1000 -- touch /share/test3.txt
# Display the contents of the mounted path inside the container.
incus exec careful-seahorse -- ls -al /share
# Display the contents of the mounted path inside the container replacing user and group names with their numeric IDs.
incus exec careful-seahorse -- ls -aln /share
# Display the contents of the source host directory.
ls -la /mnt/incus/share
# Remove file created during test.
rm /mnt/incus/share/test3.txt
# Remove the mounted host directory "share" from the container.
incus config device remove careful-seahorse share

Output:

derek@yoga-3-pro:~/Documents$ ./incus_test_mount_host_directory_with_shift_true 
+ uname -a
Linux yoga-3-pro 6.5.0-0.deb12.4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.10-1~bpo12+1 (2023-11-23) x86_64 GNU/Linux
+ incus list
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
|       NAME       |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| careful-seahorse | RUNNING | 10.158.47.161 (eth0) | fd42:6f5d:75b3:70fc:216:3eff:fe74:222e (eth0) | CONTAINER | 0         |
+------------------+---------+----------------------+-----------------------------------------------+-----------+-----------+
+ id
uid=1000(derek) gid=1000(derek) groups=1000(derek),4(adm),20(dialout),21(fax),24(cdrom),25(floppy),26(tape),27(sudo),29(audio),30(dip),44(video),46(plugdev),100(users),106(netdev),110(lpadmin),115(bluetooth),123(scanner),992(incus-admin)
+ ls -la /mnt/incus/share
total 16
drwxr-xr-x 2 derek derek 4096 Feb 12 12:46 .
drwxr-xr-x 3 root  root  4096 Feb  9 13:50 ..
-rw-r--r-- 1 derek derek    7 Feb  9 14:06 test1.txt
-rw-rw-r-- 1 derek derek    7 Feb  9 14:05 test2.txt
+ incus exec careful-seahorse -- ls -al /share
ls: cannot access '/share': No such file or directory
+ incus exec careful-seahorse --group 1000 --user 1000 -- id
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu)
+ incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share shift=true
Device share added to careful-seahorse
+ incus exec careful-seahorse -- ls -al /share
total 12
drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 17:46 .
drwxr-xr-x 1 root   root    164 Feb 12 18:04 ..
-rw-r--r-- 1 ubuntu ubuntu    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 ubuntu ubuntu    7 Feb  9 19:05 test2.txt
+ incus exec careful-seahorse --group 1000 --user 1000 -- touch /share/test3.txt
+ incus exec careful-seahorse -- ls -al /share
total 12
drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 18:04 .
drwxr-xr-x 1 root   root    164 Feb 12 18:04 ..
-rw-r--r-- 1 ubuntu ubuntu    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 ubuntu ubuntu    7 Feb  9 19:05 test2.txt
-rw-r--r-- 1 ubuntu ubuntu    0 Feb 12 18:04 test3.txt
+ incus exec careful-seahorse -- ls -aln /share
total 12
drwxr-xr-x 2 1000 1000 4096 Feb 12 18:04 .
drwxr-xr-x 1    0    0  164 Feb 12 18:04 ..
-rw-r--r-- 1 1000 1000    7 Feb  9 19:06 test1.txt
-rw-rw-r-- 1 1000 1000    7 Feb  9 19:05 test2.txt
-rw-r--r-- 1 1000 1000    0 Feb 12 18:04 test3.txt
+ ls -la /mnt/incus/share
total 16
drwxr-xr-x 2 derek derek 4096 Feb 12 13:04 .
drwxr-xr-x 3 root  root  4096 Feb  9 13:50 ..
-rw-r--r-- 1 derek derek    7 Feb  9 14:06 test1.txt
-rw-rw-r-- 1 derek derek    7 Feb  9 14:05 test2.txt
-rw-r--r-- 1 derek derek    0 Feb 12 13:04 test3.txt
+ rm /mnt/incus/share/test3.txt
+ incus config device remove careful-seahorse share
Device share removed from careful-seahorse

Notice that on the host, user derek (ID 1000) and group derek (ID 1000) own the files in /mnt/incus/share and inside the container, user ubuntu (ID 1000) and group ubuntu (ID 1000) own the files in /share. Inside and outside the container, the owners of the files have the same user and group IDs.

victoitor · February 12, 2024, 6:57pm

On the one line command part of your question, you can always create a profile, add your configuration to the profile and then you can add your configuration cleanly just by adding that profile to your instance.

Profiles are nice since they’ll contain the configuration you want ready to be applied so that you don’t need to remember it.

derekmahar · February 12, 2024, 11:06pm

This is an interesting idea. Could I add the raw.idmap configuration to one profile and the host directory mounting device configuration to another profile using commands similar to those available in incus config or would I create each profile separately using a text editor? Does Incus have any commands to print existing container configuration as a profile?

victoitor · February 12, 2024, 11:13pm

Create the profile and then replace the incus config <instance name> part of the command with incus profile <profile name> and everything should work. Then add the profile to the instance.

To get the configuration from the instance, use incus config show <instance name> --expanded to show the full instance configuration. You can use incus profile edit <profile name> and copy the appropriate parts there as well.

derekmahar · February 13, 2024, 12:36am

In my particular example, I substituted the incus config commands

incus config set careful-seahorse raw.idmap "both 1000 1000"
incus restart careful-seahorse
incus config device add careful-seahorse share disk source=/mnt/incus/share path=/share

with the incus profile commands

incus profile create idmap
incus profile set idmap raw.idmap "both 1000 1000"
incus profile add careful-seahorse idmap
incus restart careful-seahorse

incus profile create share
incus profile device add share share disk source=/mnt/incus/share path=/share
incus profile add careful-seahorse share

derekmahar · February 13, 2024, 12:39am

How can I add two or more profiles to a container in one command?

derekmahar · February 13, 2024, 12:47am

Use command incus profile assign.

My first attempt at adding two profiles to a container failed because unlike incus profile add, incus profile assign does not add profiles to a container, but instead replaces all profiles of a container with the given profiles.

derek@yoga-3-pro:~/Documents$ incus profile assign careful-seahorse idmap,share
Error: Invalid expanded devices: Failed detecting root disk device: No root device could be found

In order to resolve this error, I included profile default in the list of profiles:

derek@yoga-3-pro:~/Documents$ incus profile assign careful-seahorse default,idmap,share
Profiles default,idmap,share applied to careful-seahorse

victoitor · February 14, 2024, 2:15am

If the idmap is usually only to be used together with the share profile, you can add all configs to the same profile so they’ll come together, as they should.

derekmahar · February 14, 2024, 4:05pm

The following script combines profiles idmap and share into single profile share and adds profile share to container careful-seahorse:

incus profile create share
incus profile set share raw.idmap "both 1000 1000"
incus profile device add share share disk source=/mnt/incus/share path=/share
incus profile add careful-seahorse share
incus restart careful-seahorse

victoitor · February 14, 2024, 6:06pm

Looks nice.

Just a side note, many configuration changes in incus happen online, so no restart is needed. I’m not sure idmap changes requires a restart, but adding disk devices work online for me.

Btw, I used to mount host folders like you do (since I was used to Docker), but now I just create incus volumes which don’t need the idmap lines. Find the configuration to be much simpler.

derekmahar · February 14, 2024, 6:22pm

According to the documentation for idmaps (and my experience), changing raw.idmap requires a container restart in order to take effect, but mounting a directory into a container with shift=true does not require a container restart.

Does Incus allow you to share volumes between containers?

In this particular case, the host directory that I mount into the container is a remote NFS, SMB, or SSHFS share that I mounted onto the host directory. Does Incus allow unprivileged containers to directly mount remote shares?

victoitor · February 14, 2024, 9:40pm

Didn’t know aboiut the idmap restart.

Since it’s a remote mount, I guess what you’re doing might be somewhat the best I can imagine. An alternative could also be to mount it the remote host directly inside the container through the usual means.

derekmahar · February 14, 2024, 9:51pm

According to this article written in August 2022, an unprivileged LXD container couldn’t mount a remote SMB share. Is this still the case?

victoitor · February 14, 2024, 10:02pm

Sorry but I don’t know. It was a guess that it could be done. Maybe someone more knowledgeable can answer that.