Host path in VM does not work

zig · March 24, 2025, 3:01pm

Using incus 6.10.1 from the zabbly deb repo.

When trying to mount a host path into a running VM, first I see the volume is not available, the mount point directory is created but there’s nothing mounted on it.

Then if I restart the VM, I see virtiofsd being briefly spawned at startup on the host but disappears right away. There’s no log mentionning virtiofs in incus monitor.

This time, the volume was mounted in the VM, but any attempt to access the content freezes (for example ls or df). I guess the syscalls stay stuck because there’s no virtiofsd on the host responding.

I see incus is using virtiofsd and qemu provided by zabbly’s repo. I have been using these versions also with libvirt, and it works fine. So the virtiofsd / qemu combination itself seems fine.

stgraber · March 24, 2025, 3:59pm

What distro are you running this on?

zig · March 24, 2025, 4:00pm

The host is debian 12 with stable backports, and the VM is old ubuntu 20.04

stgraber · March 24, 2025, 4:09pm

root@d12:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@d12:~# dpkg -l | grep incus
ii  incus                                1:6.10.1-debian12-202503210253 amd64        Incus - Container and virtualization daemon
ii  incus-base                           1:6.10.1-debian12-202503210253 amd64        Incus - Container and virtualization daemon (container-only)
ii  incus-client                         1:6.10.1-debian12-202503210253 amd64        Incus - Command line client

root@d12:~# incus admin init --auto
root@d12:~# incus launch images:debian/13 d13 --vm
Launching d13
root@d12:~# incus config device add d13 etc disk source=/etc path=/mnt/etc
Device etc added to d13

root@d12:~# ps fauxww | grep virtio
root         312  0.0  0.0      0     0 ?        I    16:02   0:00  \_ [kworker/0:4-virtio_vsock]
root        2139  0.0  0.2   6860  2052 pts/0    S+   16:07   0:00      \_ grep virtio
root        2109  0.0  0.3   6236  3592 ?        Ss   16:07   0:00  \_ /opt/incus/bin/virtiofsd --fd=3 --cache=never --shared-dir=/var/lib/incus/devices/d13/disk.etc.mnt-etc
root        2113  0.0  0.2 141436  1956 ?        Sl   16:07   0:00      \_ /opt/incus/bin/virtiofsd --fd=3 --cache=never --shared-dir=/var/lib/incus/devices/d13/disk.etc.mnt-etc

root@d12:~# incus exec d13 bash
root@d13:~# ls -lh /mnt/etc/
total 592K
drwxr-xr-x 3 root root   4.0K Mar 24 05:33 X11
-rw-r--r-- 1 root root   3.0K May 25  2023 adduser.conf
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 alternatives
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 apparmor
drwxr-xr-x 8 root root   4.0K Mar 24 05:33 apparmor.d
drwxr-xr-x 8 root root   4.0K Mar 24 05:33 apt
-rw-r--r-- 1 root root   2.0K Mar 29  2024 bash.bashrc
-rw-r--r-- 1 root root    367 Mar  6 22:46 bindresvport.blacklist
drwxr-xr-x 2 root root   4.0K Mar  6 14:56 binfmt.d
drwxr-xr-x 3 root root   4.0K Mar 24 05:33 ca-certificates
-rw-r--r-- 1 root root   5.9K Mar 24 05:33 ca-certificates.conf
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 cron.d
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 cron.daily
drwxr-xr-x 4 root root   4.0K Mar 24 05:33 dbus-1
-rw-r--r-- 1 root root   2.9K Jan  8  2023 debconf.conf
-rw-r--r-- 1 root root      6 Mar  7 17:30 debian_version
drwxr-xr-x 3 root root   4.0K Mar 24 16:03 default
-rw-r--r-- 1 root root   1.7K May 25  2023 deluser.conf
drwxr-xr-x 4 root root   4.0K Mar 24 05:33 dhcp
drwxr-xr-x 4 root root   4.0K Mar 24 05:33 dpkg
-rw-r--r-- 1 root root    685 Mar  5  2023 e2scrub.conf
-rw-r--r-- 1 root root      0 Mar 24 05:33 environment
-rw-r--r-- 1 root root   1.9K Oct 17  2022 ethertypes
-rw-r--r-- 1 root root     88 Mar 24 05:34 fstab
-rw-r--r-- 1 root root   2.6K Jul 29  2022 gai.conf
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 groff
-rw-r--r-- 1 root root    646 Mar 24 16:03 group
-rw-r--r-- 1 root root    609 Mar 24 16:03 group-
drwxr-xr-x 2 root root   4.0K Mar 24 05:34 grub.d
-rw-r----- 1 root shadow  543 Mar 24 16:03 gshadow
-rw-r----- 1 root shadow  510 Mar 24 16:03 gshadow-
drwxr-xr-x 3 root root   4.0K Mar 24 05:33 gss
-rw-r--r-- 1 root root      9 Aug  7  2006 host.conf
-rw-r--r-- 1 root root      4 Mar 24 16:02 hostname
-rw-r--r-- 1 root root    123 Mar 24 16:02 hosts
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 init.d
drwxr-xr-x 5 root root   4.0K Mar 24 05:33 initramfs-tools
-rw-r--r-- 1 root root   1.9K Jan  3  2023 inputrc
drwxr-xr-x 4 root root   4.0K Mar 24 05:33 iproute2
-rw-r--r-- 1 root root     27 Mar  7 17:30 issue
-rw-r--r-- 1 root root     20 Mar  7 17:30 issue.net
drwxr-xr-x 5 root root   4.0K Mar 24 05:33 kernel
-rw-r--r-- 1 root root    13K Mar 24 16:03 ld.so.cache
-rw-r--r-- 1 root root     34 Mar  6 22:46 ld.so.conf
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 ld.so.conf.d
drwxr-xr-x 2 root root   4.0K Mar 24 16:02 ldap
-rw-r--r-- 1 root root    191 Feb  9  2023 libaudit.conf
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 libnl-3
-rw-r--r-- 1 root root   3.0K Mar  6 22:46 locale.alias
-rw-r--r-- 1 root root   9.3K Mar 24 05:33 locale.gen
lrwxrwxrwx 1 root root     27 Mar 24 05:33 localtime -> /usr/share/zoneinfo/Etc/UTC
drwxr-xr-x 3 root root   4.0K Mar 24 16:03 logcheck
-rw-r--r-- 1 root root    13K Nov 11  2022 login.defs
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 logrotate.d
-r--r--r-- 1 root root     33 Mar 24 16:02 machine-id
-rw-r--r-- 1 root root    111 Jan 28  2023 magic
-rw-r--r-- 1 root root    111 Jan 28  2023 magic.mime
-rw-r--r-- 1 root root    73K Feb 11  2023 mime.types
-rw-r--r-- 1 root root    782 Mar  5  2023 mke2fs.conf
drwxr-xr-x 2 root root   4.0K Dec 10  2022 modprobe.d
-rw-r--r-- 1 root root    248 Mar 24 05:33 modules
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 modules-load.d
-rw-r--r-- 1 root root    286 Mar  7 17:30 motd
lrwxrwxrwx 1 root root     19 Mar 24 05:33 mtab -> ../proc/self/mounts
drwxr-xr-x 3 root root   4.0K Mar 24 16:03 needrestart
-rw-r--r-- 1 root root    767 Aug 11  2022 netconfig
-rw-r--r-- 1 root root     60 Mar 24 05:33 networks
-rwxr-xr-x 1 root root    243 Sep 16  2023 nftables.conf
-rw-r--r-- 1 root root    526 Mar 24 05:33 nsswitch.conf
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 opt
lrwxrwxrwx 1 root root     21 Mar  7 17:30 os-release -> ../usr/lib/os-release
-rw-r--r-- 1 root root    552 Sep 21  2023 pam.conf
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 pam.d
-rw-r--r-- 1 root root   1.2K Mar 24 16:03 passwd
-rw-r--r-- 1 root root   1.2K Mar 24 16:03 passwd-
drwxr-xr-x 3 root root   4.0K Mar 24 05:33 polkit-1
-rw-r--r-- 1 root root    769 Apr 10  2021 profile
drwxr-xr-x 2 root root   4.0K Mar  7 17:30 profile.d
-rw-r--r-- 1 root root   3.1K Oct 17  2022 protocols
drwxr-xr-x 2 root root   4.0K Mar 24 05:34 python3
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 python3.11
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 rc0.d
drwxr-xr-x 2 root root   4.0K Sep 18  2022 rc1.d
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 rc2.d
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 rc3.d
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 rc4.d
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 rc5.d
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 rc6.d
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 rcS.d
lrwxrwxrwx 1 root root     37 Mar 24 05:34 resolv.conf -> /run/systemd/resolve/stub-resolv.conf
lrwxrwxrwx 1 root root     13 Jan 20  2024 rmt -> /usr/sbin/rmt
-rw-r--r-- 1 root root    911 Oct 17  2022 rpc
drwxr-xr-x 4 root root   4.0K Mar 24 05:33 security
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 selinux
-rw-r--r-- 1 root root    13K Mar 27  2021 services
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 sgml
-rw-r----- 1 root shadow  627 Mar 24 16:03 shadow
-rw-r----- 1 root shadow  606 Mar 24 16:03 shadow-
-rw-r--r-- 1 root root    128 Mar 24 05:33 shells
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 skel
drwxr-xr-x 3 root root   4.0K Mar 24 05:33 ssh
drwxr-xr-x 4 root root   4.0K Mar 24 05:33 ssl
-rw-r--r-- 1 root root     24 Mar 24 16:03 subgid
-rw-r--r-- 1 root root      0 Mar 24 05:33 subgid-
-rw-r--r-- 1 root root     24 Mar 24 16:03 subuid
-rw-r--r-- 1 root root      0 Mar 24 05:33 subuid-
-rw-r--r-- 1 root root   4.3K Jun 27  2023 sudo.conf
-rw-r--r-- 1 root root   9.6K Jun 27  2023 sudo_logsrvd.conf
-r--r----- 1 root root   1.7K Jun 27  2023 sudoers
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 sudoers.d
-rw-r--r-- 1 root root   2.3K Dec 19  2022 sysctl.conf
drwxr-xr-x 2 root root   4.0K Mar 24 16:03 sysctl.d
drwxr-xr-x 6 root root   4.0K Mar 24 05:34 systemd
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 terminfo
-rw-r--r-- 1 root root      8 Mar 24 05:33 timezone
drwxr-xr-x 2 root root   4.0K Mar  6 14:56 tmpfiles.d
-rw-r--r-- 1 root root   1.3K Dec 20 07:39 ucf.conf
drwxr-xr-x 4 root root   4.0K Mar 24 05:33 udev
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 update-motd.d
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 vim
-rw-r--r-- 1 root root    681 Jan 17  2023 xattr.conf
drwxr-xr-x 4 root root   4.0K Mar 24 05:33 xdg
drwxr-xr-x 2 root root   4.0K Mar 24 05:33 xml
root@d13:~#

zig · March 24, 2025, 4:13pm

So I get it’s working on your side. I am on a ARM host, maybe it explains the difference.

zig · March 24, 2025, 4:20pm

I also tried with debian 13 VM, and got the same issue.

stgraber · March 24, 2025, 4:28pm

Ah yeah, seems to be arm64 related:


root@delmak:~# cat /var/log/incus/stgraber-dev_d13/disk.etc.log 
[2025-03-24T16:19:39Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2025-03-24T16:19:39Z INFO  virtiofsd] Client connected, servicing requests
[2025-03-24T16:19:50Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidParam)

virtiofsd exits as soon as the VM attempts to perform the mount.
So that’s why it dies instantly during hotplug and takes longer on start.

The only host side flag we’re passing is the cache. I tried all 3 values we support and it fails in all 3.

QEMU logs this:


qemu-system-aarch64: Failed to write msg. Wrote -1 instead of 52.
qemu-system-aarch64: vhost_set_vring_addr failed: Invalid argument (22)
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-aarch64: Error starting vhost: 22
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost_set_vring_call failed 22
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost_set_vring_call failed 22
qemu-system-aarch64: Unexpected end-of-file before all data were read

Looking online, I only came up with a similar report where IOMMU was involved.
We don’t enable IOMMU by default, so that’s not likely to be the issue.

It may be worth reporting this to the virtiofsd folks to see if there’s some other debugging steps to better understand what’s going on.

Our VM definitions on x86 and aarch64 are as similar as we can make them, so I’m not sure why we’re not seeing this on x86.

zig · March 24, 2025, 4:36pm

Ok I see the same in qemu.log, I didn’t think about checking in there.

I know it can work because when launched through libvirt, it is working. I’ll try to compare the qemu parameters between incus and libvirt.

stgraber · March 24, 2025, 4:58pm

virtiofsd also changed a lot, especially for Debian 12.

There was a C version which I think is still what’s in Debian 12 proper, then this got rewritten completely in Rust and that’s what’s now included in the Zabbly package and what Debian 13 will be shipping with.

zig · March 25, 2025, 1:51am

Yes I saw that major change in virtiofsd, but since it works when used by libvirt (when I point it to the incus binaries for qemu and virtiofsd), I assume there’s no major issue there.
I tried copying over the version in C from debian into /opt/incus/bin, but it didn’t help, it actually got worst, now incus itself get stuck when starting the VM.

stgraber · March 25, 2025, 1:59am

Yeah, the C version doesn’t support the --cache stuff we pass it I believe.