Current state of virtiofs under virtual machines

[joshenders]

The documentation is a little brief on “path on the host” disk types. I’ve been exploring virtiofs for my Linux virtual machines with limited success this last week.

Type: disk

incus config device add <instance_name> <device_name> disk pool=<pool_name> source=<volume_name> [path=<path_in_instance>]

When I attempt to mount <volume_name> in my virtiofs enabled guest, I get this error message that the tag isn’t found:

mount -t virtiofs test /mnt/disk

58.808901] virtio-fs: tag ‹test> not found mount: /mnt/disk: wrong fs type, bad option, bad super block on test, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.

I see virtiofsd in the process list of the host, so I’m at a loss as to what could be going wrong here from a user perspective.

Haven’t gone spelunking in the code yet but any advice appreciated. Thank you!

[stgraber (Stéphane Graber)]

That’s because you normally have the agent take care of the notification and mounting for you and so don’t need to mess with virtiofs yourself:

stgraber@dakara:~$ incus launch images:ubuntu/22.04 v1 --vm
Launching v1
stgraber@dakara:~$ incus config device add v1 home disk source=/home path=/mnt/home
Device home added to v1
stgraber@dakara:~$ incus exec v1 bash
root@v1:~# df -h /mnt/home/
Filesystem      Size  Used Avail Use% Mounted on
incus_home       90G   21G   65G  24% /mnt/home
root@v1:~# ls -lh /mnt/home/
total 4.0K
drwxr-xr-x 2 root root 4.0K Sep  8 23:35 stgraber
root@v1:~# 

If you do, as can be seen above, the virtiofs share names are incus_NAME so incus_test in your case.

[joshenders]

Ah, makes perfect sense. Thank you Stéphane, you’re always so quick to respond and helpful!

Follow up question: I haven’t yet used an operating system with agent support (I’m using OpenWrt and flatcar/coreOS currently) –– does the agent also take care of ireg on the host and does incus offer dax support to the guest? Maybe I should just rely on these helpful incus abstractions and focus on getting the agent running properly under my pet distros…

Lastly, I see the invocation on the host as:

/opt/incus/bin/virtiofsd --fd=3 -o source=/var/lib/incus/devices/system/disk.test.mnt-disk

The (possibly dated) virtio docs also specify the -o cache=always flag and parameters.

./virtiofsd --socket-path=/tmp/vhostqemu -o source=$TESTDIR -o cache=always

Do you happen to know if this is beneficial ?

Lastly, as a point of interest–– it looks like you’ll be able to pull the tags from /sys soon: https://lwn.net/Articles/961448/

[stgraber (Stéphane Graber)]

I did some research on that stuff now. dax hasn’t been merged into QEMU, so it’s not something that can be used. There’s similarly no versiontable mention within the current QEMU codebase which makes me think that ireg has similarly not made it upstream yet.

On the cache front, we’re actually going to be switching to cache=never as the default of cache=auto actually causes issues with systems accessing a very large number of files.
cache=always has some reports of potential data loss/corruption happening, so I don’t think it’s something we’d feel very confident in offering at this point.

[Hristo_Netov]

Can i somehow touch virtiofsd options ?

I’m facing this issue:

Issue with virtiofs

Hello all, Not sure if this is proper category for this question, but i’m facing weird issue with VM with attached volume (filesystem). So, i’ve application that sync a lot files from internet. Sync happen exactly on virtiofs. App crashed when i was doing initial sync and it wasn’t possible to resume it, so i’ve deleted everything (all files in this directory / FS) and started from scratch. Now during the sync filesystem usage start growing super fast and i found out that actually for some r…

and probably it’s caused by cache ?

Hristo

[stgraber (Stéphane Graber)]

The next daily build of the Zabbly packages will have cache=never passed to the daemon.
If that clears our other tests, I’ll push it to the Zabbly stable packages.

For users of other packages, you could ask your distro maintainer to cherry-pick the change or wait until Incus 0.7.

[cfenell]

Hi all,
I just found this forum and signed up, since we are facing a big issue with a Qemu/KVM VM which we manage with Incus (and previously LXD),
This VM mounts a directory from the host and the mount freezes under load. The only way to restore access is to reboot the VM, which may take some 20 minutes.

From what I have seen in the logs and found online, the issue is likely to be
https://forums.opensuse.org/t/qemu-kvm-virtiofs-causing-major-freeze-problem/166959

I cannot find any ways to pass the mentioned options to virtiofsd except to edit the source code? So we wonder if there is a way around this.

[stgraber (Stéphane Graber)]

We have an issue open to allow a bit more control on the caching modes of virtiofsd but I’m not sure that this will necessarily help you given the topic you’re linking to not really conclusively providing a solution.

You can experiment by renaming virtiofsd to virtiofsd,real and then replacing virtiofsd by an executable shell script which passes additional parameters to virtiofsd.real.

Something like:

#!/bin/sh
exec /opt/incus/bin/virtiofsd.real --my-extra-parameter "$@"

That should make it pretty easy to experiment, though you’ll obviously still need to reboot the VM to change those options.

[cfenell]

Hi, thanks for your quick reply and sorry for my delay!
I have implemented the suggested workaround and can confirm that virtiofsd for my VM now starts with the option I wanted to try.

/opt/incus/bin/virtiofsd.bin --thread-pool-size=32 --fd=3 --cache=never -o source=/var/lib/incus/devices/…