As someone who has been running several Ceph clusters in IPv6-only environments for a few years now across several major versions of Ceph, this hurts me.
I am currently debugging what exactly I have to provide to Incus to get it to actually mount CephFS, since things have changed significantly with the introduction of volumes and subvolumes lately, and I have, in the middle of all that, managed to get this very same error here:
mount: /mnt: mount(2) system call failed: No route to host.
dmesg(1) may have more information after failed mount system call.
Despite every single daemon being perfectly reachable (heck, my daemons are all on a separate IP address so I can tell which address supposedly isn’t reachable even). Usually the error is in client implementations (e.g. krbd really just stack traces into your dmesg if you try accessing an RBD where the osdmap contains both IPv4 and IPv6 addresses, but single stack of either is fine).
Anyway I get this error when I try to mount the filesystem with the same option that Incus uses internally in its mount syscall, which is a ist of the Mon addresses separated by commas. This mount string is currently obsolete/deprecated as far as I can tell:
mount("[2001:41d0:700:2038::1:0]:3300,[2001:41d0:1004:1a22::1:1]:3300,[2001:41d0:602:2029::1:2]:3300:/", "/var/lib/incus/storage-pools/cephfs", "ceph", 0, "name=benaryorg-incus-cephfs,secret=[secret here],mds_namespace=") = -1 EINVAL (Invalid argument)
I actually do have both source and cephfs.path set on the storage pool, but those don’t show up in there, then again I did change them, and now I’m in a state that does neither allow me to delete the pool nor change it because it is in state “pending” so I can’t say for sure if I could change that path, but I have not had luck yet trying to mount it with that pattern anyway.
The current way to properly mount, as used by mount.ceph from a reef client package is as follows:
mount("benaryorg-incus-cephfs@94dec8c9-a487-42d7-9882-87d2b62549ac.cephfs=/volumes/benaryorg/incus/e7c5cd0c-10fa-42e2-9d48-902544f13d07", "/mnt", "ceph", 0, "name=benaryorg-incus-cephfs,ms_mode=prefer-crc,key=benaryorg-incus-cephfs,mon_addr=[2001:41d0:602:2029::1:2]:3300/[2001:41d0:700:2038::1:0]:3300/[2001:41d0:1004:1a22::1:1]:3300") = 0
Or the less populated mount command I used for this (with most parameters being drawn from /etc/ceph/ceph.conf, which Incus even uses to determine the mon addresses, it could just.… not provide them itself but I digress):
mount.ceph benaryorg-incus-cephfs@.cephfs=/volumes/benaryorg/incus/e7c5cd0c-10fa-42e2-9d48-902544f13d07 /mnt
The mount.ceph man-page has the new synopsis which should be used for mounting CephFS in newer versions.
I suspect that somehow the old syntax hits other (kernel-side) legacy code paths which trigger odd behaviour including a No Route To Host somehow (I assume it’s trying to fallback onto a random IPv4 address, which wouldn’t work in an IPv6-only setup).
@stgraber should I open an issue for the new mount syntax? Or a new discuss thread since my issue may not actually be related and I’m still trying to figure out the exact parameters?