My cephadm cluster has been yelling at me to replace the aes keys in it’s health checks, so I went about doing that, however it seems that ceph/the kernel/something isn’t new enough for the aes256k keys? My cephfs storage pool is printing the following in the event log:
context-err="Couldn't get fsid for \"camnet\": Failed to run: ceph --format json --cluster camnet --name client.incus fsid: exit status 1 (2026-10-01T19:02:38.376-0400 7f40c31e96c0 -1 auth: error parsing file /etc/ceph/camnet.client.incus.keyring: error setting modifier for [client.incus] type=key val=AgC1eb5qroCNEyAA0oPwCweJsVHfy81q3hdOlCLDhwa+xZvl9O+VUwc6FjQ=: Malformed input [buffer:3]\n2026-10-01T19:02:38.376-0400 7f40c31e96c0 -1 auth: failed to load /etc/ceph/camnet.client.incus.keyring: (5) Input/output error\n2026-10-01T19:02:38.377-0400 7f40c31e96c0 -1 auth: error parsing file /etc/ceph/camnet.client.incus.keyring: error setting modifier for [client.incus] type=key val=AgC1eb5qroCNEyAA0oPwCweJsVHfy81q3hdOlCLDhwa+xZvl9O+VUwc6FjQ=: Malformed input [buffer:3]\n2026-10-01T19:02:38.377-0400 7f40c31e96c0 -1 auth: failed to load /etc/ceph/camnet.client.incus.keyring: (5) Input/output error\n2026-10-01T19:02:38.377-0400 7f40c31e96c0 -1 auth: error parsing file /etc/ceph/camnet.client.incus.keyring: error setting modifier for [client.incus] type=key val=AgC1eb5qroCNEyAA0oPwCweJsVHfy81q3hdOlCLDhwa+xZvl9O+VUwc6FjQ=: Malformed input [buffer:3]\n2026-10-01T19:02:38.377-0400 7f40c31e96c0 -1 auth: failed to load /etc/ceph/camnet.client.incus.keyring: (5) Input/output error\n2026-10-01T19:02:38.377-0400 7f40c31e96c0 -1 monclient: keyring not found\n[errno 5] RADOS I/O error (error connecting to the cluster))" context-pool="cephfs" level="error" Failed mounting storage pool
that AgC… key is the new aes256k type from the cve rotate. the ceph client on incus still cant parse it (Malformed input).
keep client.incus on classic aes til incus ships a newer ceph:
-ceph auth rotate --key-type=aes client.incus | tee /etc/ceph/camnet.client.incus.keyring
-drop that keyring on the incus host under /etc/ceph/
-ceph --name client.incus --cluster camnet fsid
leave mon/osd/mds on aes256k. the health yell about insecure client keys is whatever til then.
What system are you on? Depending on a few factors this could be almost easy, or hard. E.g the kernel in trixie-backports supports the new keys you need to rbd mount, and for the ceph client parts (that incus uses) it is possible to use the ceph-tentacle repos from proxmox. Should work, but YMMV
Happy to help out more for any other specific questions!
They have 20.2.4 for amd64 but not for aarch64 which is why we’re still pulling 20.2.2 which then annoyingly prevents us from having proper support for the new keys…
Yeah, no, they added the support for the new algorithm at the same time as the warning as part of a security fix… I’ve been trying to fully transition a bunch of environments and that’s been a mess, between the kernel (krbd) only supporting it since recent releases and then needing to track down the right Ceph releases for userspace, this has not been smooth…
For IncusOS, I don’t know what’s up with the Proxmox arm64 repository so we may end up making our own packages of modern Ceph including this fix and then switch over to that.
