Can't attach storage volume on some hosts

Hi all, I’m running an LXD cluster with 4 hosts on Ubuntu 18.04 LTS. I’m using storage pools backed by Ceph.

Launching containers on any host in the cluster works fine, as does moving containers to different hosts. The problem is that running a container with a custom storage volume only works on 1 specific host.

Example with a container called “testing”:
$ lxc storage volume attach user-data testing-data testing vol-data /mnt
Error: Add disk devices: Failed to setup device: Failed to initialize storage volume “testing-data” of type “custom” on storage pool “user-data”: no such file or directory

After this, I can see that the correct RBD device is mapped on that host, and I can manually mount that device on the host with no problems.
I’ve attached the relevant part of syslog from that host at the bottom of this post. Note that the apparmor denied lines appear exactly the same on the working host, so I don’t think that’s the cause.

It’s also possible to attach the volume while the container isn’t running, but then it can only be started on the one host.

Back when I set up the cluster, I vaguely remember encountering this same issue and making some config change on that one host. Unfortunately, I can’t find any information on what the difference might be between the working host and the others. Any suggestions would be much appreciated!

Which version of LXD is this?

Can you show output of lxc storage ls and the output of lxc storage show <ceph pool> please.