Hmm,do you get the same running that command directly? I wonder if it’s the layering that’s causing issues?
Hey! I worked with @lachoisawesome on the above described issue.
The issue was actually caused due to the fact that the pool specified via source= was erasure coded, but as per Ceph docs here in order to use them with RBD, you need to pass the erasure coded pool via -- data-pool and a replicated pool as pool name. Example from the docs:
rbd create --size 1G --data-pool ec_pool replicated_pool/image_name
Based on that, we found the option for --data-pool in lxd (ceph.osd.data_pool_name) here.
Finally, we did the following and it all worked:
# remove the ERROR'd storage
lxc storage delete default
# Create the new one
# repl-pool - the replication pool in ceph
# lxd-pool - the erasure coded pool
lxc storage create default ceph source=repl-pool --target=node01
lxc storage create default ceph source=repl-pool --target=node01
lxc storage create default ceph source=repl-pool --target=node02
lxc storage create default ceph source=repl-pool --target=node03
lxc storage create default ceph ceph.osd.data_pool_name=lxd-pool
Maybe this erasure coded case could be described in the lxd docs ?
Either way, thanks for the assistance with the issue!
Yeah, we don’t really have any experience with erasure coded pools on our side.
Would you be able to send a suggested update to https://github.com/lxc/lxd/blob/master/doc/storage.md ?
Well, I don’t have that much experience with erasure coded pools, but I can update the doc page to mention this case.
That’d be great!
Pushed the updates to doc/storage.md
Also I did a test on creating an erasure coded pool via lxd (result below). I believe if some logic similar to the one for the ceph.osd.pool_name found here is added for ceph.osd.data_pool, lxd would be able to handle erasure coded pool creation.
Results from my test:
root@node01:~# lxc storage create ceph-pool ceph --target=node01
Storage pool alex2-repl-pool pending on member node01
root@node01:~# lxc storage create ceph-pool ceph --target=node02
Storage pool alex2-repl-pool pending on member node02
root@node01:~# lxc storage create ceph-pool ceph --target=node03
Storage pool alex2-repl-pool pending on member node03
root@node01:~# lxc storage create ceph-pool ceph ceph.osd.data_pool_name=ec-pool
Error: Failed to run: rbd --id admin --cluster ceph --pool ceph-pool \
--image-feature layering --data-pool ec-pool --size 0B create lxd_ceph-pool: \
2021-12-01T05:28:03.535-0800 7fe09e225380 -1 librbd::image::CreateRequest: \
0x5610e2113110 validate_data_pool: data pool ec-pool does not exist
rbd: create error: (2) No such file or directory
Thanks again for the assistance.