This is a single LXD server setup with LXD-4.21 and I’m trying to add new storage for the NFS attached volume, but it failed and it was something pointing to incorrect source path. Please see below reproducer.
root@amd01:~# df -t nfs4
Filesystem 1K-blocks Used Available Use% Mounted on
10.128.13.1:/nfs 284203008 52780032 216916992 20% /mnt/nfs
root@amd01:~# mkdir /mnt/nfs/lxd
root@amd01:~# chmod 777 /mnt/nfs/lxd/
root@amd01:~# lxc storage create nfs dir source=/mnt/nfs/lxd
Error: Source path '/var/lib/snapd/hostfs/mnt/nfs/lxd' isn't empty
New direcotry on the local fileystem works fine.
root@amd01:~# mkdir /localdir
root@amd01:~# chmod 777 /localdir
root@amd01:~# lxc storage create localdir dir source=/localdir
Storage pool localdir created
root@amd01:~# lxc storage list
+----------+--------+------------------------------------------------+-------------+---------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY |
+----------+--------+------------------------------------------------+-------------+---------+
| default | dir | /var/snap/lxd/common/lxd/storage-pools/default | | 1 |
+----------+--------+------------------------------------------------+-------------+---------+
| localdir | dir | /localdir | | 0 |
+----------+--------+------------------------------------------------+-------------+---------+
This is a small test binary (x86_64) which you will run the empty path detection with a bunch of debugging. Can you download that, make it executable and run it against /mnt/nfs/lxd/ ?
This isn’t going to work. LXD has no idea that this one dir pool is on a shared filesystem, so it instead assumes that it’s a normal dir pool which differs on each host.
You won’t be able to add it because the first server will initialize the directory structure and then the next server will immediately fail saying the directory isn’t empty.
Even if you could get past that issue somehow, LXD still wouldn’t make any actual use of it, it wouldn’t consider volumes to be available cluster-wide so you wouldn’t be able to attach a volume which was first defined on another system. And as it won’t know it’s a shared filesystem, copying/moving instances will result in LXD using traditional migration techniques as if dealing with standalone filesystems.
We support this kind of thing with cephfs, so someone motivated could write a similar storage backend for LXD using cephfs as a template. I’d expect such a nfs backend to come with the same limitation as cephfs which is that it can only be used for storage volumes, instances and images cannot be stored on it. It would also come with additional restrictions as NFS doesn’t support path quotas or snapshots.
I see. but, why it still doesn’t work even the new “local” dir without shared filesystem, then? Still something missing?
root@amd01:~# rm -rf /localdir
root@amd01:~# mkdir /localdir
root@amd01:~# lxc storage create localdir dir source=/localdir --target amd01
Storage pool localdir pending on member amd01
root@amd01:~# lxc storage list
+----------+--------+-------------+---------+---------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
+----------+--------+-------------+---------+---------+
| local | dir | | 1 | CREATED |
+----------+--------+-------------+---------+---------+
| localdir | dir | | 0 | PENDING |
+----------+--------+-------------+---------+---------+
root@amd01:~# lxc init -s localdir ubuntu:20.04 c0
Creating c0
Error: Failed instance creation: Failed creating instance record: Failed initialising instance: Failed to add device "root": Pool "localdir" is pending
Actually, I have two goals.
First, storing images on the network attached filesystem (e.g. NFS or Lustre or whatever, not local filesystem)
no need sharing images between nodes, but I just want to store images on the network attached Filesystem instead of local disk.
From your answer, this might work if I do create separate storage pool for each node individually?
However, i firstly need to get solution for above problem that pending state doesn’t change.
Second goal is promoting Lustre filesystem as what cephfs is doing in LXD
I was just playing NFS first to confirm if it works. If it works, Lustre should work with same way.
However, I got your point that “dir” with shared storage won’t work between nodes.
now, I am interested in cephfs template you mentioned.
would you share a pointer of cephfs’s template to me please so that I can have a look?
The reason why your storage pool is pending is because you didn’t perform a lxc storage create localdir dir following the server-specific ones (those with --target).
