Unable to add device disk cephfs

LXD
,
1

Hi,

I try to add a disk device to a container with this command :

lxc config device add container cephfs-data disk source=cephfs:/data path=/cephfs

but, while I can mount* this CephFS on the LXD host, the command return :

Error: Failed to start device "cephfs-data": Unable to mount "a.b.c.d:3300:6789,a.b.c.d:6789,w.x.y.z:6789,s.t.u.v:6789:/" at "/var/snap/lxd/common/lxd/devices/container/disk.cephfs--data.cephfs" with filesystem "ceph": invalid argument

I can see a difference in the ceph node list : the first IP address is present 2 times, with two different ports.

I tried the same command on another host, which can also mount the cephFS filesystem with mount command, but it fails too.

*The mount command (in /etc/fstab) which is running fine :

a.b.c.d:6789,s.t.u.v:6789,a.b.c.d:6789:/data /mnt/data ceph rw,relatime,name=node,acl 0 2

The ultimate goal is to mount this cephFS inside many containers, to share datas.

I think I can first do the mount on the host, and then add a “classical” disk device to containers, but it seems to be a little bit more “dirty”.

2

Can you show your ceph.conf? There’s something in there which isn’t getting parsed properly by LXD.

3

Here it is :

cat /etc/ceph/ceph.conf

[global]
        fsid = 575711fc-909c-11ec-afcc-13737cf20988
        mon_host = [v2:a.b.c.d:3300/0,v1:a.b.c.d:6789/0] [v2:w.x.y.z:3300/0,v1:w.x.y.z:6789/0] [v2:s.t.u.v:3300/0,v1:s.t.u.v:6789/0]
4

What version of LXD are you using?

5

Feeding your mon_host to our current parser, it’s getting me:

mon=[10.0.0.1:3300 10.0.0.1:6789 10.0.0.2:6789 10.0.0.3:6789] err=<nil>

which looks correct

6

I use LXD 5.0 : snap 5.0.0-b0287c1 22923

7

Ah, I see the problem.

1 Like
8
1 Like
9

Hi,
I’ve ran into the same issue here.

How I configured the cephfs storage

lxc storage create pool1 cephfs source=cephfs

After configuring, storage shows up properly

lxc storage ls
+-------+--------+----------------------------------+-------------+---------+---------+
| NAME  | DRIVER |              SOURCE              | DESCRIPTION | USED BY |  STATE  |
+-------+--------+----------------------------------+-------------+---------+---------+
| local | dir    | /var/lib/lxd/storage-pools/local |             | 2       | CREATED |
+-------+--------+----------------------------------+-------------+---------+---------+
| pool1 | cephfs | cephfs                           |             | 0       | CREATED |
+-------+--------+----------------------------------+-------------+---------+---------+

and is mounted correctly

 df -hT
Filesystem                                     Type      Size  Used Avail Use% Mounted on
udev                                           devtmpfs  205M     0  205M   0% /dev
tmpfs                                          tmpfs      46M  684K   46M   2% /run
/dev/sda1                                      ext4      9.3G  4.3G  4.5G  49% /
tmpfs                                          tmpfs     229M     0  229M   0% /dev/shm
tmpfs                                          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                                          tmpfs     229M   28K  229M   1% /var/lib/ceph/osd/ceph-0
tmpfs                                          tmpfs      46M     0   46M   0% /run/user/1000
tmpfs                                          tmpfs     100K     0  100K   0% /var/lib/lxd/shmounts
tmpfs                                          tmpfs     100K     0  100K   0% /var/lib/lxd/devlxd
10.0.0.87:6789,10.0.0.88:6789,10.0.0.89:6789:/ ceph       24G     0   24G   0% /var/lib/lxd/storage-pools/pool1

Listing the /var/lib/lxd/storage-pools/pool1 shows me the folders I’ve created separately

ls -l /var/lib/lxd/storage-pools/pool1
total 0
drwx--x--x 2 root    root    0 Aug 28 14:54 custom
drwx--x--x 2 root    root    0 Aug 28 14:54 custom-snapshots
drwxr-xr-x 2 1000000 1000000 0 Aug 28 15:22 mnt
drwxr-xr-x 2 1000000 1000000 0 Aug 28 15:22 settings

Then I configure them

lxc -v config device add "tv" "settings" disk source=cephfs:pool1/settings path="/etc/settings/"

But when trying to initialize the container, the same error occurs

 lxc start tv
Error: Failed to start device "settings": Unable to mount "10.0.0.87:6789,10.0.0.88:6789,10.0.0.89:6789:/settings" at "/var/lib/lxd/devices/tv/disk.settings.etc-settings-" with filesystem "ceph": no route to host

Should I add the disk source as /var/lib/lxd/storage-pools/pool1 instead of cephfs? That does look an alternative, but what’s the proper way to do it here?

My ceph conf file I’m using [global]# specify cluster network for monitoringcluster network = %s/24# s - Pastebin.com

Thank you,
Nuno