Introducing MicroCloud

1

On the heels of the MicroCeph announcement a few weeks ago, we’re now announcing MicroCloud!

MicroCloud is the easiest way to get a fully highly available LXD cluster up and running.
It’s a snap package which can automatically configure both LXD and Ceph across a set of servers.
It relies on mDNS to automatically detect other servers on the network, making it possible to setup a complete cluster by running a single command on one of the machines.

To try it, get yourself at least 3 systems and at least 3 additional disks for use by Ceph.

Then install microcloud, microceph and LXD with:

snap install lxd microceph microcloud

Once this has been installed on all the servers you’d like to put in your cluster, run:

microcloud init

And then go through the few initialization steps. This will detect the other servers, setup a cluster and finally prompt you to add disks to Ceph.

At the end of this, you’ll have both a Ceph and LXD cluster and LXD itself will have been configured with both networking and storage suitable for use in a cluster.

This is just the beginning of MicroCloud. We’re very excited about what’s coming up next, starting with the addition of OVN to the mix, providing distributed networking alongside the distributed storage provided by Ceph.

https://www.youtube.com/watch?v=ZSZoLnp-Ip0

18 Likes
3

Incredible stuff recently, good job team !

4

https://www.youtube.com/watch?v=ZSZoLnp-Ip0

4 Likes
5

Is there an eta on the addition of the microovn stuff?
Thanks.

6

It’s on our roadmap for this cycle, so by May at worst, but it’s next on my personal list after dealing with stable LTS releases, so hopefully much sooner than May unless we hit some unexpected roadblocks.

3 Likes
7

I am not sure where to post it as it is not very clear to me where exactly problem occur. During creating microcloud cluster after step

Adding 5 disks to MicroCeph

I got an error:

Error: Failed to run: rbd --id admin --cluster ceph --pool remote --image-feature layering --size 0B create lxd_remote: exit status 95 (2022-12-16T14:58:48.862+0100 7f92f27fc700 -1 librbd::image::CreateRequest: 0x555779821480 handle_add_image_to_directory: error adding image to directory: (95) Operation not supported
rbd: create error: (95) Operation not supported)

the same error also show up when creating microceph and lxd cluster separately
running lxd init on first lxd node with lxd init after step:

Would you like a YAML “lxd init” preseed to be printed? (yes/no) [default=no]:

8 
Error: Failed to run: rbd --id admin --cluster ceph --pool remote --image-feature layering --size 0B create lxd_remote: exit status 95 (2022-12-16T14:58:48.862+0100 7f92f27fc700 -1 librbd::image::CreateRequest: 0x555779821480 handle_add_image_to_directory: error adding image to directory: (95) Operation not supported
rbd: create error: (95) Operation not supported)

I got exact same problem after ‘lxd init’

Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
Error: Failed to create storage pool "remote": Failed to run: rbd --id admin --cluster ceph --pool lxd --image-feature layering --size 0B create lxd_lxd: exit status 95 (2022-12-22T14:54:26.185+0000 7fc67cff9700 -1 librbd::image::CreateRequest: 0x55d43e430480 handle_add_image_to_directory: error adding image to directory: (95) Operation not supported
rbd: create error: (95) Operation not supported)
9

microceph.ceph status and microceph.ceph osd status would be useful to get a general idea about the state of your ceph deployment.

10 
root@ggpu03:~# microceph cluster list
+--------+--------------------+-------+------------------------------------------------------------------+--------+
|  NAME  |      ADDRESS       | ROLE  |                           FINGERPRINT                            | STATUS |
+--------+--------------------+-------+------------------------------------------------------------------+--------+
| ggpu01 | 10.128.13.221:7000 | voter | 9a827af3f0e0d6609d01dc590f6269cbd38b33b362295ee78d3f7dba7a8988f2 | ONLINE |
+--------+--------------------+-------+------------------------------------------------------------------+--------+
| ggpu02 | 10.128.13.222:7000 | voter | e58cc929b5c02ccf56d730b82efba48353cc6a51659d340af4aa47ba7933703c | ONLINE |
+--------+--------------------+-------+------------------------------------------------------------------+--------+
| ggpu03 | 10.128.13.225:7000 | voter | 5daf2696a8524b0b7c6440463528dc87a43c90aa2322faad48ac6a48bf1ca811 | ONLINE |
+--------+--------------------+-------+------------------------------------------------------------------+--------+
root@ggpu03:~# microceph.ceph status
  cluster:
    id:     57d315f9-b2e8-4958-91d6-2bef06caccd4
    health: HEALTH_ERR
            Module 'devicehealth' has failed: disk I/O error
 
  services:
    mon: 3 daemons, quorum ggpu01,ggpu02,ggpu03 (age 19m)
    mgr: ggpu01(active, since 23m), standbys: ggpu02, ggpu03
    osd: 4 osds: 4 up (since 18m), 4 in (since 18m)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 1 objects, 0 B
    usage:   34 MiB used, 4.8 TiB / 4.8 TiB avail
    pgs:     1 active+clean
 
root@ggpu03:~# microceph.ceph osd  status
ID  HOST     USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE      
 0  ggpu01  11.7M   894G      0        0       0        0   exists,up  
 1  ggpu02  7760k   447G      0        0       0        0   exists,up  
 2  ggpu02  7760k  1788G      0        0       0        0   exists,up  
 3  ggpu03  7696k  1788G      0        0       0        0   exists,up

this could be related to a known issue Module 'devicehealth' has failed: disk I/O error · Issue #71 · canonical/microceph · GitHub ?
the problem seems to be resolved, but not fix is not ready on snap yet?

11

Nah, the I/O thing error in devicehealth can be ignored, that’s fine.
So ceph cluster health looks good in your case.

I’m a bit confused about you having issues within lxd init as lxd init isn’t needed when using microcloud. Are you using microceph standalone instead?

12

Is this the issue?

 --size 0B
13

Shouldn’t be. That’s our normal empty volume that we use for tracking LXD’s use.

1 Like
14

@stgraber sorry for confusions.
This was not microcloud, but ‘lxd init’ against microceph and got errors at the end below.

root@ggpu01:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=10.128.13.221]: 
Are you joining an existing cluster? (yes/no) [default=no]: 
What member name should be used to identify this server in the cluster? [default=ggpu01]: 
Setup password authentication on the cluster? (yes/no) [default=no]: 
Do you want to configure a new local storage pool? (yes/no) [default=yes]: no
Do you want to configure a new remote storage pool? (yes/no) [default=no]: yes
Name of the storage backend to use (cephobject, ceph, cephfs) [default=ceph]: 
Create a new CEPH pool? (yes/no) [default=yes]: 
Name of the existing CEPH cluster [default=ceph]: 
Name of the OSD storage pool [default=lxd]: 
Number of placement groups [default=32]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: yes
Name of the existing bridge or host interface: br0
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]: 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 
Error: Failed to create storage pool "remote": Failed to run: rbd --id admin --cluster ceph --pool lxd --image-feature layering --size 0B create lxd_lxd: exit status 95 (2022-12-23T09:09:30.736+0000 7fe9be7fc700 -1 librbd::image::CreateRequest: 0x5586e61cd480 handle_add_image_to_directory: error adding image to directory: (95) Operation not supported
rbd: create error: (95) Operation not supported)
15

@stgraber this issue also show up after lxc storage create remote ceph using microceph and lxd separately.

16

Anything ceph related in journalctl -n 300 on any of those systems?

17

same error when setting up microcloud

Error: Failed to run: rbd --id admin --cluster ceph --pool remote --image-feature layering --size 0B create lxd_remote: exit status 95 (2022-12-26T02:21:34.190+0000 7fc12d7fa700 -1 librbd::image::CreateRequest: 0x559be7987480 handle_add_image_to_directory: error adding image to directory: (95) Operation not supported
rbd: create error: (95) Operation not supported)
18

nothing at storage creation time.

19

journalctl -n 300 output:

this is from one of the slaves

21

Hi
Any update on this error?
tested the release today and same results

root@microcloud01:/home/nick# snap install lxd microceph microcloud
microcloud 0+git.d78a41a from Canonical✓ installed
lxd 5.9-76c110d from Canonical✓ installed
microceph 0+git.94a7cc7 from Canonical✓ installed
root@microcloud01:/home/nick# microcloud init
Please choose the address MicroCloud will be listening on [default=192.168.0.41]: 
Scanning for eligible servers...
Press enter to end scanning for servers
 Found "microcloud03" at "192.168.0.40"
 Found "microcloud02" at "192.168.0.39"

Ending scan
Initializing a new cluster
 Local MicroCloud is ready
 Local MicroCeph is ready
 Local LXD is ready
Awaiting cluster formation...
 Peer "microcloud03" has joined the cluster
 Peer "microcloud02" has joined the cluster
Cluster initialization is complete
Would you like to add additional local disks to MicroCeph? (yes/no) [default=yes]: 
Select from the available unpartitioned disks:

Select which disks to wipe:

Adding 3 disks to MicroCeph
Error: Failed to run: rbd --id admin --cluster ceph --pool remote --image-feature layering --size 0B create lxd_remote: exit status 95 (2023-01-04T23:11:59.366+0000 7f152cff9700 -1 librbd::image::CreateRequest: 0x5634aa3e9480 handle_add_image_to_directory: error adding image to directory: (95) Operation not supported
rbd: create error: (95) Operation not supported)
root@microcloud01:/home/nick#