I also verified that all hosts know each other by the correct hostname/IP. I can successfully open arbitrary ports with netcat to/from the initiating host and the other cluster hosts. I also ran nmap from the initiating host to the other hosts and the ports show as closed, not filtered.
I have a question about microcloud. From the demo it seems this is designed for having my own machine and provisioning some VMs using LXD and using microcloud with those VMs.
My question is this. Would this work with VMs from cloud providers? IE i can install microcloud on digitalocean droplets or AWS EC2 instances and have them find each other and create a microcloud cluster.
You need working multicast/mdns. So I’m pretty sure that it’s not going to work on AWS.
DigitalOcean may be providing an actual L2 network which then may work for this.
have you tried with
https://www.zerotier.com/
AWS would work if you set up your internal subnets (have not tested)
Contabo private networking works (tested)
Hetzner private network does not work (tested)
I’m trying to run microcloud on Vultr
got this output
Please choose the address MicroCloud will be listening on [default=139.180.135.155]: 10.0.1.3
Scanning for eligible servers...
Press enter to end scanning for servers
Ending scan
Initializing a new cluster
Local MicroCloud is ready
Local MicroCeph is ready
Error: Failed to bootstrap local LXD: Only /16 or /24 underlays are supported at this time
I am using /24 underlay 10.0.1.3 is using a VPC configured with /24 subnet
Then I tried again and got this
Please choose the address MicroCloud will be listening on [default=139.180.135.155]:
Scanning for eligible servers...
Press enter to end scanning for servers
Ending scan
Initializing a new cluster
Error: Failed to bootstrap local MicroCloud: Failed to initialize local remote entry: A remote with name "microcloud-01" already exists
Can you show ip -4 a s on the instance?
Hello, I have tried several times recreating the VM’s and installing microcloud into fresh ubuntu22.04 lxc images. The system is formed by 3 Ubuntu 22.04 VM with 1Gb ram, 9gb OS, 5Gb additional disk. I get the following error:
root@v21:~# sudo -i microcloud init
Please choose the address Microcloud will be listening on [default=192.168.1.45]:
Scanning for eligible servers.
Press enter to end scanning for servers
Found “v22” at “192.168.1.46”
Found “V23” at “192.168.1.47”
Ending scan
Initializing a new cluster
Local MicroCloud is ready
Local LXD is ready
Error: Failed to bootstrap local MicroCeph: Post “http://control.socket/1.0/services/microceph/cluster/control”: context deadline exceeded
root@v21:~#
Some more info:
root@v21:~# snap list
Name Version Rev Tracking Publisher Notes
core20 20221212 1782 latest/stable canonical✓ base
core22 20230110 487 latest/stable canonical✓ base
lxd 5.10-b392610 24331 latest/stable canonical✓ -
microceph 0+git.00fe8d8 122 latest/stable canonical✓ -
microcloud 0+git.d78a41a 74 latest/stable canonical✓ -
snapd 2.58 17954 latest/stable canonical✓ snapd
root@v21:~# snap services
Service Startup Current Notes
lxd.activate enabled inactive -
lxd.daemon enabled active socket-activated
lxd.user-daemon enabled inactive socket-activated
microceph.daemon enabled active -
microceph.mds enabled inactive -
microceph.mgr enabled inactive -
microceph.mon enabled active -
microceph.osd enabled active -
microcloud.daemon enabled active -
VM Configuration (3 identical: v21,v22 & v23):
lxc config show v21
architecture: aarch64
config:
image.architecture: arm64
image.description: ubuntu 22.04 LTS arm64 (release) (20230107)
image.label: release
image.os: ubuntu
image.release: jammy
image.serial: “20230107”
image.type: disk1.img
image.version: “22.04”
volatile.base_image: 42bea06871485d3ef6c91ca3476515aea07e25cd49b94239fa29ccaf79ead53b
volatile.cloud-init.instance-id: 50027179-84c2-4298-b1f2-021084a7228f
volatile.eth0.host_name: mac45dfa2a3
volatile.eth0.hwaddr: 00:16:3e:c6:79:bf
volatile.eth0.last_state.created: “false”
volatile.last_state.power: RUNNING
volatile.uuid: 6cbfedd3-e7de-45c7-806b-561eba84fad9
volatile.vsock_id: “14”
devices:
vd21:
pool: local
source: vd21
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: “”
i have a school lab i manage with proxmox, and it’s nice and all but i mostly used it because it was so easy to setup ceph with it.
I’ve been wanting to move things over to lxd for some time anyways… but id like to ask… is microcloud and microceph, production ready?
I think I’d consider it to be in a beta stage at this point.
We’re working on adding OVN to MicroCloud next which should make it feature complete at that point with our efforts then going more towards making it more flexible (adding/removing machines) and just doing general improvements to it.
I see, I guess then i have another question… is it possible to specify a mgmt network for ceph when using this method to deploy? I have a nice 100Gb switch that ive been using for mgmt, and then 10Gb lan ports for WAN connection. Can i specify which nic’s or even vlans are used for microceph mgmt ?
When running microcloud init, you should be prompted for the IP address that should be used by microcloud, use the IP address from the internet you want to have the traffic of microcloud, microceph and LXD running over, so most likely your mgmt network in this case.
Mgmt does not have WAN access though, so that is ok right? i essentially want ceph to be running on the 100Gb network for syncing and replication and stuff
Also can we do vlans?
Yep, that’s fine, that’s just use for internal communication between the various daemons.
You can do VLANs, just use the IP address on that VLAN.
You guys are doing amazing work, thank you !
Last question, will there be a way to transition over to the new microOVN(when its released) on microcloud if we have setup microcloud now ?(without ovn)
It should be possible for us to do it but it’s unlikely to be our first priority as we add OVN to the mix.
I think you should be able to manually deploy microovn though and then microcloud should deal with it fine.
I’m sorry to ask, but i just want to be sure. where would i get the information to build microovn?
I’m currently working on that project. Once microovn will work, we’ll make it available as a snap and have microcloud initialize it as it does microceph today.
Thank you so much, you are doing amazing work !