The command for growing the cluster is microcloud add from the existing cluster. Using microcloud init also sets up the services locally, so running it again would fail with an error. It shouldn’t wipe anything though, as @stgraber mentioned.
has anyone been able to get this working properly on Debian Buster yet ? It keeps hanging for me after it finds the other hosts, but then it just hangs.
I’ve installed avahi as a multicast and also snap mdns with something not happening in the clustering formation from microcloud init.
This works perfectly on an Ubuntu distro. Seems that there are some missing things when it comes to Debian, and maybe i could help with fixing this for future Debian users, just kinda need some help… From what i found, the multicast/mdns feature was the only thing i found not part of regular install… are there other modules/applications needed to make everything work?
Anything useful when looking at journalctl on all systems?
Here is snap logs microcloud output
root@debian-nuc:/home/mihai# snap logs microcloud
2023-05-13T06:43:21-07:00 systemd[1]: Stopped Service for snap application microcloud.daemon.
2023-05-13T06:43:21-07:00 systemd[1]: snap.microcloud.daemon.service: Consumed 1.059s CPU time.
2023-05-13T06:43:21-07:00 systemd[1]: Started Service for snap application microcloud.daemon.
2023-05-13T06:43:21-07:00 microcloud.daemon[5096]: time="2023-05-13T06:43:21-07:00" level=warning msg="microcluster database is uninitialized"
2023-05-14T06:05:23-07:00 systemd[1]: Stopping Service for snap application microcloud.daemon...
2023-05-14T06:05:23-07:00 systemd[1]: snap.microcloud.daemon.service: Succeeded.
2023-05-14T06:05:23-07:00 systemd[1]: Stopped Service for snap application microcloud.daemon.
2023-05-14T06:05:23-07:00 systemd[1]: snap.microcloud.daemon.service: Consumed 36.143s CPU time.
2023-05-14T06:05:23-07:00 systemd[1]: Started Service for snap application microcloud.daemon.
2023-05-14T06:05:23-07:00 microcloud.daemon[89703]: time="2023-05-14T06:05:23-07:00" level=warning msg="microcluster database is uninitialized"
journalctl showed something interesting about trust store failing… maybe thats it ? but ive tried restarting all the services …
May 12 11:53:49 debian-dell2 systemd[1]: snap.microcloud.daemon.service: Main process exited, code=exited, status=1/FAILURE
May 12 11:53:49 debian-dell2 systemd[1]: snap.microcloud.daemon.service: Failed with result 'exit-code'.
May 12 11:53:49 debian-dell2 systemd[1]: snap.microcloud.daemon.service: Scheduled restart job, restart counter is at 3.
May 12 11:53:49 debian-dell2 systemd[1]: Stopped Service for snap application microcloud.daemon.
May 12 11:53:49 debian-dell2 systemd[1]: Started Service for snap application microcloud.daemon.
May 12 11:53:49 debian-dell2 microcloud.daemon[1286]: Error: Unable to start daemon: Daemon failed to start: Failed to initialize trust store: Failed
to parse local record "". Found empty certificate
May 12 11:53:49 debian-dell2 systemd[1]: snap.microcloud.daemon.service: Main process exited, code=exited, status=1/FAILURE
May 12 11:53:49 debian-dell2 systemd[1]: snap.microcloud.daemon.service: Failed with result 'exit-code'.
May 12 11:53:49 debian-dell2 systemd[1]: snap.microcloud.daemon.service: Scheduled restart job, restart counter is at 4.
May 12 11:53:49 debian-dell2 systemd[1]: Stopped Service for snap application microcloud.daemon.
May 12 11:53:49 debian-dell2 systemd[1]: Started Service for snap application microcloud.daemon.
May 12 11:53:49 debian-dell2 microcloud.daemon[1367]: Error: Unable to start daemon: Daemon failed to start: Failed to initialize trust store: Failed
to parse local record "". Found empty certificate
And Here is one of the debian nodes that i think is being a real problem…
May 13 04:23:35 debian-nuc systemd[1]: Started Service for snap application microcloud.daemon.
May 13 04:23:35 debian-nuc microcloud.daemon[51169]: time="2023-05-13T04:23:35-07:00" level=warning msg="microcluster database is uninitialized"
May 13 04:24:51 debian-nuc systemd[1]: Stopping Service for snap application microcloud.daemon...
May 13 04:25:21 debian-nuc systemd[1]: snap.microcloud.daemon.service: State 'stop-sigterm' timed out. Killing.
May 13 04:25:21 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 51169 (microcloudd) with signal SIGKILL.
May 13 04:25:21 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 51196 (microcloudd) with signal SIGKILL.
May 13 04:25:21 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 51200 (n/a) with signal SIGKILL.
May 13 04:25:22 debian-nuc systemd[1]: snap.microcloud.daemon.service: Main process exited, code=killed, status=9/KILL
May 13 04:25:22 debian-nuc systemd[1]: snap.microcloud.daemon.service: Failed with result 'timeout'.
May 13 04:25:22 debian-nuc systemd[1]: Stopped Service for snap application microcloud.daemon.
-- Boot 71bbf0f06f6246579d5477b585d361b5 --
May 13 04:34:23 debian-nuc systemd[1]: Started Service for snap application microcloud.daemon.
May 13 04:34:25 debian-nuc microcloud.daemon[880]: time="2023-05-13T04:34:25-07:00" level=warning msg="microcluster database is uninitialized"
May 13 04:36:47 debian-nuc systemd[1]: Stopping Service for snap application microcloud.daemon...
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: State 'stop-sigterm' timed out. Killing.
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 880 (microcloudd) with signal SIGKILL.
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 1041 (microcloudd) with signal SIGKILL.
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 1042 (microcloudd) with signal SIGKILL.
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 1044 (microcloudd) with signal SIGKILL.
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 1045 (microcloudd) with signal SIGKILL.
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 1055 (microcloudd) with signal SIGKILL.
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 1056 (microcloudd) with signal SIGKILL.
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Main process exited, code=killed, status=9/KILL
May 13 04:37:17 debian-nuc systemd[1]: snap.microcloud.daemon.service: Failed with result 'timeout'.
May 13 04:37:17 debian-nuc systemd[1]: Stopped Service for snap application microcloud.daemon.
May 13 04:37:17 debian-nuc systemd[1]: Started Service for snap application microcloud.daemon.
May 13 04:37:17 debian-nuc microcloud.daemon[1651]: time="2023-05-13T04:37:17-07:00" level=warning msg="microcluster database is uninitialized"
May 13 04:37:42 debian-nuc systemd[1]: Stopping Service for snap application microcloud.daemon...
May 13 04:38:12 debian-nuc systemd[1]: snap.microcloud.daemon.service: State 'stop-sigterm' timed out. Killing.
May 13 04:38:12 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 1651 (microcloudd) with signal SIGKILL.
May 13 04:38:12 debian-nuc systemd[1]: snap.microcloud.daemon.service: Killing process 1675 (microcloudd) with signal SIGKILL.
@masnax any idea what the empty certificate issue would be here?
Im honestly not sure what it was, but a purge and reinstall of the services (including snapd) and erasing all the folders seems to have fixed everything. Odd I am sorry.
I guess my next question is that its hard to find documentation on microovn, I have not really played with OVN before, but i know its used in industry so i may as well mess around with it.
My containers do not have any internet access once microcloud and ovn is up. Do i just assume all OVN features are available? is there something missing in the micro version of ovn ? what does the snap version automatically setup ?
I followed this link of you @stgraber and it helped kinda
https://www.youtube.com/watch?v=iWZYUU8lX5A
Thank you so much for putting this out! I’m trying to teach myself about clouds, kubernetes, etc. using my homelab, and this makes a great base! It took a couple of tries, but I was able to complete the install.
Now I’m trying to interact with my new cluster using Juju. When bootstrapping, Juju cannot connect to the newly created controller using IPV4 (IPV6 works fine). I created two test containers on the default network and was able to SSH between them. However, I cannot SSH from outside the default network (though ping gets a response). Here’s my config for the default network:
config:
bridge.mtu: "1442"
ipv4.address: 10.130.250.1/24
ipv4.nat: "true"
ipv6.address: fd42:f524:8ebc:ab12::1/64
ipv6.nat: "true"
network: UPLINK
volatile.network.ipv4.address: 10.1.2.101
description: ""
name: default
type: ovn
used_by:
- /1.0/instances/test
- /1.0/profiles/default
managed: true
status: Created
locations:
- metal01.local
- metal02.local
- metal03.local
Happy to provide additional info; any help is appreciated.
Here’s the output from SSH -v when I try to connect:
OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 10.130.250.2 [10.130.250.2] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type 2
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/user/.ssh/id_xmss type -1
debug1: identity file /home/user/.ssh/id_xmss-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
kex_exchange_identification: read: Connection reset by peer
Connection reset by 10.130.250.2 port 22
Is there a list of dependancies for microcloud. I’ve discovered zfsutils-linux is one of them…
zfsutils-linux is not a dependency.
All the components of MicroCloud come as snaps and as a result cannot use anything from the host system, so there are no dependencies. In this case, all the ZFS tools come built into the LXD snap.
Hi @stgraber , I’m considering using the microcloud for the home lab redeployment. This consists of 4 servers, that should be capable to run both microcloud and microk8s side by side (which is what I’m aiming at). Would you possibly know (or did you come in contact with the microk8s devs on this), if there is a way to reuse microceph in microk8s?
Thanks, best, Pavel
Whats gonna happen to microcloud due to the Canonical takeover ?
No changes in the ambitions. MicroCloud has always been a Canonical initiative, and we are continuing to invest in it and enhancing it going forward.
I’d also be interested in hearing more about the relationship (or lack thereof) between microcloud and microk8s.
Maybe @mionaalex can update on progress, but the plan was to allow for MicroCloud to create Kubernetes clusters by spawning LXD VMs and then having those run microk8s.
So in the initial MicroCloud there was no relation with microk8s, but as that evolved, Kubernetes was meant to be added and that would be done by deploying microk8s inside of VMs.
1 Like