When installing microcloud and microceph I get the following message:
root@cluster03:~# snap install lxd microceph microcloud
2023-04-23T15:34:52Z INFO snap "microcloud" has bad plugs or slots: microovn (unknown interface "microovn")
microceph 0+git.dfece8b from Canonical✓ installed
microcloud 0+git.99d8306 from Canonical✓ installed
lxd already installed
WARNING: There is 1 new warning. See 'snap warnings'.
This is inside a LXC vm, in case it is important.
Make sure snapd is up to date, current stable snapd should be 2.59.1.
It was 2.58.3, but now it is 2.59.1
I started from scratch again and now microcloud installs without warning, microovn is not installed as a dependency. So far so good.
But now, when I do microcloud init (following Introducing MicroCloud and the “Introducing MicroCloud” youtube video) it is hanging.
root@cluster02:~# microcloud init
Select an address for MicroCloud's internal traffic
Using address "10.89.163.28" for MicroCloud
Limit search for other MicroCloud servers to 10.89.163.28/24? (yes/no) [default=yes]:
MicroOVN not found. Continue anyway? (yes/no) [default=yes]:
Scanning for eligible servers...
This hangs forever.
So maybe it needs microovn. I installed it and tried it again.
root@cluster02:~# snap install lxd microceph microcloud microovn
microovn 0+git.b60840f from Canonical✓ installed
lxd already installed
microceph already installed
microcloud already installed
root@cluster02:~# snap list
Name Version Rev Tracking Publisher Notes
core20 20230308 1852 latest/stable canonical✓ base
core22 20230325 607 latest/stable canonical✓ base
lxd 5.0.2-838e1b2 24322 5.0/stable/… canonical✓ -
microceph 0+git.dfece8b 328 latest/stable canonical✓ -
microcloud 0+git.99d8306 334 latest/stable canonical✓ -
microovn 0+git.b60840f 81 latest/stable canonical✓ -
snapd 2.59.1 18933 latest/stable canonical✓ snapd
root@cluster02:~# microcloud init
Select an address for MicroCloud's internal traffic
Using address "10.89.163.28" for MicroCloud
Limit search for other MicroCloud servers to 10.89.163.28/24? (yes/no) [default=yes]:
Scanning for eligible servers...
The “MicroOVN not found” warning is gone. But other than that it is still hanging.
What kind of environment is this running on?
The inability to find the other servers suggests you may have a network layer which doesn’t allow broadcast/multicast?
The environment is made of several VMs in LXC. The network is the default lxdbr0, created by lxd init. I think I have created the same as you did in the youtube video.
I looked with tcpdump on the VM where microcloud init is running. There is a (seemingly constant) endless repetition of these packages.
20:50:33.132757 IP 10.89.163.28.40648 > 224.0.0.251.mdns: 17412 PTR (QM)? _microcloud_0.local. (37)
20:50:33.133016 IP6 fe80::216:3eff:fefb:57a0.35718 > ff02::fb.mdns: 17412 PTR (QM)? _microcloud_0.local. (37)
20:50:33.133370 IP6 fe80::216:3eff:fe2a:b0b4.mdns > fe80::216:3eff:fefb:57a0.35718: 0*- [0q] 4/0/0 PTR cluster03_enp5s0_10.89.163.61._microcloud_0.local., SRV cluster03.:9443 10 1, A 10.89.163.61, TXT "{"Version":"1.0","Name":"cluster03","Address":"10.89.163.61","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"b11039d6ebb080d258603ce2f54fdfd5f8296a130be0447d87d14c76970cb0ce"}" (327)
20:50:33.133400 IP 10.89.163.109.mdns > 10.89.163.28.40648: 0*- [0q] 4/0/0 PTR cluster01_enp5s0_10.89.163.109._microcloud_0.local., SRV cluster01.:9443 10 1, A 10.89.163.109, TXT "{"Version":"1.0","Name":"cluster01","Address":"10.89.163.109","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"e79d16ea60638e9c18a285c49d9c4a086b050012ab40d85025d9ab7d4a9a6003"}" (329)
20:50:33.133400 IP6 fe80::216:3eff:fe7b:61a9.mdns > fe80::216:3eff:fefb:57a0.35718: 0*- [0q] 4/0/0 PTR cluster01_enp5s0_10.89.163.109._microcloud_0.local., SRV cluster01.:9443 10 1, A 10.89.163.109, TXT "{"Version":"1.0","Name":"cluster01","Address":"10.89.163.109","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"e79d16ea60638e9c18a285c49d9c4a086b050012ab40d85025d9ab7d4a9a6003"}" (329)
20:50:33.133400 IP 10.89.163.61.mdns > 10.89.163.28.40648: 0*- [0q] 4/0/0 PTR cluster03_enp5s0_10.89.163.61._microcloud_0.local., SRV cluster03.:9443 10 1, A 10.89.163.61, TXT "{"Version":"1.0","Name":"cluster03","Address":"10.89.163.61","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"b11039d6ebb080d258603ce2f54fdfd5f8296a130be0447d87d14c76970cb0ce"}" (327)
20:50:33.234176 IP 10.89.163.28.46597 > 224.0.0.251.mdns: 45393 PTR (QM)? _microcloud_1.local. (37)
20:50:33.234510 IP6 fe80::216:3eff:fefb:57a0.58514 > ff02::fb.mdns: 45393 PTR (QM)? _microcloud_1.local. (37)
20:50:33.234918 IP 10.89.163.61.mdns > 10.89.163.28.46597: 0*- [0q] 4/0/0 PTR cluster03_enp5s0_fd42:18a1:39e5:987b:216:3eff:fe2a:b0b4._microcloud_1.local., SRV cluster03.:9443 10 1, AAAA fd42:18a1:39e5:987b:216:3eff:fe2a:b0b4, TXT "{"Version":"1.0","Name":"cluster03","Address":"fd42:18a1:39e5:987b:216:3eff:fe2a:b0b4","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"b11039d6ebb080d258603ce2f54fdfd5f8296a130be0447d87d14c76970cb0ce"}" (391)
20:50:33.234950 IP6 fe80::216:3eff:fe2a:b0b4.mdns > fe80::216:3eff:fefb:57a0.58514: 0*- [0q] 4/0/0 PTR cluster03_enp5s0_fd42:18a1:39e5:987b:216:3eff:fe2a:b0b4._microcloud_1.local., SRV cluster03.:9443 10 1, AAAA fd42:18a1:39e5:987b:216:3eff:fe2a:b0b4, TXT "{"Version":"1.0","Name":"cluster03","Address":"fd42:18a1:39e5:987b:216:3eff:fe2a:b0b4","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"b11039d6ebb080d258603ce2f54fdfd5f8296a130be0447d87d14c76970cb0ce"}" (391)
20:50:33.235316 IP 10.89.163.109.mdns > 10.89.163.28.46597: 0*- [0q] 4/0/0 PTR cluster01_enp5s0_fd42:18a1:39e5:987b:216:3eff:fe7b:61a9._microcloud_1.local., SRV cluster01.:9443 10 1, AAAA fd42:18a1:39e5:987b:216:3eff:fe7b:61a9, TXT "{"Version":"1.0","Name":"cluster01","Address":"fd42:18a1:39e5:987b:216:3eff:fe7b:61a9","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"e79d16ea60638e9c18a285c49d9c4a086b050012ab40d85025d9ab7d4a9a6003"}" (391)
20:50:33.235351 IP6 fe80::216:3eff:fe7b:61a9.mdns > fe80::216:3eff:fefb:57a0.58514: 0*- [0q] 4/0/0 PTR cluster01_enp5s0_fd42:18a1:39e5:987b:216:3eff:fe7b:61a9._microcloud_1.local., SRV cluster01.:9443 10 1, AAAA fd42:18a1:39e5:987b:216:3eff:fe7b:61a9, TXT "{"Version":"1.0","Name":"cluster01","Address":"fd42:18a1:39e5:987b:216:3eff:fe7b:61a9","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"e79d16ea60638e9c18a285c49d9c4a086b050012ab40d85025d9ab7d4a9a6003"}" (391)
20:50:33.335918 IP 10.89.163.28.37073 > 224.0.0.251.mdns: 58551 PTR (QM)? _microcloud_2.local. (37)
20:50:33.336061 IP6 fe80::216:3eff:fefb:57a0.51596 > ff02::fb.mdns: 58551 PTR (QM)? _microcloud_2.local. (37)
20:50:33.436684 IP 10.89.163.28.47730 > 224.0.0.251.mdns: 14930 PTR (QM)? _microcloud_3.local. (37)
20:50:33.436738 IP6 fe80::216:3eff:fefb:57a0.36476 > ff02::fb.mdns: 14930 PTR (QM)? _microcloud_3.local. (37)
20:50:33.537588 IP 10.89.163.28.36886 > 224.0.0.251.mdns: 2094 PTR (QM)? _microcloud_4.local. (37)
20:50:33.537727 IP6 fe80::216:3eff:fefb:57a0.45690 > ff02::fb.mdns: 2094 PTR (QM)? _microcloud_4.local. (37)
20:50:33.638566 IP 10.89.163.28.45217 > 224.0.0.251.mdns: 1947 PTR (QM)? _microcloud_5.local. (37)
20:50:33.638710 IP6 fe80::216:3eff:fefb:57a0.53666 > ff02::fb.mdns: 1947 PTR (QM)? _microcloud_5.local. (37)
20:50:33.739389 IP 10.89.163.28.44083 > 224.0.0.251.mdns: 24565 PTR (QM)? _microcloud_6.local. (37)
20:50:33.739504 IP6 fe80::216:3eff:fefb:57a0.47089 > ff02::fb.mdns: 24565 PTR (QM)? _microcloud_6.local. (37)
20:50:33.840447 IP 10.89.163.28.49541 > 224.0.0.251.mdns: 15771 PTR (QM)? _microcloud_7.local. (37)
20:50:33.845413 IP6 fe80::216:3eff:fefb:57a0.54085 > ff02::fb.mdns: 15771 PTR (QM)? _microcloud_7.local. (37)
20:50:33.946165 IP 10.89.163.28.49782 > 224.0.0.251.mdns: 21187 PTR (QM)? _microcloud_8.local. (37)
20:50:33.946366 IP6 fe80::216:3eff:fefb:57a0.57376 > ff02::fb.mdns: 21187 PTR (QM)? _microcloud_8.local. (37)
20:50:34.047179 IP 10.89.163.28.49223 > 224.0.0.251.mdns: 35232 PTR (QM)? _microcloud_9.local. (37)
20:50:34.047362 IP6 fe80::216:3eff:fefb:57a0.60221 > ff02::fb.mdns: 35232 PTR (QM)? _microcloud_9.local. (37)
20:50:34.148505 IP 10.89.163.28.42074 > 224.0.0.251.mdns: 59556 PTR (QM)? _microcloud_0.local. (37)
20:50:34.148555 IP6 fe80::216:3eff:fefb:57a0.47646 > ff02::fb.mdns: 59556 PTR (QM)? _microcloud_0.local. (37)
20:50:34.149043 IP6 fe80::216:3eff:fe2a:b0b4.mdns > fe80::216:3eff:fefb:57a0.47646: 0*- [0q] 4/0/0 PTR cluster03_enp5s0_10.89.163.61._microcloud_0.local., SRV cluster03.:9443 10 1, A 10.89.163.61, TXT "{"Version":"1.0","Name":"cluster03","Address":"10.89.163.61","Interface":"enp5s0","Services":["MicroCloud","LXD"],"AuthSecret":"b11039d6ebb080d258603ce2f54fdfd5f8296a130be0447d87d14c76970cb0ce"}" (327)
BTW. It now picks up 3 out of 4 eligible servers.
Just a wild guess. Can it be a mixup of IPv4 and IPv6? I see in the return packet of a mdns request IPv6 info. For example:
19:25:57.770929 IP (tos 0x0, ttl 64, id 34418, offset 0, flags [DF], proto UDP (17), length 367)
10.89.163.73.5353 > 10.89.163.28.35623: [bad udp cksum 0x5c84 -> 0xf137!] 0*- [0q] 4/0/0 _microcloud_0.local. PTR cluster04_enp5s0_10.89.163.73._microcloud_0.local., cluster04_enp5s0_10.89.163.73._microcloud_0.local. SRV cluster04.:9443 10 1, cluster04. A 10.89.163.73, cluster04_enp5s0_10.89.163.73._microcloud_0.local. TXT "{"Version":"1.0","Name":"cluster04","Address":"10.89.163.73","Interface":"enp5s0","Services":["MicroCloud","LXD","MicroCeph"],"AuthSecret":"ec23d7fbb48bc767cea0a184c2a30e073f313693743a8c1e05b6dbe25f001720"}" (339)
19:25:57.871962 IP (tos 0x0, ttl 64, id 34421, offset 0, flags [DF], proto UDP (17), length 431)
10.89.163.73.5353 > 10.89.163.28.38323: [bad udp cksum 0x5cc4 -> 0x9dbe!] 0*- [0q] 4/0/0 _microcloud_1.local. PTR cluster04_enp5s0_fd42:18a1:39e5:987b:216:3eff:fe24:3cde._microcloud_1.local., cluster04_enp5s0_fd42:18a1:39e5:987b:216:3eff:fe24:3cde._microcloud_1.local. SRV cluster04.:9443 10 1, cluster04. AAAA fd42:18a1:39e5:987b:216:3eff:fe24:3cde, cluster04_enp5s0_fd42:18a1:39e5:987b:216:3eff:fe24:3cde._microcloud_1.local. TXT "{"Version":"1.0","Name":"cluster04","Address":"fd42:18a1:39e5:987b:216:3eff:fe24:3cde","Interface":"enp5s0","Services":["MicroCloud","LXD","MicroCeph"],"AuthSecret":"ec23d7fbb48bc767cea0a184c2a30e073f313693743a8c1e05b6dbe25f001720"}" (403)
cluster04 is the one missing.
BTW. The --verbose and --debug options don’t give any extra output. How is that suppose to work?
Try restarting microcloud on the 4th with snap restart microcloud see if that helps.
Yes that helps