Thanks to the inputs from the others on the Internet, expecially the Linux Containers, I have gathered enough information to finish my project. It is time for me to do my part to share my experience here. . Hopefully, it could be helpful to anyone.
I have been playing around with LXDd for a couple of years. Recently, I tried to setup a cluster of LXD nodes using remote storage. My choice is to use ceph as the backend storage.
First of all, this post is based on a video clip by Dimzrio, “Dimzrio Tutorial” on YouTube. As I am not a video person, I have transcribed its content and tested them here.
The system that I have set up consists of 4 nodes. They have similar configuration as below:
- Ubuntu 20.04
- One RAID 1 drive for the OS;
- Two hard drives (2T) for the ceph;
- RAM is around 64G each;
(I used cssh to execute the same command in all nodes. You need to configure sshd to allow root access.)
First thing first, install the latest LXD
In Ubuntu 20.04, the default snap package of LXD installed is 4.0. To upgrade it to the latest stable version which is, up to the writing of this post, 4.8:
sudo snap refresh lxd --channel=4.8/stable
Secondly, install ceph in all nodes
sudo apt install ceph -y
In Node1, get the UUID
$uuidgen
make a note of the uuid just generated.
In all nodes
export the generated UUID using:
export cephuid=ABCDEFG <----- replace “ABCDEFG” with the uuid generated.
sudo vi /etc/ceph/ceph.conf
Insert the following into /etc/ceph/ceph.conf
[global]
fsid=ABCDEFG
mon_initial_members = node1, node2, node3, node4
mon_host = node1_ip, node2_ip, node3_ip, node4_ip
public_network = 192.168.1.0/24 <---- replace this with your network
auth_cluster_required = none
auth_service_required = none
auth_client_required = none
osd_journal_size = 1024
osd_pool_default_size = 3
osd_pool_default_min_size = 2
osd_pool_default_pg_num = 333
osd_pool_default_pgp_num = 333
osd_crush_chooseleaf_type = 1
In Node1
Create ceph monitor secret key (host only)
ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon "allow *"
ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon "allow *" --cap mgr "allow *" --cap osd "allow *" --cap mds "allow *"
ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon "profile bootstrap-osd" --cap mgr "allow r"
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
Generate ceph monitor map
monmaptool --create --add node1 node1_ip --add node2 node2_ip --add node3 node3_ip --add node4 node4_ip --fsid $cephuid /tmp/monmap
Copy the monmap to all other nodes:
scp /tmp/monmap node2_ip:/tmp
scp /tmp/monmap node3_ip:/tmp
scp /tmp/monmap node4_ip:/tmp
Copy the ceph.client.admin.keyring to all other nodes:
scp /etc/ceph/ceph.client.admin.keyring node2_ip:/etc/ceph
scp /etc/ceph/ceph.client.admin.keyring node3_ip:/etc/ceph
scp /etc/ceph/ceph.client.admin.keyring node4_ip:/etc/ceph
Copy ceph.mon.keyring to all other nodes:
scp /tmp/ceph.mon.keyring node2_ip:/tmp
scp /tmp/ceph.mon.keyring node3_ip:/tmp
scp /tmp/ceph.mon.keyring node4_ip:/tmp
Create data directory for monitor
In Node1
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node1
sudo -u ceph ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
ls /var/lib/ceph/mon/ceph-node1
systemctl restart ceph-mon@node1
In Node2
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node2
sudo -u ceph ceph-mon --mkfs -i node2 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
systemctl restart ceph-mon@node2
In Node3
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node3
sudo -u ceph ceph-mon --mkfs -i node3 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
systemctl restart ceph-mon@node3
In Node4
sudo -u ceph mkdir /var/lib/ceph/mon/ceph-node4
sudo -u ceph ceph-mon --mkfs -i node4 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
systemctl restart ceph-mon@node4
Setup ceph manager
In Node1
ceph auth get-or-create mgr.node1 mon 'allow profile mgr' osd 'allow *' mds 'allow *'
Make a note of the displayed key
sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node1
sudo -u ceph nano /var/lib/ceph/mgr/ceph-node1/keyring
Insert the key into the file
[mgr.node1]
key =
ceph mon enable-msgr2
systemctl restart ceph-mgr@node1
In Node2
ceph auth get-or-create mgr.node2 mon 'allow profile mgr' osd 'allow *' mds 'allow *'
Make a note of the displayed key
sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node2
sudo -u ceph nano /var/lib/ceph/mgr/ceph-node2/keyring
Insert the key into the file
[mgr.node2]
key =
systemctl restart ceph-mgr@node2
In Node3
ceph auth get-or-create mgr.node3 mon 'allow profile mgr' osd 'allow *' mds 'allow *'
Make a note of the displayed key
sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node3
sudo -u ceph nano /var/lib/ceph/mgr/ceph-node3/keyring
Insert the key into the file
[mgr.node3]
key =
systemctl restart ceph-mgr@node3
In Node 4
ceph auth get-or-create mgr.node4 mon 'allow profile mgr' osd 'allow *' mds 'allow *'
Make a note of the displayed key
sudo -u ceph mkdir /var/lib/ceph/mgr/ceph-node4
sudo -u ceph nano /var/lib/ceph/mgr/ceph-node4/keyring
Insert the key into the file
[mgr.node4]
key =
systemctl restart ceph-mgr@node4
Setup NTP client
In Ubuntu 20.04, it can be done by timedatectl. In all nodes,
sudo timedatectl set-timezone Asia/Hong_Kong <---- change to your time zone
configure it to sync with a ntp server.
Setup ceph osd
In all nodes:
sudo ceph-volume lvm create --data /dev/sdb (or the specific lv)
sudo systemctl restart ceph-osd@#
(# is the node number starting from 0. One unique number for each volume). In my case, I have
node1: ceph-osd@0, ceph=osd@1
node2: ceph-osd@2, ceph=osd@3
node3: ceph-osd@4, ceph=osd@5
node4: ceph-osd@6, ceph=osd@7
If you did something wrong, you can remove the osd by
ceph osd out osd.#
ceph osd purge purge osd.# --force
ceph-volume lvm zap --destroy /dev/sd#
Setup ceph mds
In Node1
sudo -u ceph mkdir /var/lib/ceph/mds/ceph-node1 -p
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node1/keyring --gen-key -n mds.node1
ceph auth add mds.node1 osd “allow rwx” mds “allow” mon “allow profile mds” -i /var/lib/ceph/mds/ceph-node1/keyring
systemctl restart ceph-mds@node1
In Node2
sudo -u ceph mkdir /var/lib/ceph/mds/ceph-node2 -p
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node2/keyring --gen-key -n mds.node2
ceph auth add mds.node2 osd “allow rwx” mds “allow” mon “allow profile mds” -i /var/lib/ceph/mds/ceph-node2/keyring
systemctl restart ceph-mds@node2
In Node3
sudo -u ceph mkdir /var/lib/ceph/mds/ceph-node3 -p
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node3/keyring --gen-key -n mds.node3
ceph auth add mds.node3 osd “allow rwx” mds “allow” mon “allow profile mds” -i /var/lib/ceph/mds/ceph-node3/keyring
systemctl restart ceph-mds@node3
In Node4
sudo -u ceph mkdir /var/lib/ceph/mds/ceph-node4 -p
ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-node4/keyring --gen-key -n mds.node4
ceph auth add mds.node4 osd “allow rwx” mds “allow” mon “allow profile mds” -i /var/lib/ceph/mds/ceph-node4/keyring
systemctl restart ceph-mds@node4
add the following into /etc/ceph/ceph.conf in all nodes:
[mds.node1]
host = node1
[mds.node2]
host = node2
[mds.node3]
host = node3
[mds.node4]
host = node4
Restart all ceph services
In Node1,
systemctl restart ceph-mon@node1
systemctl restart ceph-mgr@node1
systemctl restart ceph-mds@node1
systemctl restart ceph-osd@0
systemctl restart ceph-osd@1
In Node2,
systemctl restart ceph-mon@node2
systemctl restart ceph-mgr@node2
systemctl restart ceph-mds@node2
systemctl restart ceph-osd@2
systemctl restart ceph-osd@3
In Node3,
systemctl restart ceph-mon@node3
systemctl restart ceph-mgr@node3
systemctl restart ceph-mds@node3
systemctl restart ceph-osd@4
systemctl restart ceph-osd@5
In Node4,
systemctl restart ceph-mon@node4
systemctl restart ceph-mgr@node4
systemctl restart ceph-mds@node4
systemctl restart ceph-osd@6
systemctl restart ceph-osd@7
Now, you may check the status of the newly configured ceph
ceph -s
To check the osd tree,
ceph osd tree
Create a new pool,
sudo ceph osd pool create lxd-ceph 250
250 is the placement group number calculated according to my setup. You may use a different one.
If the ceph is running fine, it is time to initiate the LXD
Initiate LXD
In Node1,
sudo lxd init
answer the qustions with the following answers:
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What name should be used to identify this node in the cluster? [default=node1]:
What IP address or DNS name should be used to reach this node? [default=node1_ip]:
Are you joining an existing cluster? (yes/no) [default=no]: no
Setup password authentication on the cluster? (yes/no) [default=yes]: yes
Do you want to configure a new local storage pool? (yes/no) [default=yes]: no
Do you want to configure a new remote storage pool? (yes/no) [default=no]: yes
Name of the storage backend to use (ceph, cephfs) [default=ceph]: ceph
Create a new CEPH pool? (yes/no) [default=yes]:
Name of the existing CEPH cluster [default=ceph]:
Name of the OSD storage pool [default=lxd]: lxd-ceph
Number of placement groups [default=32]: 250
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]:
Would you like to create a new Fan overlay network? (yes/no) [default=yes]:
What subnet should be used as the Fan underlay? [default=auto]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML “lxd init” preseed to be printed? (yes/no) [default=no]:
The LXD should be up and running. Initialise the LXD in other nodes. Answer YES to “Are you joining an existing cluster?”.
Well, that is it! I hope that you will enjoy this.
Regards,
Terry Ng.