gabriel
May 10, 2021, 5:38am
1
Hi All,
5x small servers, Ubuntu 20.04, LXC/LXD 4.0.6, JUJU for Charmed Kubernetes (k8s).
I had some success with a standalone non-clustered installation of lxd with charmed k8s.
Then attempted a clustered installation of lxd with charmed k8s, however, are unable to complete the k8s deployment due to problems with the lxd storage pool.
I think the problem has arisen from settings entered during lxd init.
Receiving the error message below?
Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
Have posted further information below.
Appreciate any help.
Gabriel
tomp
May 10, 2021, 10:26am
2
Did you run sudo lxd init on each so that it creates a default profile that uses the desired storage pool?
gabriel
May 10, 2021, 6:45pm
3
Yes, ran each lxd init on each node to create a default profile.
Have spent a few hours reviewing lxc profiles, still unsure about the clustered storage pool configuration, this seems to be the problem for deployment of k8s.
Default profiles, storage pools and juju status posted below for non-clustered and clustered deployments.
Noticing a difference between defaut and local, storage pools?
gabriel
May 11, 2021, 2:18am
5
Non-clustered lxd, juju charmed k8s
profile
$ lxc profile show default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: default
used_by:
- /1.0/instances/juju-72bfc4-0
- /1.0/instances/juju-760a74-0
- /1.0/instances/juju-760a74-1
- /1.0/instances/juju-760a74-2
- /1.0/instances/juju-760a74-4
- /1.0/instances/juju-760a74-3
- /1.0/instances/juju-760a74-8
- /1.0/instances/juju-760a74-7
- /1.0/instances/juju-760a74-6
- /1.0/instances/juju-760a74-9
- /1.0/instances/juju-760a74-5
storage pool
$ lxc storage show default
config:
source: /var/snap/lxd/common/lxd/storage-pools/default
description: ""
name: default
driver: dir
used_by:
- /1.0/instances/juju-72bfc4-0
- /1.0/instances/juju-760a74-0
- /1.0/instances/juju-760a74-1
- /1.0/instances/juju-760a74-2
- /1.0/instances/juju-760a74-3
- /1.0/instances/juju-760a74-4
- /1.0/instances/juju-760a74-5
- /1.0/instances/juju-760a74-6
- /1.0/instances/juju-760a74-7
- /1.0/instances/juju-760a74-8
- /1.0/instances/juju-760a74-9
- /1.0/profiles/default
status: Created
locations:
- none
status
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
k8s localhost-localhost localhost/localhost 2.9.0 unsupported 14:12:00+12:00
App Version Status Scale Charm Store Channel Rev OS Message
containerd 1.3.3 active 5 containerd charmstore stable 105 ubuntu Container runtime available
easyrsa 3.0.1 active 1 easyrsa charmstore stable 359 ubuntu Certificate Authority connected.
etcd 3.4.5 active 3 etcd charmstore stable 569 ubuntu Healthy with 3 known peers
flannel 0.11.0 active 5 flannel charmstore stable 533 ubuntu Flannel subnet 10.1.31.1/24
kubeapi-load-balancer 1.18.0 active 1 kubeapi-load-balancer charmstore stable 772 ubuntu Loadbalancer ready.
kubernetes-master 1.21.0 waiting 2 kubernetes-master charmstore stable 974 ubuntu Waiting for 3 kube-system pods to start
kubernetes-worker 1.21.0 active 3 kubernetes-worker charmstore stable 743 ubuntu Kubernetes worker running.
Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 10.3.30.176 Certificate Authority connected.
etcd/0* active idle 1 10.3.30.146 2379/tcp Healthy with 3 known peers
etcd/1 active idle 2 10.3.30.115 2379/tcp Healthy with 3 known peers
etcd/2 active idle 3 10.3.30.120 2379/tcp Healthy with 3 known peers
kubeapi-load-balancer/0* active idle 4 10.3.30.225 443/tcp Loadbalancer ready.
kubernetes-master/0 waiting idle 5 10.3.30.116 6443/tcp Waiting for 3 kube-system pods to start
containerd/3 active idle 10.3.30.116 Container runtime available
flannel/3 active idle 10.3.30.116 Flannel subnet 10.1.74.1/24
kubernetes-master/1* waiting idle 6 10.3.30.6 6443/tcp Waiting for 3 kube-system pods to start
containerd/4 active idle 10.3.30.6 Container runtime available
flannel/4 active idle 10.3.30.6 Flannel subnet 10.1.98.1/24
kubernetes-worker/0 active idle 7 10.3.30.52 80/tcp,443/tcp Kubernetes worker running.
containerd/2 active idle 10.3.30.52 Container runtime available
flannel/2 active idle 10.3.30.52 Flannel subnet 10.1.10.1/24
kubernetes-worker/1 active idle 8 10.3.30.156 80/tcp,443/tcp Kubernetes worker running.
containerd/0* active idle 10.3.30.156 Container runtime available
flannel/0* active idle 10.3.30.156 Flannel subnet 10.1.31.1/24
kubernetes-worker/2* active idle 9 10.3.30.110 80/tcp,443/tcp Kubernetes worker running.
containerd/1 active idle 10.3.30.110 Container runtime available
flannel/1 active idle 10.3.30.110 Flannel subnet 10.1.97.1/24
Machine State DNS Inst id Series AZ Message
0 started 10.3.30.176 juju-760a74-0 focal Running
1 started 10.3.30.146 juju-760a74-1 focal Running
2 started 10.3.30.115 juju-760a74-2 focal Running
3 started 10.3.30.120 juju-760a74-3 focal Running
4 started 10.3.30.225 juju-760a74-4 focal Running
5 started 10.3.30.116 juju-760a74-5 focal Running
6 started 10.3.30.6 juju-760a74-6 focal Running
7 started 10.3.30.52 juju-760a74-7 focal Running
8 started 10.3.30.156 juju-760a74-8 focal Running
9 started 10.3.30.110 juju-760a74-9 focal Running
gabriel
May 11, 2021, 2:21am
6
Clustered lxd, juju charmed k8s
cluster
$ lxc cluster list
+---------+----------------------------+----------+--------+-------------------+--------------+
| NAME | URL | DATABASE | STATE | MESSAGE | ARCHITECTURE |
+---------+----------------------------+----------+--------+-------------------+--------------+
| hal-126 | https://192.168.n.nnn:8443 | YES | ONLINE | Fully operational | x86_64 |
+---------+----------------------------+----------+--------+-------------------+--------------+
| hal-127 | https://192.168.n.nnn:8443 | YES | ONLINE | Fully operational | x86_64 |
+---------+----------------------------+----------+--------+-------------------+--------------+
| hal-128 | https://192.168.n.nnn:8443 | YES | ONLINE | Fully operational | x86_64 |
+---------+----------------------------+----------+--------+-------------------+--------------+
profile
$ lxc profile show default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdfan0
type: nic
root:
path: /
pool: local
type: disk
name: default
used_by:
- /1.0/instances/juju-0a3132-0
storage pool
$ lxc storage show local
config: {}
description: ""
name: local
driver: dir
used_by:
- /1.0/instances/juju-0a3132-0
- /1.0/profiles/default
- /1.0/profiles/hal-ubuntu-18-04
status: Created
locations:
- hal-126
- hal-127
- hal-128
status
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
k8s localhost-localhost localhost/localhost 2.9.0 unsupported 14:21:34+12:00
App Version Status Scale Charm Store Channel Rev OS Message
containerd unknown 0 containerd charmstore stable 119 ubuntu
easyrsa waiting 0/1 easyrsa charmstore stable 373 ubuntu waiting for machine
etcd waiting 0/3 etcd charmstore stable 583 ubuntu waiting for machine
flannel unknown 0 flannel charmstore stable 547 ubuntu
kubeapi-load-balancer waiting 0/1 kubeapi-load-balancer charmstore stable 786 ubuntu waiting for machine
kubernetes-master waiting 0/2 kubernetes-master charmstore stable 990 ubuntu waiting for machine
kubernetes-worker waiting 0/3 kubernetes-worker charmstore stable 757 ubuntu waiting for machine
Unit Workload Agent Machine Public address Ports Message
easyrsa/0 waiting allocating 0 waiting for machine
etcd/0 waiting allocating 1 waiting for machine
etcd/1 waiting allocating 2 waiting for machine
etcd/2 waiting allocating 3 waiting for machine
kubeapi-load-balancer/0 waiting allocating 4 waiting for machine
kubernetes-master/0 waiting allocating 5 waiting for machine
kubernetes-master/1 waiting allocating 6 waiting for machine
kubernetes-worker/0 waiting allocating 7 waiting for machine
kubernetes-worker/1 waiting allocating 8 waiting for machine
kubernetes-worker/2 waiting allocating 9 waiting for machine
Machine State DNS Inst id Series AZ Message
0 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
1 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
2 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
3 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
4 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
5 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
6 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
7 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
8 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
9 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
tomp
May 11, 2021, 9:00am
7
What happens if you run:
lxc init images:ubuntu/focal c1
tomp
May 11, 2021, 9:00am
8
Also, are you using projects by any chance?
gabriel
May 11, 2021, 7:07pm
9
Tested a few commands as suggested, not running lxd projects.
$ lxc init images:ubuntu/focal c1
Creating c1
$ lxc init images:ubuntu/focal c2
Creating c2
$ lxc list
+---------------+---------+----------------------+------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| c1 | STOPPED | | | CONTAINER | 0 | hal-127 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| c2 | STOPPED | | | CONTAINER | 0 | hal-128 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| juju-4d0137-0 | RUNNING | 240.126.0.222 (eth0) | | CONTAINER | 0 | hal-126 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
started containers
$ lxc start c1
$ lxc start c2
$ lxc launch ubuntu:20.04 c3
Creating c3
Starting c3
$ lxc list
+---------------+---------+----------------------+------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| c1 | RUNNING | 240.127.0.63 (eth0) | | CONTAINER | 0 | hal-127 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| c2 | RUNNING | 240.128.0.90 (eth0) | | CONTAINER | 0 | hal-128 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| c3 | RUNNING | 240.126.0.197 (eth0) | | CONTAINER | 0 | hal-126 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| juju-4d0137-0 | RUNNING | 240.126.0.222 (eth0) | | CONTAINER | 0 | hal-126 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
gabriel
May 11, 2021, 7:14pm
10
Working through the clustering documentation, it looked like was missing a couple of steps on storage pool and voume creation, tested these steps but did not seem to help:
https://lxd.readthedocs.io/en/latest/clustering/
$ lxc storage create --target hal-126 data dir source=/data/lxd
Storage pool data pending on member hal-126
$ lxc storage create --target hal-127 data dir source=/data/lxd
Storage pool data pending on member hal-127
$ lxc storage create --target hal-128 data dir source=/data/lxd
Storage pool data pending on member hal-128
$ lxc storage volume create data lxdvol --target hal-126
$ lxc storage volume create data lxdvol --target hal-127
$ lxc storage volume create data lxdvol --target hal-128
$ lxc storage volume show data lxdvol --target hal-126
config: {}
description: ""
name: lxdvol
type: custom
used_by: []
location: hal-126
content_type: filesystem
$ lxc storage volume show data lxdvol --target hal-127
config: {}
description: ""
name: lxdvol
type: custom
used_by: []
location: hal-127
content_type: filesystem
$ lxc storage volume show data lxdvol --target hal-128
config: {}
description: ""
name: lxdvol
type: custom
used_by: []
location: hal-128
content_type: filesystem
updated default profile storage pool
$ lxc profile show default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdfan0
type: nic
root:
path: /
pool: data
type: disk
name: default
used_by:
- /1.0/instances/juju-4d0137-0
- /1.0/instances/c1
- /1.0/instances/c2
- /1.0/instances/c3
gabriel
May 11, 2021, 7:16pm
11
not sure how to verify the new storage pool and volumes are working correctly within the containers, have set dir paths to custom directory
tomp
May 12, 2021, 7:51am
12
This doesn’t look quite right:
After you have done:
lxc storage create --target hal-126 data dir source=/data/lxd
lxc storage create --target hal-127 data dir source=/data/lxd
lxc storage create --target hal-128 data dir source=/data/lxd
This creates a pending storage pool on each member, you then need to do run:
lxc storage create data dir
on any member, to finalise the creation.
Its interesting that you’ve been able to create a storage volume on a pending pool, that doesn’t seem like it should work (it might be a bug).
tomp
May 12, 2021, 7:57am
13
Can you show me the output of:
lxc storage ls
gabriel
May 12, 2021, 8:37am
14
Appreciate the help.
storage pool state:
$ lxc storage ls
+-------+-------------+--------+---------+---------+
| NAME | DESCRIPTION | DRIVER | STATE | USED BY |
+-------+-------------+--------+---------+---------+
| data | | dir | CREATED | 6 |
+-------+-------------+--------+---------+---------+
| local | | dir | CREATED | 1 |
+-------+-------------+--------+---------+---------+
cluster state:
$ lxc list
+---------------+---------+----------------------+------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| c1 | RUNNING | 240.127.0.63 (eth0) | | CONTAINER | 0 | hal-127 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| c2 | RUNNING | 240.128.0.90 (eth0) | | CONTAINER | 0 | hal-128 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| c3 | RUNNING | 240.126.0.197 (eth0) | | CONTAINER | 0 | hal-126 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
| juju-4d0137-0 | RUNNING | 240.126.0.222 (eth0) | | CONTAINER | 0 | hal-126 |
+---------------+---------+----------------------+------+-----------+-----------+----------+
tomp
May 12, 2021, 8:38am
15
Great, so if I’m reading this correctly, you have a storage pool created successfully, you have a profile that references it, and you can create and launch instances using it.
Remind me what the problem is again?
gabriel
May 12, 2021, 8:45am
16
when using the juju to deploy k8s on the lxd cluster, juju does not deploy and reports:
Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
when using the juju controller to deploy k8s on a non-clustered lxd node , juju deploys k8s without problem
tomp
May 12, 2021, 8:47am
17
OK, so I don’t know anything about juju.
Is it possible for you to show me the commands it is using to deploy so we can see where its going wrong?
tomp
May 12, 2021, 8:48am
18
It looks like its specifying the wrong storage pool.
gabriel
May 12, 2021, 8:50am
19
Post lxd
$ sudo snap install juju --classic
$ juju bootstrap localhost
$ juju add-model k8s
$ juju deploy charmed-kubernetes
$ watch -c juju status --color
Every 2.0s: juju status --color hal-126: Wed May 12 20:50:02 2021
Model Controller Cloud/Region Version SLA Timestamp
k8s localhost-localhost localhost/localhost 2.9.0 unsupported 20:50:02+12:00
App Version Status Scale Charm Store Channel Rev OS Message
containerd unknown 0 containerd charmstore stable 119 ubuntu
easyrsa waiting 0/1 easyrsa charmstore stable 373 ubuntu waiting for machine
etcd waiting 0/3 etcd charmstore stable 583 ubuntu waiting for machine
flannel unknown 0 flannel charmstore stable 547 ubuntu
kubeapi-load-balancer waiting 0/1 kubeapi-load-balancer charmstore stable 786 ubuntu waiting for machine
kubernetes-master waiting 0/2 kubernetes-master charmstore stable 990 ubuntu waiting for machine
kubernetes-worker waiting 0/3 kubernetes-worker charmstore stable 757 ubuntu waiting for machine
Unit Workload Agent Machine Public address Ports Message
easyrsa/0 waiting allocating 0 waiting for machine
etcd/0 waiting allocating 1 waiting for machine
etcd/1 waiting allocating 2 waiting for machine
etcd/2 waiting allocating 3 waiting for machine
kubeapi-load-balancer/0 waiting allocating 4 waiting for machine
kubernetes-master/0 waiting allocating 5 waiting for machine
kubernetes-master/1 waiting allocating 6 waiting for machine
kubernetes-worker/0 waiting allocating 7 waiting for machine
kubernetes-worker/1 waiting allocating 8 waiting for machine
kubernetes-worker/2 waiting allocating 9 waiting for machine
Machine State DNS Inst id Series AZ Message
0 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
1 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
2 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
3 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
4 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
5 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
6 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
7 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
8 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
9 down pending focal Failed creating instance record: Failed initialising instance: Failed loading storage pool: No such object
gabriel
May 12, 2021, 8:51am
20
if you scroll across the status messages you’ll see the storage pool related messages
procedure works fine on single non-clustered node
tomp
May 12, 2021, 8:52am
21
@gabriel mmm, I see the error, but I don’t know how we can see what actual commands its running against LXD.
@stgraber are you able to assist with this?
