However, after several tries, maybe 3 times, LXD will show
Creating test4
Error: Failed instance creation: Locate image 2cfc5a5567b8d74c0986f3d8a77a2a78e58fe22ea9abd2693112031f85afa1a1 in the cluster: image not available on any online node
I fill an issue report in github. Hope these info could help. Thank you.
Base information
Distribution: Ubuntu
Distribution version: 18.04.4 LTS (Bionic Beaver)
Kernel version: Linux machine1 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
LXC version: 4.0.0
LXD version: 4.0.0
Snap revision: 14663
Detailed snap information
name: lxd
summary: System container manager and API
publisher: Canonical✓
store-url: https://snapcraft.io/lxd
contact: https://github.com/lxc/lxd/issues
license: unset
description: |
**LXD is a system container manager**
With LXD you can run hundreds of containers of a variety of Linux
distributions, apply resource limits, pass in directories, USB devices
or GPUs and setup any network and storage you want.
LXD containers are lightweight, secure by default and a great
alternative to running Linux virtual machines.
**Run any Linux distribution you want**
Pre-made images are available for Ubuntu, Alpine Linux, ArchLinux,
CentOS, Debian, Fedora, Gentoo, OpenSUSE and more.
A full list of available images can be found here: https://images.linuxcontainers.org
Can't find the distribution you want? It's easy to make your own images too, either using our
`distrobuilder` tool or by assembling your own image tarball by hand.
**Containers at scale**
LXD is network aware and all interactions go through a simple REST API,
making it possible to remotely interact with containers on remote
systems, copying and moving them as you wish.
Want to go big? LXD also has built-in clustering support,
letting you turn dozens of servers into one big LXD server.
**Configuration options**
Supported options for the LXD snap (`snap set lxd KEY=VALUE`):
- criu.enable: Enable experimental live-migration support [default=false]
- daemon.debug: Increases logging to debug level [default=false]
- daemon.group: Group of users that can interact with LXD [default=lxd]
- ceph.builtin: Use snap-specific ceph configuration [default=false]
- openvswitch.builtin: Run a snap-specific OVS daemon [default=false]
Documentation: https://lxd.readthedocs.io
commands:
- lxd.benchmark
- lxd.buginfo
- lxd.check-kernel
- lxd.lxc
- lxd
- lxd.migrate
services:
lxd.activate: oneshot, enabled, inactive
lxd.daemon: simple, enabled, active
snap-id: J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking: latest/stable
refresh-date: today at 15:56 CST
channels:
latest/stable: 4.0.0 2020-04-17 (14663) 62MB -
latest/candidate: 4.0.0 2020-04-17 (14663) 62MB -
latest/beta: ↑
latest/edge: git-0c71615 2020-04-17 (14671) 62MB -
4.0/stable: 4.0.0 2020-04-17 (14663) 62MB -
4.0/candidate: 4.0.0 2020-04-17 (14663) 62MB -
4.0/beta: ↑
4.0/edge: ↑
3.23/stable: 3.23 2020-03-30 (14133) 70MB -
3.23/candidate: 3.23 2020-03-30 (14133) 70MB -
3.23/beta: ↑
3.23/edge: ↑
3.22/stable: 3.22 2020-03-18 (13901) 70MB -
3.22/candidate: 3.22 2020-03-19 (13911) 70MB -
3.22/beta: ↑
3.22/edge: ↑
3.21/stable: 3.21 2020-02-24 (13522) 69MB -
3.21/candidate: 3.21 2020-03-04 (13588) 69MB -
3.21/beta: ↑
3.21/edge: ↑
3.20/stable: 3.20 2020-02-06 (13300) 69MB -
3.20/candidate: 3.20 2020-02-06 (13300) 69MB -
3.20/beta: ↑
3.20/edge: ↑
3.19/stable: 3.19 2020-01-27 (13162) 67MB -
3.19/candidate: 3.19 2020-01-27 (13162) 67MB -
3.19/beta: ↑
3.19/edge: ↑
3.18/stable: 3.18 2019-12-02 (12631) 57MB -
3.18/candidate: 3.18 2019-12-02 (12631) 57MB -
3.18/beta: ↑
3.18/edge: ↑
3.0/stable: 3.0.4 2019-10-10 (11348) 55MB -
3.0/candidate: 3.0.4 2019-10-10 (11348) 55MB -
3.0/beta: ↑
3.0/edge: git-81b81b9 2019-10-10 (11362) 55MB -
2.0/stable: 2.0.11 2019-10-10 (8023) 28MB -
2.0/candidate: 2.0.11 2019-10-10 (8023) 28MB -
2.0/beta: ↑
2.0/edge: git-160221d 2020-01-13 (12854) 27MB -
installed: 4.0.0 (14663) 62MB -
+--------------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+--------------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
| ubuntu-18.04 | 2cfc5a5567b8 | no | ubuntu 18.04 LTS amd64 (release) (20200407) | x86_64 | CONTAINER | 179.02MB | Apr 17, 2020 at 2:26pm (UTC) |
+--------------+--------------+--------+---------------------------------------------+--------------+-----------+----------+------------------------------+
Storage pools
+-------+-------------+--------+---------+---------+
| NAME | DESCRIPTION | DRIVER | STATE | USED BY |
+-------+-------------+--------+---------+---------+
| local | | dir | CREATED | 3 |
+-------+-------------+--------+---------+---------+
Networks
+---------+----------+---------+-------------+---------+---------+
| NAME | TYPE | MANAGED | DESCRIPTION | USED BY | STATE |
+---------+----------+---------+-------------+---------+---------+
| eth0 | physical | NO | | 0 | |
+---------+----------+---------+-------------+---------+---------+
| lxdfan0 | bridge | YES | | 6 | CREATED |
+---------+----------+---------+-------------+---------+---------+
Projects
+-------------------+--------+----------+-----------------+---------+
| NAME | IMAGES | PROFILES | STORAGE VOLUMES | USED BY |
+-------------------+--------+----------+-----------------+---------+
| default (current) | YES | YES | YES | 8 |
+-------------------+--------+----------+-----------------+---------+
Profiles
+---------+---------+
| NAME | USED BY |
+---------+---------+
| default | 6 |
+---------+---------+
After several times, maybe 3~4 times, I get the erro :
Error: Failed instance creation: Locate image 2cfc5a5567b8d74c0986f3d8a77a2a78e58fe22ea9abd2693112031f85afa1a1 in the cluster: image not available on any online node
I’ve tried several times, the isuue is not self-recovery.
Then I try the command in other node, it’s the same as in node1.
Steps to reproduce
Step one
Create a cluster.
Step two
Launch instance one after aonther immediately.
Information to attach
Any relevant kernel output (dmesg)
Container log (lxc info NAME --show-log)
Container configuration (lxc config show NAME --expanded)
Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
holytiny@machine1:~$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
t=2020-04-17T23:27:01+0800 lvl=info msg="LXD 4.0.0 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2020-04-17T23:27:01+0800 lvl=info msg="Kernel uid/gid map:"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - u 0 0 4294967295"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - g 0 0 4294967295"
t=2020-04-17T23:27:01+0800 lvl=info msg="Configured LXD uid/gid map:"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - u 0 1000000 1000000000"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - g 0 1000000 1000000000"
t=2020-04-17T23:27:01+0800 lvl=info msg="Kernel features:"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - netnsid-based network retrieval: no"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - uevent injection: no"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - seccomp listener: no"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - seccomp listener continue syscalls: no"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - unprivileged file capabilities: yes"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - cgroup layout: hybrid"
t=2020-04-17T23:27:01+0800 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - shiftfs support: disabled"
t=2020-04-17T23:27:01+0800 lvl=info msg="Initializing local database"
t=2020-04-17T23:27:01+0800 lvl=info msg="Starting /dev/lxd handler:"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2020-04-17T23:27:01+0800 lvl=info msg="REST API daemon:"
t=2020-04-17T23:27:01+0800 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2020-04-17T23:27:01+0800 lvl=info msg=" - binding TCP socket" socket=172.26.140.101:8443
t=2020-04-17T23:27:01+0800 lvl=info msg="Initializing global database"
t=2020-04-17T23:27:01+0800 lvl=warn msg="Dqlite: server unavailable err=failed to establish network connection: 503 Service Unavailable address=172.26.140.101:8443 attempt=0"
t=2020-04-17T23:27:01+0800 lvl=info msg="Firewall loaded driver \"xtables\""
t=2020-04-17T23:27:01+0800 lvl=info msg="Initializing storage pools"
t=2020-04-17T23:27:01+0800 lvl=info msg="Initializing daemon storage mounts"
t=2020-04-17T23:27:01+0800 lvl=info msg="Initializing networks"
t=2020-04-17T23:27:01+0800 lvl=info msg="Pruning leftover image files"
t=2020-04-17T23:27:01+0800 lvl=info msg="Done pruning leftover image files"
t=2020-04-17T23:27:01+0800 lvl=info msg="Loading daemon configuration"
t=2020-04-17T23:27:02+0800 lvl=info msg="Pruning expired images"
t=2020-04-17T23:27:02+0800 lvl=info msg="Done pruning expired images"
t=2020-04-17T23:27:02+0800 lvl=info msg="Pruning expired instance backups"
t=2020-04-17T23:27:02+0800 lvl=info msg="Done pruning expired instance backups"
t=2020-04-17T23:27:02+0800 lvl=info msg="Updating instance types"
t=2020-04-17T23:27:02+0800 lvl=info msg="Done updating instance types"
t=2020-04-17T23:27:02+0800 lvl=info msg="Expiring log files"
t=2020-04-17T23:27:02+0800 lvl=info msg="Updating images"
t=2020-04-17T23:27:02+0800 lvl=info msg="Done expiring log files"
t=2020-04-17T23:27:02+0800 lvl=info msg="Done updating images"
t=2020-04-17T23:27:02+0800 lvl=info msg="Starting container" action=start created=2020-04-17T22:31:45+0800 ephemeral=false name=test1 project=default stateful=false used=2020-04-17T22:31:59+0800
t=2020-04-17T23:27:02+0800 lvl=info msg="Started container" action=start created=2020-04-17T22:31:45+0800 ephemeral=false name=test1 project=default stateful=false used=2020-04-17T22:31:59+0800
t=2020-04-17T23:27:02+0800 lvl=info msg="Starting container" action=start created=2020-04-17T22:43:05+0800 ephemeral=false name=test5 project=default stateful=false used=2020-04-17T22:43:19+0800
t=2020-04-17T23:27:03+0800 lvl=info msg="Started container" action=start created=2020-04-17T22:43:05+0800 ephemeral=false name=test5 project=default stateful=false used=2020-04-17T22:43:19+0800
t=2020-04-17T23:27:08+0800 lvl=info msg="Refreshing forkdns peers for lxdfan0"
t=2020-04-17T23:27:30+0800 lvl=warn msg="Dqlite client proxy Unix -> TLS: read unix @->@00175: use of closed network connection"
t=2020-04-17T23:27:30+0800 lvl=warn msg="Dqlite server proxy Unix -> TLS: read unix @->@00173: use of closed network connection"
t=2020-04-17T23:27:30+0800 lvl=warn msg="Dqlite client proxy Unix -> TLS: read unix @->@00174: use of closed network connection"
t=2020-04-17T23:27:31+0800 lvl=warn msg="Failed to get events from node 172.26.140.102:8443: Unable to connect to: 172.26.140.102:8443"
t=2020-04-17T23:27:32+0800 lvl=warn msg="Failed to get events from node 172.26.140.102:8443: Unable to connect to: 172.26.140.102:8443"
t=2020-04-17T23:28:07+0800 lvl=warn msg="Dqlite server proxy Unix -> TLS: read unix @->@00173: use of closed network connection"
t=2020-04-17T23:28:07+0800 lvl=warn msg="Dqlite server proxy Unix -> TLS: read unix @->@00173: use of closed network connection"
t=2020-04-17T23:28:07+0800 lvl=warn msg="Dqlite client proxy Unix -> TLS: read unix @->@00233: use of closed network connection"
t=2020-04-17T23:28:07+0800 lvl=warn msg="Failed to get events from node 172.26.140.103:8443: Unable to connect to: 172.26.140.103:8443"
t=2020-04-17T23:28:08+0800 lvl=warn msg="Failed to get events from node 172.26.140.103:8443: Unable to connect to: 172.26.140.103:8443"
t=2020-04-17T23:28:09+0800 lvl=warn msg="Failed to get events from node 172.26.140.103:8443: Unable to connect to: 172.26.140.103:8443"
t=2020-04-17T23:28:22+0800 lvl=warn msg="Dqlite client proxy Unix -> TLS: read unix @->@00235: use of closed network connection"
t=2020-04-17T23:28:22+0800 lvl=warn msg="Dqlite server proxy Unix -> TLS: read unix @->@00173: use of closed network connection"
t=2020-04-17T23:28:22+0800 lvl=warn msg="Dqlite server proxy Unix -> TLS: read unix @->@00173: use of closed network connection"
t=2020-04-17T23:28:23+0800 lvl=warn msg="Failed to get events from node 172.26.140.105:8443: Unable to connect to: 172.26.140.105:8443"
t=2020-04-17T23:28:24+0800 lvl=warn msg="Failed to get events from node 172.26.140.105:8443: Unable to connect to: 172.26.140.105:8443"
t=2020-04-17T23:28:25+0800 lvl=warn msg="Failed to get events from node 172.26.140.105:8443: Unable to connect to: 172.26.140.105:8443"
holytiny@machine1:~$ ping 172.26.140.105
PING 172.26.140.105 (172.26.140.105) 56(84) bytes of data.
64 bytes from 172.26.140.105: icmp_seq=1 ttl=64 time=0.173 ms
64 bytes from 172.26.140.105: icmp_seq=2 ttl=64 time=0.122 ms
^C
--- 172.26.140.105 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.122/0.147/0.173/0.028 ms
Output of the client with --debug
Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)
lxc image info ubuntu-18.04
Fingerprint: 2cfc5a5567b8d74c0986f3d8a77a2a78e58fe22ea9abd2693112031f85afa1a1
Size: 179.02MB
Architecture: x86_64
Type: container
Public: no
Timestamps:
Created: 2020/04/07 00:00 UTC
Uploaded: 2020/04/17 14:26 UTC
Expires: 2023/04/26 00:00 UTC
Last used: 2020/04/17 14:43 UTC
Properties:
architecture: amd64
description: ubuntu 18.04 LTS amd64 (release) (20200407)
label: release
os: ubuntu
release: bionic
serial: 20200407
type: squashfs
version: 18.04
Aliases:
- ubuntu-18.04
Cached: no
Auto update: disabled
Source:
Server: https://cloud-images.ubuntu.com/releases
Protocol: simplestreams
Alias: 18.04
Profiles:
- default
Oh, sorry I forgot to append this info.
Because the nodes of the cluster remained offline after updating to 4.0, I executed the command below before I did my test.
