Error launching container in project after upgrade to 4.9

1

LXD cluster upgraded to 4.9 and it cannot launch a container from ubuntu:20.04 image in a specific project, but it can launch a container which is based on image created from local containers in that project. Solved the problem by deleting the image from the project and than creating a container from ubuntu:20.04, so it downloaded the ubuntu fresh image and worked fine. One more caveat is rootfs download of ubuntu:20.04 is very slow on aws instances. The metadata download is fast in MB/s but for rootfs is around 300 kB/s.

Following is the output.

$ lxc launch ubuntu:20.04 test-1 --project testproj
Creating test-1
Error: Failed instance creation: Failed creating instance record: Failed initialising instance: Invalid devices: Failed detecting root disk device: No root device could be found

$ lxc project show testproj
description: ""
config:
  features.images: "true"
  features.profiles: "false"
  features.storage.volumes: "false"
name: testproj
used_by:
- /1.0/images/27782dbdcd2613dbe421fec9dc1c63096eba6c49abf133ae1d0471b0e8f00daa?project=testproj
- /1.0/images/295380c67ce8b70177d5755ad461291ca63f00497879e66ee91468bb4d9e7882?project=testproj
- /1.0/images/2e8d975e1ac663922e877c3b1cc71587af56dd8c17fb9cce7e10c61efafb338a?project=testproj
- /1.0/images/95d4230091724e8d774b6c9b2053e70bebee51570afb896f6157bf8b99625e59?project=testproj
- /1.0/images/e0c3495ffd489748aa5151628fa56619e6143958f041223cb4970731ef939cb6?project=testproj
- /1.0/instances/test-tip-1?project=testproj

$ lxc storage info local
info:
  description: ""
  driver: zfs
  name: local
  space used: 11.19GB
  total space: 528.42GB
used by:
  images:
  - 95d4230091724e8d774b6c9b2053e70bebee51570afb896f6157bf8b99625e59?project=testproj
  - e0c3495ffd489748aa5151628fa56619e6143958f041223cb4970731ef939cb6?project=testproj
  - e4c8dd5922cefe65861e2b2b96026de2cae16b0a6d27533760161407891cb0d0
  instances:
  - test-tip-2?project=testproj
  - logs-1
  - pypi-1
  - test
  profiles:
  - default

$ lxc image delete e0c3495ffd48 --project testproj

$ lxc launch ubuntu:20.04 test-1 --project testproj
Creating test-1
Retrieving image: rootfs: 10% (325.47kB/s)
Starting test-1
2

Yes I believe there is a known issue that can occur when using clustering that can cause images to become dissociated to a profile, leading to the error you see when using the image.

We’ve not been able to isolate what causes this, but we suspect it is to do with how in a cluster, the different nodes can race to update the image concurrently. This is due to be looked at in the current cycle by reworking the image update logic to assign a single node to update specific images.

This is more likely to occur after an upgrade because restarting LXD will trigger the image update check process.