LXD Image vanishing automatically

LXD
#1

Hey team!

I am facing a very strange issue.

I have an automation setup through which an LXD image is mirrored from my LXD Image server onto the VM and then the VM is automatically shut down until I need it for my work. And then I start the VM and then once my work is done, I delete the VM or shut down again.

But recently I observed that the LXD image is not there when I start the VM after a week or so.

I’m not sure how the image just vanishes as I have log of my automation setup and it confirms that the Image was successfully mirrored before shutting down the VM last time.

LXD logs also suggest that the image was not deleted automatically. Not sure what’s happening here.

#2

Can you show the lxc image info output for the image in that VM?

#3

Hi Stephan. Unfortunately I had deleted that VM that day itself before posting this thread. Otherwise I’d have captured this requested info.

I had created another VM that day and so far it has the Image available in it. It’s been 9 days. I usually see that the image vanishes if I restart the VM after 14-15 days.

If it repeats this time then I’ll follow up on this thread with the requested info. Thanks!

#4

Hi @stgraber ,

This happened again. So I took a copy of the logs before the VM got killed.

t=2022-04-08T09:04:26+0000 lvl=info msg="LXD is starting" mode=normal path=/var/snap/lxd/common/lxd version=4.0.9

t=2022-04-08T09:04:26+0000 lvl=info msg="Kernel uid/gid map:"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - u 0 0 4294967295"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - g 0 0 4294967295"

t=2022-04-08T09:04:26+0000 lvl=info msg="Configured LXD uid/gid map:"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - u 0 1000000 1000000000"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - g 0 1000000 1000000000"

t=2022-04-08T09:04:26+0000 lvl=info msg="Kernel features:"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - closing multiple file descriptors efficiently: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - netnsid-based network retrieval: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - pidfds: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - core scheduling: no"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - uevent injection: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - seccomp listener: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - seccomp listener continue syscalls: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - seccomp listener add file descriptors: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - attach to namespaces via pidfds: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - safe native terminal allocation : yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - unprivileged file capabilities: yes"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - cgroup layout: hybrid"

t=2022-04-08T09:04:26+0000 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - shiftfs support: disabled"

t=2022-04-08T09:04:26+0000 lvl=info msg="Initializing local database"

t=2022-04-08T09:04:26+0000 lvl=info msg="Set client certificate to server certificate" fingerprint=263d96d7af35f15ef0161328b043c79b0503335bbf153379d2cc41e4adf1d4af

t=2022-04-08T09:04:26+0000 lvl=info msg="Starting database node" id=1 local=1 role=voter

t=2022-04-08T09:04:26+0000 lvl=info msg="Starting /dev/lxd handler:"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock

t=2022-04-08T09:04:26+0000 lvl=info msg="REST API daemon:"

t=2022-04-08T09:04:26+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket

t=2022-04-08T09:04:26+0000 lvl=info msg="Initializing global database"

t=2022-04-08T09:04:26+0000 lvl=info msg="Connecting to global database"

t=2022-04-08T09:04:26+0000 lvl=info msg="Connected to global database"

t=2022-04-08T09:04:26+0000 lvl=info msg="Initialized global database"

t=2022-04-08T09:04:26+0000 lvl=info msg="Firewall loaded driver" driver=xtables

t=2022-04-08T09:04:26+0000 lvl=info msg="Initializing storage pools"

t=2022-04-08T09:04:26+0000 lvl=info msg="Initializing daemon storage mounts"

t=2022-04-08T09:04:26+0000 lvl=info msg="Loading daemon configuration"

t=2022-04-08T09:04:26+0000 lvl=info msg="Initializing networks"

t=2022-04-08T09:04:26+0000 lvl=info msg="Pruning leftover image files"

t=2022-04-08T09:04:26+0000 lvl=info msg="Done pruning leftover image files"

t=2022-04-08T09:04:26+0000 lvl=info msg="Starting device monitor"

t=2022-04-08T09:04:26+0000 lvl=warn msg="Failed to initialize fanotify, falling back on fsnotify" err="Failed to watch directory \"/dev\": no such device"

t=2022-04-08T09:04:26+0000 lvl=info msg="Started seccomp handler" path=/var/snap/lxd/common/lxd/seccomp.socket

t=2022-04-08T09:04:26+0000 lvl=info msg="Pruning expired images"

t=2022-04-08T09:04:26+0000 lvl=info msg="Done pruning expired images"

t=2022-04-08T09:04:26+0000 lvl=info msg="Pruning expired instance backups"

t=2022-04-08T09:04:26+0000 lvl=info msg="Done pruning expired instance backups"

t=2022-04-08T09:04:26+0000 lvl=info msg="Daemon started"

t=2022-04-08T09:04:26+0000 lvl=info msg="Expiring log files"

t=2022-04-08T09:04:26+0000 lvl=info msg="Done expiring log files"

t=2022-04-08T09:04:26+0000 lvl=info msg="Updating images"

t=2022-04-08T09:04:26+0000 lvl=info msg="Updating instance types"

t=2022-04-08T09:04:26+0000 lvl=info msg="Done updating images"

t=2022-04-08T09:04:26+0000 lvl=info msg="Done updating instance types"

I have another VM on which the Image was recently copied from LXD Image server, here’s the output of the command:
lxc image info im450-v3

Output:

Fingerprint: b41b71eaefd5412996c58dae791fe356b0e5c1a42be5b7254fb2f96c630d9321
Size: 12856.20MB
Architecture: x86_64
Type: container
Public: no
Timestamps:
    Created: 2021/07/20 20:58 UTC
    Uploaded: 2022/04/08 20:28 UTC
    Expires: never
    Last used: 2022/04/08 20:28 UTC
Properties:
    architecture: x86_64
    description: Ubuntu 20.04 LTS server (20210720)
    os: ubuntu
    release: focal
Aliases:
Cached: yes
Auto update: enabled
Source:
    Server: LXD_SERVER_URL
    Protocol: lxd
    Alias: im450-v3
Profiles:
    - default
#5

Ok, so it has the cached flag set which means that 10 days after its last use (see Last used in the timestamps section), the image will automatically be deleted by LXD.

So in the case of that other server you’re looking at, if no new instance is created from the image, it will be deleted on the 18th.

#6

That’s usually not an issue since the image is just a cached entry so a new instance launched using the reference to your image server will just have LXD download the image again and keep it around for another 10 days unless it’s used again since.

#7

Thanks Stephane for sharing this.

My concern is that I don’t want the image to be deleted unless I want it to be deleted or updated. So if I set the cached flag to false then does it solve the issue?

I think setting it false would make the VM download the image every time from LXD Image server whenever there’s a new request.

#8

Set images.remote_cache_expiry to 0 in your server config.

#9

Thanks again Stephane. I’ll try this out. :slight_smile: