Unable to create stateless snapshot of a particular VM - 'invalid option [...] "volatile.uuid"'

Incus
, ,
1

Hello,

After migrating from LXD to Incus, I’m unable to create snapshots of a particular VM, eBPF. It reports the following error: Error: Create instance snapshot: Invalid option for volume "eBPF/failtest" option "volatile.uuid" for the command: incus snapshot create eBPF failtest

I’m using a ZFS storage pool on an Arch Linux host, and created a near-identical (also Arch) VM and was able to take a snapshot without issue. I also noticed that snapshots (now removed) had been taken prior to the migration without issue.

I traced Incus and found that it’s not running any ZFS commands to create the snapshot, and checking with debug output doesn’t show anything useful; just that it’s going to create a snapshot and then this same error.

Below is the configuration for the VM which fails:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Archlinux current amd64 (20230803_19:26)
  image.os: Archlinux
  image.release: current
  image.requirements.secureboot: "false"
  image.serial: "20230803_19:26"
  image.type: disk-kvm.img
  image.variant: default
  limits.cpu: "10"
  limits.memory: 12GB
  security.secureboot: "false"
  volatile.base_image: 70921956fc3d5796732c64ec979f43fcbc69e17d0a2970de92d55788bfa0f4d5
  volatile.cloud-init.instance-id: 3d9662f0-cc6b-46bb-823b-db92969bf126
  volatile.eth0.hwaddr: 00:16:3e:6a:1e:90
  volatile.last_state.power: STOPPED
  volatile.uuid: 536da963-952b-4933-b94e-290d4196526d
  volatile.uuid.generation: 536da963-952b-4933-b94e-290d4196526d
  volatile.vsock_id: "2794766878"
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

Below is the configuration for the VM which can make snapshots:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Archlinux current amd64 (20240428_04:18)
  image.os: Archlinux
  image.release: current
  image.requirements.secureboot: "false"
  image.serial: "20240428_04:18"
  image.type: disk-kvm.img
  image.variant: default
  security.secureboot: "false"
  volatile.base_image: 715aeae03c3c4ea4598b8b8011cd3cf7e78ae86c46ea21cd8b50976ce59dbe50
  volatile.cloud-init.instance-id: 1e157760-1f14-4e85-899e-8c4dab30af64
  volatile.eth0.hwaddr: 00:16:3e:f6:54:ba
  volatile.last_state.power: STOPPED
  volatile.uuid: 99ec42c1-1cc2-4e7b-a19b-2bfc38bee366
  volatile.uuid.generation: 99ec42c1-1cc2-4e7b-a19b-2bfc38bee366
  volatile.vsock_id: "1189134851"
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

I’m unsure how exactly volatile.uuid would be preventing creating a snapshot. As a test, I was able to take ZFS snapshots manually like I imagine LXD would, but that’s of course not what I really need - I need to be able to quickly snapshot and restore. I don’t think I can give replication instructions given the uniqueness, but at least would like to know what next steps I could take to get more information about what exactly this error is.

2

Can you show incus storage volume show default virtual-machine/eBPF

1 Like
3

Thanks for helping!

The storage volume is “store” so I did the following:

$incus storage volume show store virtual-machine/eBPF
config:
  volatile.uuid: 8bdb1c74-34d1-439d-9c49-badcae60fcb7 
description: ""
name: eBPF
type: virtual-machine
used_by:
- /1.0/instances/eBPF
location: none
content_type: block
project: default
created_at: 2023-08-03T20:54:19.862425718Z

Doing it with the literal text you gave resulted in “Error: Storage pool not found” so I interpreted it as best I could and assumed this is what was needed.

I did the same for the comparison VM (test) and see there’s no volatile.uuid in the config there! I wonder how this could happen and don’t know what it means to have that set.

4

Yeah, no idea how this could have ever made it there, definitely a very odd case…

This should take care of it though:

incus admin sql global "DELETE FROM storage_volumes_config WHERE key='volatile.uuid'"
2 Likes
5

FWIW, I was affected by the same problem (same error logged when attempting to create snapshots)

brian@db:~$ incus admin sql global "SELECT * from storage_volumes_config WHERE key='volatile.uuid'"
+----+-------------------+---------------+--------------------------------------+
| id | storage_volume_id |      key      |                value                 |
+----+-------------------+---------------+--------------------------------------+
| 1  | 241               | volatile.uuid | c69e970e-e5cc-4197-bb37-a408f461b1f2 |
| 2  | 19807             | volatile.uuid | f07bf977-8097-4cfb-8c75-4458b0829474 |
| 3  | 1                 | volatile.uuid | b927381f-3733-40a0-9f9e-41a6b2a37c7e |
+----+-------------------+---------------+--------------------------------------+

This host has three containers and all three volumes were affected.

Thanks for the solution!