Lxd recover not working on migrated zpool

LXD
, ,
1

Hello,

Following a hardware problem with my server, I would like to import my containers on another dataset.

To migrate my data I first cut lxd by doing a snap stop lxd. I then migrated my data from my rpool-old/lxd dataset to rpool/lxd. Then I tried to do a zfs recover but it doesn’t work.

root@srv-home:~ # lxd recover
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
This LXD server currently has the following storage pools:
 - default (backend="zfs", source="rpool/lxd")
Would you like to recover another storage pool? (yes/no) [default=no]: 
The recovery process will be scanning the following storage pools:
 - EXISTING: "default" (backend="zfs", source="rpool/lxd")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
Error: Failed validation request: Failed checking volumes on pool "default": Instance "home-assistant" in project "default" has a different instance type in its backup file ("")

How can I restore my containers?

Thank you in advance for your help.

2

Can you show the backup.yaml for that instance (it’s in its zfs dataset)?
Looks like we may be dealing with a broken/missing metadata file here.

3

I have other errors that appear if I relaunch the command:

WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
This LXD server currently has the following storage pools:
 - default (backend="zfs", source="rpool/lxd")
Would you like to recover another storage pool? (yes/no) [default=no]: 
The recovery process will be scanning the following storage pools:
 - EXISTING: "default" (backend="zfs", source="rpool/lxd")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
Error: Failed validation request: Failed checking volumes on pool "default": Instance "secret-management" in project "default" has snapshot inconsistency: Snapshot count in backup config and storage device are different: Backup snapshots mismatch

root@srv-home:~ # lxd recover
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
This LXD server currently has the following storage pools:
 - default (backend="zfs", source="rpool/lxd")
Would you like to recover another storage pool? (yes/no) [default=no]: 
The recovery process will be scanning the following storage pools:
 - EXISTING: "default" (backend="zfs", source="rpool/lxd")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
Error: Failed validation request: Failed checking volumes on pool "default": Instance "rocketchat" in project "default" has snapshot inconsistency: Snapshot count in backup config and storage device are different:  Backup snapshots mismatch

If I mount the file system using these commands I get my backup.yaml files every time:

root@srv-home:~ # zfs set mountpoint=/var/snap/lxd/common/lxd/storage-pools/default/containers/rocketchat rpool/lxd/containers/rocketchat
root@srv-home:~ # zfs mount rpool/lxd/containers/rocketchat
root@srv-home:~ # ll /var/snap/lxd/common/lxd/storage-pools/default/containers/rocketchat
total 26K
-r--------  1 root    root     12K Dec 26 02:10 backup.yaml
-rw-r--r--  1 root    root    1.1K Nov 13  2019 metadata.yaml
drwxr-xr-x 22 1000000 1000000   22 Nov 13  2019 rootfs
drwxr-xr-x  2 root    root       7 Nov 13  2019 templates
4

Can you show the contents of backup.yaml and metadata.yaml please.

5

I guess I have to remove the snapshot part? These snapshots are deleted by another system. Would it be possible to add a function that ignores the snapshots during import?

container:
  architecture: x86_64
  config:
    boot.autostart: "true"
    boot.autostart.delay: "10"
    boot.autostart.priority: "70"
    image.architecture: amd64
    image.description: ubuntu 18.04 LTS amd64 (release) (20191113)
    image.label: release
    image.os: ubuntu
    image.release: bionic
    image.serial: "20191113"
    image.type: squashfs
    image.version: "18.04"
    security.nesting: "true"
    volatile.base_image: 96b124c2a0cbc57e6b2cc110105781c3123b26587acdf145ec32e549d4503e8d
    volatile.eth0.host_name: veth09a0f450
    volatile.eth0.hwaddr: 00:16:3e:0b:35:72
    volatile.eth0.name: eth0
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
    volatile.uuid: 7f1a477a-9944-4d26-b9f8-6871bcb54f6b
  devices: {}
  ephemeral: false
  profiles:
  - default
  - eth0
  - mem-4GB
  stateful: false
  description: ""
  created_at: 2019-11-14T23:24:06.265771939+01:00
  expanded_config:
    boot.autostart: "true"
    boot.autostart.delay: "10"
    boot.autostart.priority: "70"
    image.architecture: amd64
    image.description: ubuntu 18.04 LTS amd64 (release) (20191113)
    image.label: release
    image.os: ubuntu
    image.release: bionic
    image.serial: "20191113"
    image.type: squashfs
    image.version: "18.04"
    limits.memory: 4GB
    security.nesting: "true"
    volatile.base_image: 96b124c2a0cbc57e6b2cc110105781c3123b26587acdf145ec32e549d4503e8d
    volatile.eth0.host_name: veth09a0f450
    volatile.eth0.hwaddr: 00:16:3e:0b:35:72
    volatile.eth0.name: eth0
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
    volatile.uuid: 7f1a477a-9944-4d26-b9f8-6871bcb54f6b
  expanded_devices:
    eth0:
      nictype: bridged
      parent: br0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: rocketchat
  status: Stopped
  status_code: 102
  last_used_at: 2021-12-26T02:09:37.266170489+01:00
  location: none
  type: container
snapshots:
- expires_at: 0001-01-01T00:00:00Z
  architecture: x86_64
  config:
    image.architecture: amd64
    image.description: ubuntu 18.04 LTS amd64 (release) (20191113)
    image.label: release
    image.os: ubuntu
    image.release: bionic
    image.serial: "20191113"
    image.type: squashfs
    image.version: "18.04"
    volatile.base_image: 96b124c2a0cbc57e6b2cc110105781c3123b26587acdf145ec32e549d4503e8d
    volatile.eth0.host_name: veth16fdfa94
    volatile.eth0.hwaddr: 00:16:3e:0b:35:72
    volatile.eth0.name: eth0
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
  created_at: 2019-11-14T23:34:03.264283631+01:00
  devices: {}
  ephemeral: false
  expanded_config:
    image.architecture: amd64
    image.description: ubuntu 18.04 LTS amd64 (release) (20191113)
    image.label: release
    image.os: ubuntu
    image.release: bionic
    image.serial: "20191113"
    image.type: squashfs
    image.version: "18.04"
    limits.memory: 4GB
    volatile.base_image: 96b124c2a0cbc57e6b2cc110105781c3123b26587acdf145ec32e549d4503e8d
    volatile.eth0.host_name: veth16fdfa94
    volatile.eth0.hwaddr: 00:16:3e:0b:35:72
    volatile.eth0.name: eth0
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
  expanded_devices:
    eth0:
      nictype: bridged
      parent: br0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  last_used_at: 0001-01-01T00:00:00Z
  name: before_ansible
  profiles:
  - default
  - eth0
  - mem-4GB
  stateful: false
  size: -1
- expires_at: 0001-01-01T00:00:00Z
  architecture: x86_64
  config:
    boot.autostart: "true"
    boot.autostart.delay: "10"
    boot.autostart.priority: "70"
    image.architecture: amd64
    image.description: ubuntu 18.04 LTS amd64 (release) (20191113)
    image.label: release
    image.os: ubuntu
    image.release: bionic
    image.serial: "20191113"
    image.type: squashfs
    image.version: "18.04"
    security.nesting: "true"
    volatile.base_image: 96b124c2a0cbc57e6b2cc110105781c3123b26587acdf145ec32e549d4503e8d
    volatile.eth0.host_name: veth6669ea8f
    volatile.eth0.hwaddr: 00:16:3e:0b:35:72
    volatile.eth0.name: eth0
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
    volatile.uuid: 7f1a477a-9944-4d26-b9f8-6871bcb54f6b
  created_at: 2021-11-17T15:52:04.378918815+01:00
  devices: {}
  ephemeral: false
  expanded_config:
    boot.autostart: "true"
    boot.autostart.delay: "10"
    boot.autostart.priority: "70"
    image.architecture: amd64
    image.description: ubuntu 18.04 LTS amd64 (release) (20191113)
    image.label: release
    image.os: ubuntu
    image.release: bionic
    image.serial: "20191113"
    image.type: squashfs
    image.version: "18.04"
    limits.memory: 4GB
    security.nesting: "true"
    volatile.base_image: 96b124c2a0cbc57e6b2cc110105781c3123b26587acdf145ec32e549d4503e8d
    volatile.eth0.host_name: veth6669ea8f
    volatile.eth0.hwaddr: 00:16:3e:0b:35:72
    volatile.eth0.name: eth0
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
    volatile.uuid: 7f1a477a-9944-4d26-b9f8-6871bcb54f6b
  expanded_devices:
    eth0:
      nictype: bridged
      parent: br0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  last_used_at: 0001-01-01T00:00:00Z
  name: before_upgrade_to_4.x
  profiles:
  - default
  - eth0
  - mem-4GB
  stateful: false
  size: -1
- expires_at: 0001-01-01T00:00:00Z
  architecture: x86_64
  config:
    boot.autostart: "true"
    boot.autostart.delay: "10"
    boot.autostart.priority: "70"
    image.architecture: amd64
    image.description: ubuntu 18.04 LTS amd64 (release) (20191113)
    image.label: release
    image.os: ubuntu
    image.release: bionic
    image.serial: "20191113"
    image.type: squashfs
    image.version: "18.04"
    security.nesting: "true"
    volatile.base_image: 96b124c2a0cbc57e6b2cc110105781c3123b26587acdf145ec32e549d4503e8d
    volatile.eth0.host_name: veth6669ea8f
    volatile.eth0.hwaddr: 00:16:3e:0b:35:72
    volatile.eth0.name: eth0
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
    volatile.uuid: 7f1a477a-9944-4d26-b9f8-6871bcb54f6b
  created_at: 2021-11-17T17:51:35.906400067+01:00
  devices: {}
  ephemeral: false
  expanded_config:
    boot.autostart: "true"
    boot.autostart.delay: "10"
    boot.autostart.priority: "70"
    image.architecture: amd64
    image.description: ubuntu 18.04 LTS amd64 (release) (20191113)
    image.label: release
    image.os: ubuntu
    image.release: bionic
    image.serial: "20191113"
    image.type: squashfs
    image.version: "18.04"
    limits.memory: 4GB
    security.nesting: "true"
    volatile.base_image: 96b124c2a0cbc57e6b2cc110105781c3123b26587acdf145ec32e549d4503e8d
    volatile.eth0.host_name: veth6669ea8f
    volatile.eth0.hwaddr: 00:16:3e:0b:35:72
    volatile.eth0.name: eth0
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
    volatile.uuid: 7f1a477a-9944-4d26-b9f8-6871bcb54f6b
  expanded_devices:
    eth0:
      nictype: bridged
      parent: br0
      type: nic
    root:
      path: /
      pool: default
      type: disk
  last_used_at: 0001-01-01T00:00:00Z
  name: before_upgrade_20.04
  profiles:
  - default
  - eth0
  - mem-4GB
  stateful: false
  size: -1
pool:
  config:
    source: rpool/lxd
    volatile.initial_source: rpool/lxd
    zfs.pool_name: rpool/lxd
  description: ""
  name: default
  driver: zfs
  used_by: []
  status: Created
  locations:
  - none
volume:
  config: {}
  description: ""
  name: rocketchat
  type: container
  used_by: []
  location: none
  content_type: filesystem
6

I just realized that I didn’t send the metadata file.
After a lot of trouble and testing I didn’t manage to put the backup.yaml file in order.

What I did is that I took the essential information from the backup.yaml file. I then created all the containers, then I deleted the dataset and I replaced it with my backup. Then modified all the container configurations by replacing them with the config I made.

7

Ah yes so that would be a problem, as the recover system checks that what is contained within the backup.yaml matches what is seen on disk. If it doesn’t we would expect manual intervention (as you have done) to check and correct the issue rather than LXD silently doing a partial recovery or recovering database entries that do not exist on the storage device.

As a general position it is advisable not to have an external system remove snapshots that LXD created and where LXD doesn’t know about their removal as this will cause issues (such as this) later on. It is better to remove snapshots via LXD commands if they were created by LXD.

1 Like