Lxd-to-incus fails due to unavailable storage pools

1

I ran lxd-to-incus
It completed, but incus wouldn’t start afterwards

# lxd-to-incus
=> Looking for source server
==> Detected: snap package
=> Looking for target server
==> Detected: systemd
=> Connecting to source server
=> Connecting to the target server
=> Checking server versions
==> Source version: 4.0.9
==> Target version: 6.0.0
=> Validating version compatibility
=> Checking that the source server isn't empty
=> Checking that the target server is empty
=> Validating source server configuration

The migration is now ready to proceed.
At this point, the source server and all its instances will be stopped.
Instances will come back online once the migration is complete.
Proceed with the migration? [default=no]: yes
=> Stopping the source server
=> Stopping the target server
=> Wiping the target server
=> Migrating the data
=> Migrating database
=> Writing database patch
=> Cleaning up target paths
=> Starting the target server
Error: Failed to start the target server: Failed to run: systemctl start incus.service incus.socket: exit status 1 (Job for incus.service failed because the control process exited with error code.
See "systemctl status incus.service" and "journalctl -xe" for details.)

# systemctl status incus.service|more
● incus.service - Incus - Daemon
     Loaded: loaded (/lib/systemd/system/incus.service; indirect; vendor preset: enabled)
     Active: activating (start-post) (Result: exit-code) since Sat 2024-04-20 21:04:10 UTC; 7min ago
TriggeredBy: ● incus.socket
    Process: 965699 ExecStart=/opt/incus/lib/systemd/incusd --group incus-admin $INCUS_OPTS --logfile /var/log/incus/incusd.log (code=exited, status=1/FAILURE)
   Main PID: 965699 (code=exited, status=1/FAILURE); Control PID: 965700 (incusd)
      Tasks: 19
     Memory: 29.1M
     CGroup: /system.slice/incus.service
             └─.control
               └─965700 incusd waitready --timeout=600

Apr 20 21:04:10 armadillo systemd[1]: Starting Incus - Daemon...
Apr 20 21:04:10 armadillo incusd[965699]: time="2024-04-20T21:04:10Z" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
Apr 20 21:04:10 armadillo incusd[965699]: time="2024-04-20T21:04:10Z" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored
"
Apr 20 21:04:12 armadillo incusd[965699]: time="2024-04-20T21:04:12Z" level=error msg="Failed mounting storage pool" err="Failed to run: zpool import lxd_containers: ex
it status 1 (cannot import 'lxd_containers': no such pool available)" pool=lxd_containers
Apr 20 21:04:12 armadillo incusd[965699]: time="2024-04-20T21:04:12Z" level=error msg="Failed to start the daemon" err="Failed applying patch \"storage_missing_snapshot
_records\": Unvailable storage pools: [lxd_containers]"
Apr 20 21:04:12 armadillo incusd[965699]: Error: Failed applying patch "storage_missing_snapshot_records": Unvailable storage pools: [lxd_containers]
Apr 20 21:04:12 armadillo systemd[1]: incus.service: Main process exited, code=exited, status=1/FAILURE

# cat /var/log/incus/incusd.log
time="2024-04-20T20:44:10Z" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
time="2024-04-20T20:44:10Z" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
time="2024-04-20T20:44:11Z" level=error msg="Failed mounting storage pool" err="Failed to run: zpool import lxd_containers: exit status 1 (cannot import 'lxd_containers': no such pool available)" pool=lxd_containers
time="2024-04-20T20:44:11Z" level=error msg="Failed to start the daemon" err="Failed applying patch \"storage_missing_snapshot_records\": Unvailable storage pools: [lxd_containers]"
time="2024-04-20T20:54:10Z" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
time="2024-04-20T20:54:10Z" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
time="2024-04-20T20:54:11Z" level=error msg="Failed mounting storage pool" err="Failed to run: zpool import lxd_containers: exit status 1 (cannot import 'lxd_containers': no such pool available)" pool=lxd_containers
time="2024-04-20T20:54:11Z" level=error msg="Failed to start the daemon" err="Failed applying patch \"storage_missing_snapshot_records\": Unvailable storage pools: [lxd_containers]"
time="2024-04-20T21:04:10Z" level=warning msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
time="2024-04-20T21:04:10Z" level=warning msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
time="2024-04-20T21:04:12Z" level=error msg="Failed mounting storage pool" err="Failed to run: zpool import lxd_containers: exit status 1 (cannot import 'lxd_containers': no such pool available)" pool=lxd_containers
time="2024-04-20T21:04:12Z" level=error msg="Failed to start the daemon" err="Failed applying patch \"storage_missing_snapshot_records\": Unvailable storage pools: [lxd_containers]"

After this, there doesn’t appear to be any zpool available after the migration

# zpool list
no pools available

and any incus commands that do more than list help commands get stuck
e.g. - these commands get stuck until Ctrl-C
incus storage list
incus list
incus image list

Running strace, seems to show the commands might be stuck using /var/lib/incus/unix.socket

Also, stopping and starting doesn’t go well either

# systemctl stop incus.socket
# systemctl stop incus
# systemctl start incus.socket
# systemctl start incus

That is stuck on the last command

Any thoughts?
Regards,
Conor

2

What kind of zpool is that lxd_containers zpool?
Was that a loop one (if so, check what’s in /var/lib/incus/disks) or something else?

3

It’s a file zpool - /var/lxd_containers
The /var/lib/incus/disks/ directory exists, but is empty
Creating a link to /var/lxd_containers in /var/lib/incus/disks/ hasn’t changed the incus behaviour that I am observing

4

Can you show ls -lh /var/lxd_containers?

5 
#  ls -lh /var/lxd_containers
-rw-r--r-- 1 root root 94G Apr 20 20:44 /var/lxd_containers
6

Both Incus and LXD use zpool import -d to load zpools from their disks directory, if the pool is located elsewhere, you need to do this manually.

Most likely what happened here is that the full shutdown of LXD has caused an export of the pool and because it’s not a file in /var/lib/incus/disks, it didn’t get imported.

Try zpool import -d /var lxd_containers, if that makes it show up in zpool status then Incus will see it and that should then work fine.

1 Like
7

That solved it. Many thanks.