HELP ! "Required tool 'zpool' is missing"

LXD
1

I have just screwed up by trying to install Ubuntu-core on a container rather than a VM (Clearly not enough coffee this morning!) . What has happened? Any recommendations for recover? I’d prefer not to have lost all my containers!.

Now LXD will not start, the journalctl log shows :

Apr 12 09:06:43 ash lxd.daemon[113042]: 1: fd: 6: name=systemd

Apr 12 09:06:43 ash lxd.daemon[113042]: 2: fd: 7: cpuset

Apr 12 09:06:43 ash lxd.daemon[113042]: 3: fd: 8: cpu,cpuacct

Apr 12 09:06:43 ash lxd.daemon[113042]: 4: fd: 9: rdma

Apr 12 09:06:43 ash lxd.daemon[113042]: 5: fd: 10: hugetlb

Apr 12 09:06:43 ash lxd.daemon[113042]: 6: fd: 11: freezer

Apr 12 09:06:43 ash lxd.daemon[113042]: 7: fd: 12: memory

Apr 12 09:06:43 ash lxd.daemon[113042]: 8: fd: 13: pids

Apr 12 09:06:43 ash lxd.daemon[113042]: 9: fd: 14: net_cls,net_prio

Apr 12 09:06:43 ash lxd.daemon[113042]: 10: fd: 15: perf_event

Apr 12 09:06:43 ash lxd.daemon[113042]: 11: fd: 16: devices

Apr 12 09:06:43 ash lxd.daemon[113042]: 12: fd: 17: blkio

Apr 12 09:06:43 ash lxd.daemon[113042]: Kernel supports pidfds

Apr 12 09:06:43 ash lxd.daemon[113042]: Kernel does not support swap accounting

Apr 12 09:06:43 ash lxd.daemon[113042]: api_extensions:

Apr 12 09:06:43 ash lxd.daemon[113042]: - cgroups

Apr 12 09:06:43 ash lxd.daemon[113042]: - sys_cpu_online

Apr 12 09:06:43 ash lxd.daemon[113042]: - proc_cpuinfo

Apr 12 09:06:43 ash lxd.daemon[113042]: - proc_diskstats

Apr 12 09:06:43 ash lxd.daemon[113042]: - proc_loadavg

Apr 12 09:06:43 ash lxd.daemon[113042]: - proc_meminfo

Apr 12 09:06:43 ash lxd.daemon[113042]: - proc_stat

Apr 12 09:06:43 ash lxd.daemon[113042]: - proc_swaps

Apr 12 09:06:43 ash lxd.daemon[113042]: - proc_uptime

Apr 12 09:06:43 ash lxd.daemon[113042]: - proc_slabinfo

Apr 12 09:06:43 ash lxd.daemon[113042]: - shared_pidns

Apr 12 09:06:43 ash lxd.daemon[113042]: - cpuview_daemon

Apr 12 09:06:43 ash lxd.daemon[113042]: - loadavg_daemon

Apr 12 09:06:43 ash lxd.daemon[113042]: - pidfds

Apr 12 09:06:44 ash lxd.daemon[112906]: => Starting LXD

Apr 12 09:06:44 ash lxd.daemon[113053]: time=“2024-04-12T09:06:44Z” level=warning msg=" - Couldn’t find the CGroup blkio.weight, disk priority will be ignored"

Apr 12 09:06:44 ash lxd.daemon[113053]: time=“2024-04-12T09:06:44Z” level=warning msg=" - Couldn’t find the CGroup memory swap accounting, swap limits will be ignored"

Apr 12 09:06:45 ash lxd.daemon[113053]: time=“2024-04-12T09:06:45Z” level=error msg=“Failed loading storage pool” err=“Required tool ‘zpool’ is missing” pool=default

Apr 12 09:06:45 ash lxd.daemon[113053]: time=“2024-04-12T09:06:45Z” level=error msg=“Failed to start the daemon” err="Failed applying patch "storage_move_custom_iso_block_volumes_v2": Failed loading pool >

Apr 12 09:06:45 ash lxd.daemon[113053]: Error: Failed applying patch “storage_move_custom_iso_block_volumes_v2”: Failed loading pool “default”: Required tool ‘zpool’ is missing

Apr 12 09:06:46 ash lxd.daemon[112906]: Killed

Apr 12 09:06:46 ash lxd.daemon[112906]: => LXD failed to start

Apr 12 09:06:46 ash systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE

Apr 12 09:06:46 ash systemd[1]: snap.lxd.daemon.service: Failed with result ‘exit-code’.

Apr 12 09:06:46 ash systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 394.

Apr 12 09:06:46 ash systemd[1]: Stopped Service for snap application lxd.daemon.

Apr 12 09:06:46 ash systemd[1]: Started Service for snap application lxd.daemon.

Apr 12 09:06:46 ash lxd.daemon[113138]: => Preparing the system (28155)

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Loading snap configuration

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Setting up mntns symlink (mnt:[4026533642])

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Setting up kmod wrapper

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Preparing /boot

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Preparing a clean copy of /run

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Preparing /run/bin

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Preparing a clean copy of /etc

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Preparing a clean copy of /usr/share/misc

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Setting up ceph configuration

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Setting up LVM configuration

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Setting up OVN configuration

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Rotating logs

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Unsupported ZFS version (0.8)

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Escaping the systemd cgroups

Apr 12 09:06:46 ash lxd.daemon[113138]: ====> Detected cgroup V1

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Escaping the systemd process resource limits

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Enabling LXD UI

Apr 12 09:06:46 ash lxd.daemon[113138]: ==> Exposing LXD documentation

Apr 12 09:06:46 ash lxd.daemon[113042]: Running destructor lxcfs_exit

Apr 12 09:06:46 ash lxd.daemon[113138]: => Starting LXCFS

Apr 12 09:06:46 ash lxd.daemon[113273]: Starting LXCFS at lxcfs

Apr 12 09:06:46 ash lxd.daemon[113273]: Running constructor lxcfs_init to reload liblxcfs

Apr 12 09:06:46 ash lxd.daemon[113273]: mount namespace: 4

Apr 12 09:06:46 ash lxd.daemon[113273]: hierarchies:

Apr 12 09:06:46 ash lxd.daemon[113273]: 0: fd: 5:

Apr 12 09:06:46 ash lxd.daemon[113273]: 1: fd: 6: name=systemd

Apr 12 09:06:46 ash lxd.daemon[113273]: 2: fd: 7: cpuset

Apr 12 09:06:46 ash lxd.daemon[113273]: 3: fd: 8: cpu,cpuacct

Apr 12 09:06:46 ash lxd.daemon[113273]: 4: fd: 9: rdma

Apr 12 09:06:46 ash lxd.daemon[113273]: 5: fd: 10: hugetlb

Apr 12 09:06:46 ash lxd.daemon[113273]: 6: fd: 11: freezer

Apr 12 09:06:46 ash lxd.daemon[113273]: 7: fd: 12: memory

Apr 12 09:06:46 ash lxd.daemon[113273]: 8: fd: 13: pids

Apr 12 09:06:46 ash lxd.daemon[113273]: 9: fd: 14: net_cls,net_prio

Apr 12 09:06:46 ash lxd.daemon[113273]: 10: fd: 15: perf_event

Apr 12 09:06:46 ash lxd.daemon[113273]: 11: fd: 16: devices

Apr 12 09:06:46 ash lxd.daemon[113273]: 12: fd: 17: blkio

Apr 12 09:06:46 ash lxd.daemon[113273]: Kernel supports pidfds

Apr 12 09:06:46 ash lxd.daemon[113273]: Kernel does not support swap accounting

Apr 12 09:06:46 ash lxd.daemon[113273]: api_extensions:

Apr 12 09:06:46 ash lxd.daemon[113273]: - cgroups

Apr 12 09:06:46 ash lxd.daemon[113273]: - sys_cpu_online

Apr 12 09:06:46 ash lxd.daemon[113273]: - proc_cpuinfo

Apr 12 09:06:46 ash lxd.daemon[113273]: - proc_diskstats

Apr 12 09:06:46 ash lxd.daemon[113273]: - proc_loadavg

Apr 12 09:06:46 ash lxd.daemon[113273]: - proc_meminfo

Apr 12 09:06:46 ash lxd.daemon[113273]: - proc_stat

Apr 12 09:06:46 ash lxd.daemon[113273]: - proc_swaps

Apr 12 09:06:46 ash lxd.daemon[113273]: - proc_uptime

Apr 12 09:06:46 ash lxd.daemon[113273]: - proc_slabinfo

Apr 12 09:06:46 ash lxd.daemon[113273]: - shared_pidns

Apr 12 09:06:46 ash lxd.daemon[113273]: - cpuview_daemon

Apr 12 09:06:46 ash lxd.daemon[113273]: - loadavg_daemon

Apr 12 09:06:46 ash lxd.daemon[113273]: - pidfds

Apr 12 09:06:47 ash lxd.daemon[113138]: => Starting LXD

Apr 12 09:06:47 ash lxd.daemon[113284]: time=“2024-04-12T09:06:47Z” level=warning msg=" - Couldn’t find the CGroup blkio.weight, disk priority will be ignored"

Apr 12 09:06:47 ash lxd.daemon[113284]: time=“2024-04-12T09:06:47Z” level=warning msg=" - Couldn’t find the CGroup memory swap accounting, swap limits will be ignored"

2

So I’ve found the zfs “partition” and mounted the containers: means I can recover my work which is a big relief, but seems a bit flakey that doing something dumb inside a container kills LXD for the entire machine.

Presumably there’s something fundamental I’m missing (I use LXD containers daily but interact with LXD only occasionally, means my expertise is pretty minimal).

Any advice very welcome, but at least I’m not looking at redoing 2 weeks work now!

1 Like
3

One more point of clarification, the storage pool I was using was/is 1TB so plenty big enough, I wandered if the Ubuntu install had blown the storage pool.

4

No help here, other than to report that I’ve had the same issue.

I noticed some of my containers were running “laggy” and when I remoted into the host, I was unable to issue any lxc commands as the socket was unavailable. I was about to depart on a work trip so decided to do a quick host reboot, which resulted in lxd not starting and all my containers being down whilst I’ve been away. I’m less than happy and ruing the decision to allow the snap to auto update!

Any pointers are appreciated.

5

This link might be helpful though. I’ll ensure I’ve the HWE kernel and reboot.

6

And confirmed - installed the HWE kernel, did a reboot, LXD started and the containers came up!

7

Lucky you ! I ended up rebuilding the server after copying my data off the container ZFS pool (which I manually mounted outside of LXD).

Pain in the ass but all I could think to do in the circumstances.