Rebooted system and all incus containers gone!

trumee · October 17, 2024, 5:30pm

Hello,

I rebooted my system and dont see any incus containers.

The system log shows this prior to reboot.

# journalctl --since "2024-10-17 21:00:00"|grep incus
Oct 17 22:07:17 tvbox systemd[1]: var-lib-incus-storage\x2dpools-pool1-images-67116b02f98564230887588368f1d1ff91749d617100bd7d5bdea247a03ba832.mount: Deactivated successfully.
Oct 17 22:07:19 tvbox incusd[890010]: time="2024-10-17T22:07:19+05:30" level=error msg="Failed to update the image" err="Failed getting remote image info: Failed getting image: The requested image couldn't be found" fingerprint=6b4a1d3ad551e1f62ad9fde737e2978663f00fa6376a2fef030ed36633918fc8
Oct 17 22:07:30 tvbox incusd[890010]: time="2024-10-17T22:07:30+05:30" level=error msg="Failed to update the image" err="Failed getting remote image info: Failed getting image: The requested image couldn't be found" fingerprint=acb8f742bfffa41f924939857b470b1ee6b55314e7219c65a7e982e0319ef3c5
Oct 17 22:07:31 tvbox incusd[890010]: time="2024-10-17T22:07:31+05:30" level=error msg="Failed to update the image" err="Failed getting remote image info: Failed getting image: The requested image couldn't be found" fingerprint=c533845b5db1747674ee915cbb20df6eb47c953bb7caf1fec5b35ae9ccf98c18
Oct 17 22:40:35 tvbox systemd[1]: incus.service: Stopping timed out. Terminating.
Oct 17 22:40:35 tvbox incusd[890010]: time="2024-10-17T22:40:35+05:30" level=warning msg="Ignoring signal, shutdown already in progress" signal=terminated
Oct 17 22:40:35 tvbox systemd[1]: incus.service: Control process exited, code=killed, status=15/TERM
Oct 17 22:41:05 tvbox systemd[1]: incus.service: State 'stop-sigterm' timed out. Killing.
Oct 17 22:41:05 tvbox systemd[1]: incus.service: Killing process 890010 (incusd) with signal SIGKILL.
Oct 17 22:41:05 tvbox systemd[1]: incus.service: Main process exited, code=killed, status=9/KILL
Oct 17 22:41:05 tvbox systemd[1]: incus.service: Failed with result 'timeout'.
Oct 17 22:41:05 tvbox systemd[1]: incus.service: Unit process 890200 (dnsmasq) remains running after unit stopped.
Oct 17 22:41:05 tvbox systemd[1]: incus.service: Unit process 1096567 (incusd) remains running after unit stopped.
Oct 17 22:41:05 tvbox systemd[1]: incus.service: Unit process 2967195 (incusd) remains running after unit stopped.
Oct 17 22:41:05 tvbox systemd[1]: incus.service: Unit process 514292 (incusd) remains running after unit stopped.
Oct 17 22:41:05 tvbox systemd[1]: incus.service: Consumed 33min 1.715s CPU time, 521.9M memory peak.
Oct 17 22:41:06 tvbox systemd[1]: incus.socket: Deactivated successfully.
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/guestapi...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/shmounts...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/dbserver...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/docker1...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/icecast...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/mqtt...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/mythvm...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/openhab...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/proxy...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/rssh...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/default/containers/voip2...
Oct 17 22:41:07 tvbox systemd[1]: Unmounting /var/lib/incus/storage-pools/pool1/containers/esphome...
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-guestapi.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/guestapi.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-shmounts.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/shmounts.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-dbserver.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/dbserver.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-docker1.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/docker1.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-icecast.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/icecast.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-mqtt.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/mqtt.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-mythvm.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/mythvm.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-openhab.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/openhab.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-proxy.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/proxy.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-rssh.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/rssh.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-default-containers-voip2.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/default/containers/voip2.
Oct 17 22:41:11 tvbox systemd[1]: var-lib-incus-storage\x2dpools-pool1-containers-esphome.mount: Deactivated successfully.
Oct 17 22:41:11 tvbox systemd[1]: Unmounted /var/lib/incus/storage-pools/pool1/containers/esphome.
Oct 17 22:41:12 tvbox systemd[1]: Unmounting var-lib-incus.mount...
Oct 17 22:41:12 tvbox umount[408863]: umount: /var/lib/incus: target is busy.
Oct 17 22:41:12 tvbox systemd[1]: var-lib-incus.mount: Mount process exited, code=exited, status=32/n/a
Oct 17 22:41:12 tvbox systemd[1]: Failed unmounting var-lib-incus.mount.
Oct 17 22:46:40 tvbox incusd[9609]: time="2024-10-17T22:46:40+05:30" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
Oct 17 22:46:40 tvbox incusd[9609]: time="2024-10-17T22:46:40+05:30" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
Oct 17 22:46:40 tvbox incusd[9609]: time="2024-10-17T22:46:40+05:30" level=error msg="Failed writing error for HTTP response" err="write unix /var/lib/incus/unix.socket->@: write: broken pipe" url=/1.0 writeErr="write unix /var/lib/incus/unix.socket->@: write: broken pipe"
Oct 17 22:46:50 tvbox incusd[9609]: time="2024-10-17T22:46:50+05:30" level=warning msg="Failed to update instance types: Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": net/http: TLS handshake timeout"
Oct 17 22:46:50 tvbox incusd[9609]: time="2024-10-17T22:46:50+05:30" level=error msg="Failed updating instance types" err="Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": net/http: TLS handshake timeout"

If i try to list incus containers i get,

# incus list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

# tree /var/lib/incus --du -hs
[ 24M]  /var/lib/incus
├── [   8]  backups
│   ├── [   2]  custom
│   └── [   2]  instances
├── [   2]  containers
├── [   2]  containers-snapshots
├── [ 24M]  database
│   ├── [ 24M]  global
│   │   ├── [  64]  0000000000000001-0000000000000001
│   │   ├── [  32]  metadata1
│   │   ├── [8.0M]  open-1
│   │   ├── [8.0M]  open-2
│   │   └── [8.0M]  open-3
│   └── [ 48K]  local.db
├── [   2]  devices
├── [   2]  disks
├── [  60]  guestapi
│   └── [   0]  sock
├── [   2]  images
├── [   2]  networks
├── [  14]  security
│   ├── [   8]  apparmor
│   │   ├── [   2]  cache
│   │   └── [   2]  profiles
│   └── [   2]  seccomp
├── [ 745]  server.crt
├── [ 288]  server.key
├── [  40]  shmounts
├── [   2]  storage-pools
├── [   0]  unix.socket
├── [   2]  virtual-machines
└── [   2]  virtual-machines-snapshots

  72M used in 22 directories, 10 files

The zfs dataset are still present.

rpool/lxd                                                                                   29.1G   222G    96K  legacy
rpool/lxd/buckets                                                                             96K   222G    96K  legacy
rpool/lxd/containers                                                                        23.3G   222G    96K  legacy
rpool/lxd/containers/dbserver                                                               1.28G   222G   723M  legacy
rpool/lxd/containers/docker1                                                                3.59G   222G  3.27G  legacy
rpool/lxd/containers/icecast                                                                 860M   222G   837M  legacy
rpool/lxd/containers/mqtt                                                                   3.84M   222G  9.75M  legacy
rpool/lxd/containers/mythvm                                                                 2.90G   222G  2.37G  legacy
rpool/lxd/containers/openhab                                                                2.63G   222G  1.80G  legacy
rpool/lxd/containers/proxy                                                                  5.96G   222G  4.83G  legacy
rpool/lxd/containers/rssh                                                                   7.64M   222G  14.0M  legacy
rpool/lxd/containers/voip2                                                                  6.07G   222G  5.25G  legacy

How do i get my containers back?

trumee · October 17, 2024, 6:37pm

I was able to recover the containers,

# incus admin recover
This server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: yes
Name of the storage pool: default
Name of the storage backend (btrfs, dir, zfs): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): rpool/lxd
Additional storage pool configuration property (KEY=VALUE, empty when done): 
Would you like to recover another storage pool? (yes/no) [default=no]: 
The recovery process will be scanning the following storage pools:
 - NEW: "default" (backend="zfs", source="rpool/lxd")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
The following unknown storage pools have been found:
 - Storage pool "default" of type "zfs"
The following unknown volumes have been found:
 - Container "dbserver" on pool "default" in project "default" (includes 1 snapshots)
 - Container "rssh" on pool "default" in project "default" (includes 0 snapshots)
 - Container "voip2" on pool "default" in project "default" (includes 1 snapshots)
 - Container "mythvm" on pool "default" in project "default" (includes 0 snapshots)
 - Container "icecast" on pool "default" in project "default" (includes 0 snapshots)
 - Container "openhab" on pool "default" in project "default" (includes 2 snapshots)
 - Container "proxy" on pool "default" in project "default" (includes 0 snapshots)
 - Container "docker1" on pool "default" in project "default" (includes 0 snapshots)
 - Container "mqtt" on pool "default" in project "default" (includes 0 snapshots)
Would you like those to be recovered? (yes/no) [default=no]: yes
Starting recovery...

However it is a mystery why did i lose the containers.

stgraber · October 18, 2024, 4:52am

All the clues are in your logs:

Oct 17 22:41:12 tvbox umount[408863]: umount: /var/lib/incus: target is busy.

This shows that prior to the system rebooting, you had something mounted over /var/lib/incus.

Following the reboot, this mount probably was not applied again, leading you to get a clean /var/lib/incus, explaining the empty database and instance list.

If you can track down what was mounted at /var/lib/incus and get it mounted again reliably, you should then get your old Incus database and files back.