Containers no longer start, lxd hangs on shutdown after problems with zfs pool

pajot · December 24, 2021, 8:45am

I’m running LXD 4.21 on Gentoo, kernel 5.4.156. After some problems with zfs, which I described here, my containers no longer start. Worse, once lxd is running, the host system cannot be shut down cleanly, because lxd refuses to stop.

pajot@t470 2021-12-24 01:09:45 ~ $ lxc list
+---------------+---------+------+------+-----------+-----------+
|     NAME      |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+---------------+---------+------+------+-----------+-----------+
| gentoo-PG-C01 | STOPPED |      |      | CONTAINER | 0         |
+---------------+---------+------+------+-----------+-----------+
| gentoo-WS-C01 | STOPPED |      |      | CONTAINER | 1         |
+---------------+---------+------+------+-----------+-----------+
| ubuntu-PG-C01 | STOPPED |      |      | CONTAINER | 0         |
+---------------+---------+------+------+-----------+-----------+
pajot@t470 2021-12-24 01:09:46 ~ $ lxc start gentoo-WS-C01

Now lxc start hangs and consumes 100% CPU. Here is the top output:

top - 01:15:58 up  1:14,  7 users,  load average: 1.21, 1.10, 0.82
Tasks: 253 total,   1 running, 252 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.6 us, 24.7 sy,  0.0 ni, 72.6 id,  0.1 wa,  0.0 hi,  2.0 si,  0.0 st
MiB Mem :  31985.2 total,  27222.6 free,   2102.1 used,   2660.4 buff/cache
MiB Swap:  32044.0 total,  32044.0 free,      0.0 used.  29343.4 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                
26102 root      20   0 1971832 167872  36444 S 100.0   0.5   6:27.01 lxd

Here’s the kernel version:

# uname -a
Linux t470 5.4.156-gentoo #4 SMP Mon Dec 20 14:03:19 MST 2021 x86_64 Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz GenuineIntel GNU/Linux

I have lxd log level set to debug. I don’t see any error messages in the lxd log. In the container log, there are messages like this:

lxc 20211224081945.619 TRACE commands - commands.c:lxc_cmd:509 - Verbindungsaufbau abgelehnt - Command "get_state" failed to connect command socket

The pool is up and seems fine:

root@t470 2021-12-24 01:43:50 /var/log/lxd/gentoo-WS-C01 # zpool status
  pool: zfs_lxd
 state: ONLINE
  scan: resilvered 45K in 00:00:00 with 0 errors on Thu Aug 19 07:00:27 2021
config:

        NAME                           STATE     READ WRITE CKSUM
        zfs_lxd                        ONLINE       0     0     0
          mirror-0                     ONLINE       0     0     0
            /var/lib/zfs_img/zfs0.img  ONLINE       0     0     0
            /var/lib/zfs_img/zfs1.img  ONLINE       0     0     0
          mirror-1                     ONLINE       0     0     0
            /var/lib/zfs_img/zfs2.img  ONLINE       0     0     0
            /var/lib/zfs_img/zfs3.img  ONLINE       0     0     0

errors: No known data errors

Here’s what happens when I try to stop the lxd daemon. (Keep in mind I’m running Gentoo with OpenRC and not systemd):

# rc-service lxd stop
 * Stopping lxd service and containers, waiting 40s ...
 * start-stop-daemon: 1 process refused to stop                                                                                                                       [ !! ]
 * ERROR: lxd failed to stop

cemzafer · December 24, 2021, 12:21pm

Hi @pajot,
Could you stop the lxd service and trigger from the command line lxd --debug --group lxd and post the output please? Have you ever checked the output of the lxc operation ls command?
Regards.

pajot · December 24, 2021, 9:20pm

Hi Cem,

thanks for your very helpful post! Here is the debug output you requested:

DBUG[12-24|14:12:27] Connecting to a local LXD over a Unix socket 
DBUG[12-24|14:12:27] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
INFO[12-24|14:12:27] LXD is starting                          path=/var/lib/lxd version=4.21 mode=normal
INFO[12-24|14:12:27] Kernel uid/gid map: 
INFO[12-24|14:12:27]  - u 0 0 4294967295 
INFO[12-24|14:12:27]  - g 0 0 4294967295 
INFO[12-24|14:12:27] Configured LXD uid/gid map: 
INFO[12-24|14:12:27]  - u 0 1000000 65536 
INFO[12-24|14:12:27]  - g 0 1000000 65536 
WARN[12-24|14:12:27] AppArmor support has been disabled because of lack of kernel support 
INFO[12-24|14:12:27] Kernel features: 
INFO[12-24|14:12:27]  - closing multiple file descriptors efficiently: no 
INFO[12-24|14:12:27]  - netnsid-based network retrieval: yes 
INFO[12-24|14:12:27]  - pidfds: yes 
INFO[12-24|14:12:27]  - core scheduling: no 
INFO[12-24|14:12:27]  - uevent injection: yes 
INFO[12-24|14:12:27]  - seccomp listener: yes 
INFO[12-24|14:12:27]  - seccomp listener continue syscalls: no 
INFO[12-24|14:12:27]  - seccomp listener add file descriptors: no 
INFO[12-24|14:12:27]  - attach to namespaces via pidfds: no 
INFO[12-24|14:12:27]  - safe native terminal allocation : yes 
INFO[12-24|14:12:27]  - unprivileged file capabilities: yes 
INFO[12-24|14:12:27]  - cgroup layout: cgroup2 
WARN[12-24|14:12:27]  - AppArmor support has been disabled, Disabled because of lack of kernel support 
WARN[12-24|14:12:27]  - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored 
WARN[12-24|14:12:27]  - Couldn't find the CGroup network priority controller, network priority will be ignored 
WARN[12-24|14:12:27]  - Couldn't find the CGroup memory swap accounting, swap limits will be ignored 
INFO[12-24|14:12:27]  - shiftfs support: no 
INFO[12-24|14:12:27] - idmapped mounts kernel support: no 
WARN[12-24|14:12:27] Instance type not operational            type=virtual-machine driver=qemu err="vhost_vsock kernel module not loaded: Failed to run: modprobe -b vhost_vsock: modprobe: FATAL: Module vhost_vsock not found in directory /lib/modules/5.4.156-gentoo"
INFO[12-24|14:12:27] Initializing local database 
DBUG[12-24|14:12:27] Refreshing local trusted certificate cache 
INFO[12-24|14:12:27] Set client certificate to server certificate fingerprint=96067af6b94aee1d46250a2b946a5f073e6f1f7ba37336cf0651e2c756251671
DBUG[12-24|14:12:27] Initializing database gateway 
INFO[12-24|14:12:27] Starting database node                   id=1 address=1 role=voter
DBUG[12-24|14:12:28] Connecting to a local LXD over a Unix socket 
DBUG[12-24|14:12:28] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
DBUG[12-24|14:12:28] Detected stale unix socket, deleting 
INFO[12-24|14:12:28] Starting /dev/lxd handler: 
INFO[12-24|14:12:28]  - binding devlxd socket                 socket=/var/lib/lxd/devlxd/sock
INFO[12-24|14:12:28] REST API daemon: 
INFO[12-24|14:12:28]  - binding Unix socket                   socket=/var/lib/lxd/unix.socket
INFO[12-24|14:12:28] Initializing global database 
INFO[12-24|14:12:28] Connecting to global database 
DBUG[12-24|14:12:28] Dqlite: attempt 1: server 1: connected 
INFO[12-24|14:12:28] Connected to global database 
INFO[12-24|14:12:28] Initialized global database 
DBUG[12-24|14:12:28] Firewall detected "nftables" incompatibility: Backend command "nft" missing 
INFO[12-24|14:12:28] Firewall loaded driver                   driver=xtables
INFO[12-24|14:12:28] Initializing storage pools 
DBUG[12-24|14:12:28] Initializing and checking storage pool   pool=lxd_zfs
DBUG[12-24|14:12:28] Mount started                            driver=zfs pool=lxd_zfs
DBUG[12-24|14:12:28] Mount finished                           driver=zfs pool=lxd_zfs
INFO[12-24|14:12:28] Initializing daemon storage mounts 
INFO[12-24|14:12:28] Loading daemon configuration 
INFO[12-24|14:12:28] Initializing networks 
DBUG[12-24|14:12:28] Start                                    network=lxdbr0 project=default driver=bridge
DBUG[12-24|14:12:28] Setting up network                       network=lxdbr0 project=default driver=bridge
DBUG[12-24|14:12:28] Stable MAC generated                     network=lxdbr0 project=default driver=bridge seed=96067af6b94aee1d46250a2b946a5f073e6f1f7ba37336cf0651e2c756251671.0.3 hwAddr=00:16:3e:95:9a:23
WARN[12-24|14:12:28] Failed enabling VLAN filtering on bridge "lxdbr0": open /sys/class/net/lxdbr0/bridge/vlan_filtering: permission denied network=lxdbr0 project=default driver=bridge
WARN[12-24|14:12:28] Failed setting bridge VLAN default PVID for "lxdbr0": open /sys/class/net/lxdbr0/bridge/default_pvid: permission denied network=lxdbr0 project=default driver=bridge
DBUG[12-24|14:12:28] Clearing firewall                        network=lxdbr0 project=default driver=bridge
DBUG[12-24|14:12:28] Setting up firewall                      network=lxdbr0 project=default driver=bridge
DBUG[12-24|14:12:28] New task Operation: 3d468492-014a-4a22-8316-79c057697078 
INFO[12-24|14:12:28] Pruning leftover image files 
DBUG[12-24|14:12:28] Started task operation: 3d468492-014a-4a22-8316-79c057697078 
INFO[12-24|14:12:28] Done pruning leftover image files 
INFO[12-24|14:12:28] Starting device monitor 
WARN[12-24|14:12:28] Failed to initialize fanotify, falling back on fsnotify err="Failed to initialize fanotify: function not implemented"
DBUG[12-24|14:12:28] Success for task operation: 3d468492-014a-4a22-8316-79c057697078 
DBUG[12-24|14:12:28] Initialized filesystem monitor           path=/dev
DBUG[12-24|14:12:28] Registering running instances 
INFO[12-24|14:12:28] Started seccomp handler                  path=/var/lib/lxd/seccomp.socket
DBUG[12-24|14:12:28] Refreshing trusted certificate cache 
DBUG[12-24|14:12:28] New task Operation: 8b7f143e-1c08-4715-b3b0-58daca005b05 
INFO[12-24|14:12:28] Pruning expired images 
DBUG[12-24|14:12:28] Started task operation: 8b7f143e-1c08-4715-b3b0-58daca005b05 
INFO[12-24|14:12:28] Done pruning expired images 
DBUG[12-24|14:12:28] New task Operation: 3aa3db99-b7bd-419f-b39f-87e780628c1c 
INFO[12-24|14:12:28] Pruning expired instance backups 
DBUG[12-24|14:12:28] Started task operation: 3aa3db99-b7bd-419f-b39f-87e780628c1c 
DBUG[12-24|14:12:28] Success for task operation: 8b7f143e-1c08-4715-b3b0-58daca005b05 
INFO[12-24|14:12:28] Done pruning expired instance backups 
DBUG[12-24|14:12:28] Success for task operation: 3aa3db99-b7bd-419f-b39f-87e780628c1c 
DBUG[12-24|14:12:28] New task Operation: 3f91ca1d-0a4f-43db-9e15-bca807b2c733 
DBUG[12-24|14:12:28] New task Operation: e52c45e2-7f24-47ea-9ea9-a267cdac0e10 
INFO[12-24|14:12:28] Updating instance types 
DBUG[12-24|14:12:28] Started task operation: e52c45e2-7f24-47ea-9ea9-a267cdac0e10 
INFO[12-24|14:12:28] Done updating instance types 
DBUG[12-24|14:12:28] Start started                            instanceType=container instance=gentoo-PG-C01 project=default stateful=false
DBUG[12-24|14:12:28] Instance operation lock created          project=default instance=gentoo-PG-C01 action=start reusable=false
DBUG[12-24|14:12:28] New task Operation: 16eda46c-ae8b-45cf-aca4-a9f5d39aaa15 
INFO[12-24|14:12:28] Expiring log files 
DBUG[12-24|14:12:28] Started task operation: 3f91ca1d-0a4f-43db-9e15-bca807b2c733 
INFO[12-24|14:12:28] Done expiring log files 
DBUG[12-24|14:12:28] Success for task operation: 3f91ca1d-0a4f-43db-9e15-bca807b2c733 
INFO[12-24|14:12:28] Pruning resolved warnings 
DBUG[12-24|14:12:28] Started task operation: 16eda46c-ae8b-45cf-aca4-a9f5d39aaa15 
DBUG[12-24|14:12:28] New task Operation: d00d947b-478e-4379-bbac-9cb0d1a37673 
INFO[12-24|14:12:28] Done pruning resolved warnings 
DBUG[12-24|14:12:28] MountInstance started                    driver=zfs pool=lxd_zfs project=default instance=gentoo-PG-C01
DBUG[12-24|14:12:28] Success for task operation: 16eda46c-ae8b-45cf-aca4-a9f5d39aaa15 
INFO[12-24|14:12:28] Updating images 
DBUG[12-24|14:12:28] Started task operation: d00d947b-478e-4379-bbac-9cb0d1a37673 
INFO[12-24|14:12:28] Done updating images 
DBUG[12-24|14:12:28] Processing image                         alias=gentoo fingerprint=874c26d35039097edd5b2063b679d839988d1dcdb2c039ba9aa9c41a532ab2df server=https://images.linuxcontainers.org protocol=simplestreams
DBUG[12-24|14:12:28] Connecting to a remote simplestreams server URL=https://images.linuxcontainers.org
DBUG[12-24|14:12:28] Mounted ZFS dataset                      driver=zfs pool=lxd_zfs path=/var/lib/lxd/storage-pools/lxd_zfs/containers/gentoo-PG-C01 dev=zfs_lxd/containers/gentoo-PG-C01
DBUG[12-24|14:12:28] MountInstance finished                   driver=zfs pool=lxd_zfs project=default instance=gentoo-PG-C01
DBUG[12-24|14:12:28] Starting device                          instanceType=container instance=gentoo-PG-C01 project=default device=eth0 type=nic
DBUG[12-24|14:12:29] Starting device                          instanceType=container instance=gentoo-PG-C01 project=default device=root type=disk
DBUG[12-24|14:12:29] Stopping device                          instanceType=container instance=gentoo-PG-C01 project=default device=root type=disk
DBUG[12-24|14:12:29] Stopping device                          instanceType=container instance=gentoo-PG-C01 project=default device=eth0 type=nic
DBUG[12-24|14:12:29] Clearing instance firewall static filters parent=lxdbr0 dev=eth0 host_name=veth5eab6e92 hwaddr=00:16:3e:80:df:1d ipv4=0.0.0.0 ipv6=:: project=default instance=gentoo-PG-C01
EROR[12-24|14:12:29] Failed to remove static IP network filters for "eth0": Failed to get a list of network filters to for "eth0": Failed to run: ebtables -L --Lmac2 --Lx: modprobe: FATAL: Module ebtables not found in directory /lib/modules/5.4.156-gentoo
The kernel doesn't support the ebtables 'filter' table. 
DBUG[12-24|14:12:29] Clearing instance firewall dynamic filters instance=gentoo-PG-C01 parent=lxdbr0 dev=eth0 host_name=veth5eab6e92 hwaddr=00:16:3e:80:df:1d ipv4=<nil> ipv6=<nil> project=default
EROR[12-24|14:12:29] Failed to remove DHCP network assigned filters  for "eth0": Failed to get a list of network filters to for "eth0": Failed to run: ebtables -L --Lmac2 --Lx: modprobe: FATAL: Module ebtables not found in directory /lib/modules/5.4.156-gentoo
The kernel doesn't support the ebtables 'filter' table. 
DBUG[12-24|14:12:29] UnmountInstance started                  driver=zfs pool=lxd_zfs project=default instance=gentoo-PG-C01
DBUG[12-24|14:12:33] Success for task operation: e52c45e2-7f24-47ea-9ea9-a267cdac0e10 
EROR[12-24|14:12:33] Failed to update the image               err="Failed getting remote image info: Failed getting image: The requested image couldn't be found" fingerprint=874c26d35039097edd5b2063b679d839988d1dcdb2c039ba9aa9c41a532ab2df
DBUG[12-24|14:12:33] Success for task operation: d00d947b-478e-4379-bbac-9cb0d1a37673 
DBUG[12-24|14:12:58] Instance operation lock finished         project=default instance=gentoo-PG-C01 action=start reusable=false err="Instance \"start\" operation timed out after 30 seconds"

Although I doubt it’s the cause, I’m rebuilding the kernel with ebtables support now.

lxd started in this way also does not respond to signals and eats up 100% of one processor core.

pajot · December 24, 2021, 9:50pm

lxc operation ls gives me an empty table:

$ LANG=C lxc operation ls
+----+------+-------------+--------+------------+---------+
| ID | TYPE | DESCRIPTION | STATUS | CANCELABLE | CREATED |
+----+------+-------------+--------+------------+---------+
$

pajot · December 24, 2021, 11:48pm

After rebuilding the kernel with ebtables support, I tried redirecting the output of lxd with

lxd --debug --group lxd 2>&1 | tee debug_log.txt

Instead of the nice readable output that appears on the screen, I get output that looks like the log output. Here is what I ended up with:

t=2021-12-24T15:18:38-0700 lvl=info msg="LXD is starting" mode=normal path=/var/lib/lxd version=4.21
t=2021-12-24T15:18:38-0700 lvl=info msg="Kernel uid/gid map:" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - u 0 0 4294967295" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - g 0 0 4294967295" 
t=2021-12-24T15:18:38-0700 lvl=info msg="Configured LXD uid/gid map:" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - u 0 1000000 65536" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - g 0 1000000 65536" 
t=2021-12-24T15:18:38-0700 lvl=warn msg="AppArmor support has been disabled because of lack of kernel support" 
t=2021-12-24T15:18:38-0700 lvl=info msg="Kernel features:" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - closing multiple file descriptors efficiently: no" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - netnsid-based network retrieval: yes" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - pidfds: yes" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - core scheduling: no" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - uevent injection: yes" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - seccomp listener: yes" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - seccomp listener continue syscalls: no" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - seccomp listener add file descriptors: no" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - attach to namespaces via pidfds: no" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - safe native terminal allocation : yes" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - cgroup layout: cgroup2" 
t=2021-12-24T15:18:38-0700 lvl=warn msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support" 
t=2021-12-24T15:18:38-0700 lvl=warn msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored" 
t=2021-12-24T15:18:38-0700 lvl=warn msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored" 
t=2021-12-24T15:18:38-0700 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored" 
t=2021-12-24T15:18:38-0700 lvl=info msg=" - shiftfs support: no" 
t=2021-12-24T15:18:38-0700 lvl=info msg="- idmapped mounts kernel support: no" 
t=2021-12-24T15:18:38-0700 lvl=warn msg="Instance type not operational" driver=qemu err="vhost_vsock kernel module not loaded: Failed to run: modprobe -b vhost_vsock: modprobe: FATAL: Module vhost_vsock not found in directory /lib/modules/5.4.156-gentoo" type=virtual-machine
t=2021-12-24T15:18:38-0700 lvl=info msg="Initializing local database" 
t=2021-12-24T15:18:38-0700 lvl=dbug msg="Refreshing local trusted certificate cache" 
t=2021-12-24T15:18:38-0700 lvl=info msg="Set client certificate to server certificate" fingerprint=96067af6b94aee1d46250a2b946a5f073e6f1f7ba37336cf0651e2c756251671
t=2021-12-24T15:18:38-0700 lvl=dbug msg="Initializing database gateway" 
t=2021-12-24T15:18:38-0700 lvl=info msg="Starting database node" address=1 id=1 role=voter
t=2021-12-24T15:18:39-0700 lvl=info msg="Starting /dev/lxd handler:" 
t=2021-12-24T15:18:39-0700 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2021-12-24T15:18:39-0700 lvl=info msg="REST API daemon:" 
t=2021-12-24T15:18:39-0700 lvl=info msg=" - binding Unix socket" socket=/var/lib/lxd/unix.socket
t=2021-12-24T15:18:39-0700 lvl=info msg="Initializing global database" 
t=2021-12-24T15:18:39-0700 lvl=info msg="Connecting to global database" 
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Dqlite: attempt 1: server 1: connected" 
t=2021-12-24T15:18:39-0700 lvl=info msg="Connected to global database" 
t=2021-12-24T15:18:39-0700 lvl=info msg="Initialized global database" 
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Firewall detected \"nftables\" incompatibility: Backend command \"nft\" missing" 
t=2021-12-24T15:18:39-0700 lvl=info msg="Firewall loaded driver" driver=xtables
t=2021-12-24T15:18:39-0700 lvl=info msg="Initializing storage pools" 
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Initializing and checking storage pool" pool=lxd_zfs
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Mount started" driver=zfs pool=lxd_zfs
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Mount finished" driver=zfs pool=lxd_zfs
t=2021-12-24T15:18:39-0700 lvl=info msg="Initializing daemon storage mounts" 
t=2021-12-24T15:18:39-0700 lvl=info msg="Loading daemon configuration" 
t=2021-12-24T15:18:39-0700 lvl=info msg="Initializing networks" 
t=2021-12-24T15:18:39-0700 lvl=dbug msg=Start driver=bridge network=lxdbr0 project=default
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Setting up network" driver=bridge network=lxdbr0 project=default
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Stable MAC generated" driver=bridge hwAddr=00:16:3e:95:9a:23 network=lxdbr0 project=default seed=96067af6b94aee1d46250a2b946a5f073e6f1f7ba37336cf0651e2c756251671.0.3
t=2021-12-24T15:18:39-0700 lvl=warn msg="Failed enabling VLAN filtering on bridge \"lxdbr0\": open /sys/class/net/lxdbr0/bridge/vlan_filtering: permission denied" driver=bridge network=lxdbr0 project=default
t=2021-12-24T15:18:39-0700 lvl=warn msg="Failed setting bridge VLAN default PVID for \"lxdbr0\": open /sys/class/net/lxdbr0/bridge/default_pvid: permission denied" driver=bridge network=lxdbr0 project=default
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Clearing firewall" driver=bridge network=lxdbr0 project=default
t=2021-12-24T15:18:39-0700 lvl=dbug msg="Setting up firewall" driver=bridge network=lxdbr0 project=default
t=2021-12-24T15:18:40-0700 lvl=dbug msg="New task Operation: 5fd44a2d-5150-42d6-a11a-111f2732e803" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Pruning leftover image files" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Started task operation: 5fd44a2d-5150-42d6-a11a-111f2732e803" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Done pruning leftover image files" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Starting device monitor" 
t=2021-12-24T15:18:40-0700 lvl=warn msg="Failed to initialize fanotify, falling back on fsnotify" err="Failed to initialize fanotify: function not implemented"
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Success for task operation: 5fd44a2d-5150-42d6-a11a-111f2732e803" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Initialized filesystem monitor" path=/dev
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Registering running instances" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Started seccomp handler" path=/var/lib/lxd/seccomp.socket
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Refreshing trusted certificate cache" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="New task Operation: aab2f483-675d-4380-a499-251a4463689c" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Pruning expired images" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Started task operation: aab2f483-675d-4380-a499-251a4463689c" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Done pruning expired images" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="New task Operation: 7b6940ed-0e9e-40b7-99a8-b53b48fcd846" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Pruning expired instance backups" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Started task operation: 7b6940ed-0e9e-40b7-99a8-b53b48fcd846" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Success for task operation: aab2f483-675d-4380-a499-251a4463689c" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Done pruning expired instance backups" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Success for task operation: 7b6940ed-0e9e-40b7-99a8-b53b48fcd846" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="New task Operation: 207b0c00-1cd0-4ce8-97b8-89c4025bd8bf" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Expiring log files" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Started task operation: 207b0c00-1cd0-4ce8-97b8-89c4025bd8bf" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Done expiring log files" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Start started" instance=gentoo-PG-C01 instanceType=container project=default stateful=false
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Instance operation lock created" action=start instance=gentoo-PG-C01 project=default reusable=false
t=2021-12-24T15:18:40-0700 lvl=dbug msg="New task Operation: ab6fcbc5-f6b5-4f37-a08c-4b51ba18a13f" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="New task Operation: dd322094-fb14-419a-9824-f62fb52986fa" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Updating images" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Started task operation: ab6fcbc5-f6b5-4f37-a08c-4b51ba18a13f" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Success for task operation: 207b0c00-1cd0-4ce8-97b8-89c4025bd8bf" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Done updating images" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="New task Operation: a6e44671-cd90-42b7-8477-83d97f34cb29" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Pruning resolved warnings" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Started task operation: dd322094-fb14-419a-9824-f62fb52986fa" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Updating instance types" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Started task operation: a6e44671-cd90-42b7-8477-83d97f34cb29" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Success for task operation: dd322094-fb14-419a-9824-f62fb52986fa" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Done updating instance types" 
t=2021-12-24T15:18:40-0700 lvl=info msg="Done pruning resolved warnings" 
t=2021-12-24T15:18:40-0700 lvl=dbug msg="MountInstance started" driver=zfs instance=gentoo-PG-C01 pool=lxd_zfs project=default
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Processing image" alias=gentoo fingerprint=874c26d35039097edd5b2063b679d839988d1dcdb2c039ba9aa9c41a532ab2df protocol=simplestreams server=https://images.linuxcontainers.org
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Connecting to a remote simplestreams server" URL=https://images.linuxcontainers.org
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Mounted ZFS dataset" dev=zfs_lxd/containers/gentoo-PG-C01 driver=zfs path=/var/lib/lxd/storage-pools/lxd_zfs/containers/gentoo-PG-C01 pool=lxd_zfs
t=2021-12-24T15:18:40-0700 lvl=dbug msg="MountInstance finished" driver=zfs instance=gentoo-PG-C01 pool=lxd_zfs project=default
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Starting device" device=eth0 instance=gentoo-PG-C01 instanceType=container project=default type=nic
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Starting device" device=root instance=gentoo-PG-C01 instanceType=container project=default type=disk
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Stopping device" device=root instance=gentoo-PG-C01 instanceType=container project=default type=disk
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Stopping device" device=eth0 instance=gentoo-PG-C01 instanceType=container project=default type=nic
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Clearing instance firewall static filters" dev=eth0 host_name=vethdf95f895 hwaddr=00:16:3e:80:df:1d instance=gentoo-PG-C01 ipv4=0.0.0.0 ipv6=:: parent=lxdbr0 project=default
t=2021-12-24T15:18:40-0700 lvl=dbug msg="Clearing instance firewall dynamic filters" dev=eth0 host_name=vethdf95f895 hwaddr=00:16:3e:80:df:1d instance=gentoo-PG-C01 ipv4=<nil> ipv6=<nil> parent=lxdbr0 project=default
t=2021-12-24T15:18:40-0700 lvl=dbug msg="UnmountInstance started" driver=zfs instance=gentoo-PG-C01 pool=lxd_zfs project=default
t=2021-12-24T15:18:43-0700 lvl=eror msg="Failed to update the image" err="Failed getting remote image info: Failed getting image: The requested image couldn't be found" fingerprint=874c26d35039097edd5b2063b679d839988d1dcdb2c039ba9aa9c41a532ab2df
t=2021-12-24T15:18:43-0700 lvl=dbug msg="Success for task operation: ab6fcbc5-f6b5-4f37-a08c-4b51ba18a13f" 
t=2021-12-24T15:18:44-0700 lvl=dbug msg="Success for task operation: a6e44671-cd90-42b7-8477-83d97f34cb29" 
t=2021-12-24T15:19:10-0700 lvl=dbug msg="Instance operation lock finished" action=start err="Instance \"start\" operation timed out after 30 seconds" instance=gentoo-PG-C01 project=default reusable=false
t=2021-12-24T15:22:11-0700 lvl=dbug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=sfbosch
t=2021-12-24T15:22:11-0700 lvl=dbug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB/jCCAYSgAwIBAgIRAMfoyV53mestzWvAosNEv3EwCgYIKoZIzj0EAwMwMjEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEB0NDcw\\nMB4XDTIwMDcxNjE1MjMyNloXDTMwMDcxNDE1MjMyNlowMjEcMBoGA1UEChMTbGlu\\ndXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEB0NDcwMHYwEAYHKoZIzj0C\\nAQYFK4EEACIDYgAElqZAgy/Obtmn3DaNYL/cXpkHca4DtOwoUTsvo34Ug+S0R+NQ\\nLTT1WpGPxDoBOp6LvA+Rm9ZePrDNLMuVorKrDB+0jbCOnAW4aJA1WneuWal77Rje\\nOl2+HV5487l83pTEo14wXDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYB\\nBQUHAwEwDAYDVR0TAQH/BAIwADAnBgNVHREEIDAeggR0NDcwhwR/AAABhxAAAAAA\\nAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMBuT6tY3PvOOILqBBtFfaKs8\\nRS/tiG/nOmQw3s2PeAJqm0tJNsE2pVQpeHEBfGzxRgIxANho9MXoEuCuADVeoku6\\nAt7CdzVxVvUDcnBW60jp5t7LQXAi2EcIGdrot+sfeFiibg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"96067af6b94aee1d46250a2b946a5f073e6f1f7ba37336cf0651e2c756251671\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"4.0.11\",\n\t\t\t\t\"firewall\": \"xtables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"5.4.156-gentoo\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Gentoo\",\n\t\t\t\t\"os_version\": \"\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_name\": \"t470\",\n\t\t\t\t\"server_pid\": 5614,\n\t\t\t\t\"server_version\": \"4.21\",\n\t\t\t\t\"storage\": \"zfs\",\n\t\t\t\t\"storage_version\": \"2.0.6-r0-gentoo\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.02.188(2) (2021-05-07) / 1.02.172 (2021-05-07) / 4.41.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"zfs\",\n\t\t\t\t\t\t\"Version\": \"2.0.6-r0-gentoo\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
t=2021-12-24T15:22:11-0700 lvl=dbug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=sfbosch
t=2021-12-24T15:22:11-0700 lvl=dbug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB/jCCAYSgAwIBAgIRAMfoyV53mestzWvAosNEv3EwCgYIKoZIzj0EAwMwMjEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEB0NDcw\\nMB4XDTIwMDcxNjE1MjMyNloXDTMwMDcxNDE1MjMyNlowMjEcMBoGA1UEChMTbGlu\\ndXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEB0NDcwMHYwEAYHKoZIzj0C\\nAQYFK4EEACIDYgAElqZAgy/Obtmn3DaNYL/cXpkHca4DtOwoUTsvo34Ug+S0R+NQ\\nLTT1WpGPxDoBOp6LvA+Rm9ZePrDNLMuVorKrDB+0jbCOnAW4aJA1WneuWal77Rje\\nOl2+HV5487l83pTEo14wXDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYB\\nBQUHAwEwDAYDVR0TAQH/BAIwADAnBgNVHREEIDAeggR0NDcwhwR/AAABhxAAAAAA\\nAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMBuT6tY3PvOOILqBBtFfaKs8\\nRS/tiG/nOmQw3s2PeAJqm0tJNsE2pVQpeHEBfGzxRgIxANho9MXoEuCuADVeoku6\\nAt7CdzVxVvUDcnBW60jp5t7LQXAi2EcIGdrot+sfeFiibg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"96067af6b94aee1d46250a2b946a5f073e6f1f7ba37336cf0651e2c756251671\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"4.0.11\",\n\t\t\t\t\"firewall\": \"xtables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"5.4.156-gentoo\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Gentoo\",\n\t\t\t\t\"os_version\": \"\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_name\": \"t470\",\n\t\t\t\t\"server_pid\": 5614,\n\t\t\t\t\"server_version\": \"4.21\",\n\t\t\t\t\"storage\": \"zfs\",\n\t\t\t\t\"storage_version\": \"2.0.6-r0-gentoo\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.02.188(2) (2021-05-07) / 1.02.172 (2021-05-07) / 4.41.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"zfs\",\n\t\t\t\t\t\t\"Version\": \"2.0.6-r0-gentoo\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
t=2021-12-24T15:22:11-0700 lvl=dbug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/storage-pools?recursion=1" username=sfbosch
t=2021-12-24T15:22:11-0700 lvl=dbug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": [\n\t\t\t{\n\t\t\t\t\"config\": {\n\t\t\t\t\t\"source\": \"zfs_lxd\",\n\t\t\t\t\t\"volatile.initial_source\": \"zfs_lxd\",\n\t\t\t\t\t\"zfs.pool_name\": \"zfs_lxd\"\n\t\t\t\t},\n\t\t\t\t\"description\": \"\",\n\t\t\t\t\"name\": \"lxd_zfs\",\n\t\t\t\t\"driver\": \"zfs\",\n\t\t\t\t\"used_by\": [\n\t\t\t\t\t\"/1.0/images/874c26d35039097edd5b2063b679d839988d1dcdb2c039ba9aa9c41a532ab2df\",\n\t\t\t\t\t\"/1.0/images/8d87094883e69a195a8af8f988ba517d1ccc02b90b4ff984c97dc1e76bbe199d\",\n\t\t\t\t\t\"/1.0/instances/gentoo-PG-C01\",\n\t\t\t\t\t\"/1.0/instances/gentoo-WS-C01\",\n\t\t\t\t\t\"/1.0/instances/ubuntu-PG-C01\",\n\t\t\t\t\t\"/1.0/profiles/default\"\n\t\t\t\t],\n\t\t\t\t\"status\": \"Created\",\n\t\t\t\t\"locations\": [\n\t\t\t\t\t\"none\"\n\t\t\t\t]\n\t\t\t}\n\t\t]\n\t}" http_code=200
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Handling API request" ip=@ method=POST protocol=unix url=/internal/recover/validate username=sfbosch
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Mount started" driver=zfs pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Mount finished" driver=zfs pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/containers pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/custom pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/deleted pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/deleted/containers pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/deleted/custom pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/deleted/images pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/deleted/images/0c0942818e3d45e3132185283138f77d42f3167825031fa1e56550fadf8b6486 pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/deleted/images/25bc836a6adef83da97aa68893d0232d2c3809084e60afa094e33bbf664303a7 pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/deleted/virtual-machines pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/images pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="Ignoring unrecognised volume type" driver=zfs name=zfs_lxd/virtual-machines pool=lxd_zfs
t=2021-12-24T15:22:24-0700 lvl=dbug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"UnknownVolumes\": null,\n\t\t\t\"DependencyErrors\": null\n\t\t}\n\t}" http_code=200
t=2021-12-24T15:22:35-0700 lvl=dbug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=sfbosch
t=2021-12-24T15:22:35-0700 lvl=dbug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nMIIB/jCCAYSgAwIBAgIRAMfoyV53mestzWvAosNEv3EwCgYIKoZIzj0EAwMwMjEc\\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEB0NDcw\\nMB4XDTIwMDcxNjE1MjMyNloXDTMwMDcxNDE1MjMyNlowMjEcMBoGA1UEChMTbGlu\\ndXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEB0NDcwMHYwEAYHKoZIzj0C\\nAQYFK4EEACIDYgAElqZAgy/Obtmn3DaNYL/cXpkHca4DtOwoUTsvo34Ug+S0R+NQ\\nLTT1WpGPxDoBOp6LvA+Rm9ZePrDNLMuVorKrDB+0jbCOnAW4aJA1WneuWal77Rje\\nOl2+HV5487l83pTEo14wXDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYB\\nBQUHAwEwDAYDVR0TAQH/BAIwADAnBgNVHREEIDAeggR0NDcwhwR/AAABhxAAAAAA\\nAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMBuT6tY3PvOOILqBBtFfaKs8\\nRS/tiG/nOmQw3s2PeAJqm0tJNsE2pVQpeHEBfGzxRgIxANho9MXoEuCuADVeoku6\\nAt7CdzVxVvUDcnBW60jp5t7LQXAi2EcIGdrot+sfeFiibg==\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"96067af6b94aee1d46250a2b946a5f073e6f1f7ba37336cf0651e2c756251671\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"4.0.11\",\n\t\t\t\t\"firewall\": \"xtables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"false\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"5.4.156-gentoo\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Gentoo\",\n\t\t\t\t\"os_version\": \"\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_name\": \"t470\",\n\t\t\t\t\"server_pid\": 5614,\n\t\t\t\t\"server_version\": \"4.21\",\n\t\t\t\t\"storage\": \"zfs\",\n\t\t\t\t\"storage_version\": \"2.0.6-r0-gentoo\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.02.188(2) (2021-05-07) / 1.02.172 (2021-05-07) / 4.41.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"zfs\",\n\t\t\t\t\t\t\"Version\": \"2.0.6-r0-gentoo\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
t=2021-12-24T15:22:35-0700 lvl=dbug msg="Handling API request" ip=@ method=GET protocol=unix url="/1.0/instances?recursion=2" username=sfbosch
t=2021-12-24T15:22:35-0700 lvl=dbug msg="GetInstanceUsage started" driver=zfs instance=gentoo-WS-C01 pool=lxd_zfs project=default
t=2021-12-24T15:22:35-0700 lvl=dbug msg="GetInstanceUsage started" driver=zfs instance=gentoo-PG-C01 pool=lxd_zfs project=default
t=2021-12-24T15:22:35-0700 lvl=dbug msg="GetInstanceUsage started" driver=zfs instance=ubuntu-PG-C01 pool=lxd_zfs project=default
t=2021-12-24T15:22:35-0700 lvl=dbug msg="GetInstanceUsage finished" driver=zfs instance=gentoo-WS-C01 pool=lxd_zfs project=default
t=2021-12-24T15:22:35-0700 lvl=dbug msg="GetInstanceUsage finished" driver=zfs instance=gentoo-PG-C01 pool=lxd_zfs project=default
t=2021-12-24T15:22:35-0700 lvl=dbug msg="GetInstanceUsage finished" driver=zfs instance=ubuntu-PG-C01 pool=lxd_zfs project=default
t=2021-12-24T15:22:35-0700 lvl=dbug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": [\n\t\t\t{\n\t\t\t\t\"architecture\": \"x86_64\",\n\t\t\t\t\"config\": {\n\t\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\t\"image.description\": \"Gentoo current amd64 (20210817_16:07)\",\n\t\t\t\t\t\"image.os\": \"Gentoo\",\n\t\t\t\t\t\"image.release\": \"current\",\n\t\t\t\t\t\"image.serial\": \"20210817_16:07\",\n\t\t\t\t\t\"image.type\": \"squashfs\",\n\t\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\t\"volatile.base_image\": \"0c0942818e3d45e3132185283138f77d42f3167825031fa1e56550fadf8b6486\",\n\t\t\t\t\t\"volatile.eth0.hwaddr\": \"00:16:3e:80:df:1d\",\n\t\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.idmap\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.power\": \"RUNNING\",\n\t\t\t\t\t\"volatile.uuid\": \"e6bdb388-5096-4f98-9e5a-59d4f5c131b6\"\n\t\t\t\t},\n\t\t\t\t\"devices\": {},\n\t\t\t\t\"ephemeral\": false,\n\t\t\t\t\"profiles\": [\n\t\t\t\t\t\"default\"\n\t\t\t\t],\n\t\t\t\t\"stateful\": false,\n\t\t\t\t\"description\": \"\",\n\t\t\t\t\"created_at\": \"2021-08-19T07:50:18.797963063-06:00\",\n\t\t\t\t\"expanded_config\": {\n\t\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\t\"image.description\": \"Gentoo current amd64 (20210817_16:07)\",\n\t\t\t\t\t\"image.os\": \"Gentoo\",\n\t\t\t\t\t\"image.release\": \"current\",\n\t\t\t\t\t\"image.serial\": \"20210817_16:07\",\n\t\t\t\t\t\"image.type\": \"squashfs\",\n\t\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\t\"volatile.base_image\": \"0c0942818e3d45e3132185283138f77d42f3167825031fa1e56550fadf8b6486\",\n\t\t\t\t\t\"volatile.eth0.hwaddr\": \"00:16:3e:80:df:1d\",\n\t\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.idmap\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.power\": \"RUNNING\",\n\t\t\t\t\t\"volatile.uuid\": \"e6bdb388-5096-4f98-9e5a-59d4f5c131b6\"\n\t\t\t\t},\n\t\t\t\t\"expanded_devices\": {\n\t\t\t\t\t\"eth0\": {\n\t\t\t\t\t\t\"name\": \"eth0\",\n\t\t\t\t\t\t\"network\": \"lxdbr0\",\n\t\t\t\t\t\t\"type\": \"nic\"\n\t\t\t\t\t},\n\t\t\t\t\t\"root\": {\n\t\t\t\t\t\t\"path\": \"/\",\n\t\t\t\t\t\t\"pool\": \"lxd_zfs\",\n\t\t\t\t\t\t\"type\": \"disk\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"name\": \"gentoo-PG-C01\",\n\t\t\t\t\"status\": \"Stopped\",\n\t\t\t\t\"status_code\": 102,\n\t\t\t\t\"last_used_at\": \"2021-12-16T17:16:21.450732008-07:00\",\n\t\t\t\t\"location\": \"none\",\n\t\t\t\t\"type\": \"container\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"backups\": null,\n\t\t\t\t\"state\": {\n\t\t\t\t\t\"status\": \"Stopped\",\n\t\t\t\t\t\"status_code\": 102,\n\t\t\t\t\t\"disk\": {\n\t\t\t\t\t\t\"root\": {\n\t\t\t\t\t\t\t\"usage\": 1179592704\n\t\t\t\t\t\t}\n\t\t\t\t\t},\n\t\t\t\t\t\"memory\": {\n\t\t\t\t\t\t\"usage\": 0,\n\t\t\t\t\t\t\"usage_peak\": 0,\n\t\t\t\t\t\t\"swap_usage\": 0,\n\t\t\t\t\t\t\"swap_usage_peak\": 0\n\t\t\t\t\t},\n\t\t\t\t\t\"network\": null,\n\t\t\t\t\t\"pid\": 0,\n\t\t\t\t\t\"processes\": 0,\n\t\t\t\t\t\"cpu\": {\n\t\t\t\t\t\t\"usage\": 0\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"snapshots\": null\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"architecture\": \"x86_64\",\n\t\t\t\t\"config\": {\n\t\t\t\t\t\"boot.autostart\": \"true\",\n\t\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\t\"image.description\": \"Gentoo  amd64 (20200901_1326)\",\n\t\t\t\t\t\"image.name\": \"gentoo--amd64-default-20200901_1326\",\n\t\t\t\t\t\"image.os\": \"gentoo\",\n\t\t\t\t\t\"image.serial\": \"20200901_1326\",\n\t\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\t\"limits.memory\": \"2GB\",\n\t\t\t\t\t\"volatile.base_image\": \"8d87094883e69a195a8af8f988ba517d1ccc02b90b4ff984c97dc1e76bbe199d\",\n\t\t\t\t\t\"volatile.eth0.hwaddr\": \"00:16:3e:a4:13:e5\",\n\t\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.idmap\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.power\": \"RUNNING\",\n\t\t\t\t\t\"volatile.uuid\": \"a77c2a7b-3fed-4e8f-a348-8cebfa629375\"\n\t\t\t\t},\n\t\t\t\t\"devices\": {},\n\t\t\t\t\"ephemeral\": false,\n\t\t\t\t\"profiles\": [\n\t\t\t\t\t\"default\"\n\t\t\t\t],\n\t\t\t\t\"stateful\": false,\n\t\t\t\t\"description\": \"\",\n\t\t\t\t\"created_at\": \"2020-09-01T07:49:12.648584412-06:00\",\n\t\t\t\t\"expanded_config\": {\n\t\t\t\t\t\"boot.autostart\": \"true\",\n\t\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\t\"image.description\": \"Gentoo  amd64 (20200901_1326)\",\n\t\t\t\t\t\"image.name\": \"gentoo--amd64-default-20200901_1326\",\n\t\t\t\t\t\"image.os\": \"gentoo\",\n\t\t\t\t\t\"image.serial\": \"20200901_1326\",\n\t\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\t\"limits.memory\": \"2GB\",\n\t\t\t\t\t\"volatile.base_image\": \"8d87094883e69a195a8af8f988ba517d1ccc02b90b4ff984c97dc1e76bbe199d\",\n\t\t\t\t\t\"volatile.eth0.hwaddr\": \"00:16:3e:a4:13:e5\",\n\t\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.idmap\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.power\": \"RUNNING\",\n\t\t\t\t\t\"volatile.uuid\": \"a77c2a7b-3fed-4e8f-a348-8cebfa629375\"\n\t\t\t\t},\n\t\t\t\t\"expanded_devices\": {\n\t\t\t\t\t\"eth0\": {\n\t\t\t\t\t\t\"name\": \"eth0\",\n\t\t\t\t\t\t\"network\": \"lxdbr0\",\n\t\t\t\t\t\t\"type\": \"nic\"\n\t\t\t\t\t},\n\t\t\t\t\t\"root\": {\n\t\t\t\t\t\t\"path\": \"/\",\n\t\t\t\t\t\t\"pool\": \"lxd_zfs\",\n\t\t\t\t\t\t\"type\": \"disk\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"name\": \"gentoo-WS-C01\",\n\t\t\t\t\"status\": \"Stopped\",\n\t\t\t\t\"status_code\": 102,\n\t\t\t\t\"last_used_at\": \"2021-12-16T17:16:21.639335858-07:00\",\n\t\t\t\t\"location\": \"none\",\n\t\t\t\t\"type\": \"container\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"backups\": null,\n\t\t\t\t\"state\": {\n\t\t\t\t\t\"status\": \"Stopped\",\n\t\t\t\t\t\"status_code\": 102,\n\t\t\t\t\t\"disk\": {\n\t\t\t\t\t\t\"root\": {\n\t\t\t\t\t\t\t\"usage\": 5334328320\n\t\t\t\t\t\t}\n\t\t\t\t\t},\n\t\t\t\t\t\"memory\": {\n\t\t\t\t\t\t\"usage\": 0,\n\t\t\t\t\t\t\"usage_peak\": 0,\n\t\t\t\t\t\t\"swap_usage\": 0,\n\t\t\t\t\t\t\"swap_usage_peak\": 0\n\t\t\t\t\t},\n\t\t\t\t\t\"network\": null,\n\t\t\t\t\t\"pid\": 0,\n\t\t\t\t\t\"processes\": 0,\n\t\t\t\t\t\"cpu\": {\n\t\t\t\t\t\t\"usage\": 0\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"snapshots\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"expires_at\": \"0001-01-01T00:00:00Z\",\n\t\t\t\t\t\t\"architecture\": \"x86_64\",\n\t\t\t\t\t\t\"config\": {\n\t\t\t\t\t\t\t\"boot.autostart\": \"true\",\n\t\t\t\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\t\t\t\"image.description\": \"Gentoo  amd64 (20200901_1326)\",\n\t\t\t\t\t\t\t\"image.name\": \"gentoo--amd64-default-20200901_1326\",\n\t\t\t\t\t\t\t\"image.os\": \"gentoo\",\n\t\t\t\t\t\t\t\"image.serial\": \"20200901_1326\",\n\t\t\t\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\t\t\t\"limits.memory\": \"2GB\",\n\t\t\t\t\t\t\t\"volatile.base_image\": \"8d87094883e69a195a8af8f988ba517d1ccc02b90b4ff984c97dc1e76bbe199d\",\n\t\t\t\t\t\t\t\"volatile.eth0.hwaddr\": \"00:16:3e:a4:13:e5\",\n\t\t\t\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\t\t\"volatile.last_state.idmap\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\t\t\"volatile.last_state.power\": \"STOPPED\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"created_at\": \"2021-03-12T05:07:13.132775812-07:00\",\n\t\t\t\t\t\t\"devices\": {},\n\t\t\t\t\t\t\"ephemeral\": false,\n\t\t\t\t\t\t\"expanded_config\": {\n\t\t\t\t\t\t\t\"boot.autostart\": \"true\",\n\t\t\t\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\t\t\t\"image.description\": \"Gentoo  amd64 (20200901_1326)\",\n\t\t\t\t\t\t\t\"image.name\": \"gentoo--amd64-default-20200901_1326\",\n\t\t\t\t\t\t\t\"image.os\": \"gentoo\",\n\t\t\t\t\t\t\t\"image.serial\": \"20200901_1326\",\n\t\t\t\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\t\t\t\"limits.memory\": \"2GB\",\n\t\t\t\t\t\t\t\"volatile.base_image\": \"8d87094883e69a195a8af8f988ba517d1ccc02b90b4ff984c97dc1e76bbe199d\",\n\t\t\t\t\t\t\t\"volatile.eth0.hwaddr\": \"00:16:3e:a4:13:e5\",\n\t\t\t\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\t\t\"volatile.last_state.idmap\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\t\t\"volatile.last_state.power\": \"STOPPED\"\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"expanded_devices\": {\n\t\t\t\t\t\t\t\"eth0\": {\n\t\t\t\t\t\t\t\t\"name\": \"eth0\",\n\t\t\t\t\t\t\t\t\"network\": \"lxdbr0\",\n\t\t\t\t\t\t\t\t\"type\": \"nic\"\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\"root\": {\n\t\t\t\t\t\t\t\t\"path\": \"/\",\n\t\t\t\t\t\t\t\t\"pool\": \"lxd_zfs\",\n\t\t\t\t\t\t\t\t\"type\": \"disk\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"last_used_at\": \"0001-01-01T00:00:00Z\",\n\t\t\t\t\t\t\"name\": \"snap_matomo\",\n\t\t\t\t\t\t\"profiles\": [\n\t\t\t\t\t\t\t\"default\"\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"stateful\": false,\n\t\t\t\t\t\t\"size\": -1\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t},\n\t\t\t{\n\t\t\t\t\"architecture\": \"x86_64\",\n\t\t\t\t\"config\": {\n\t\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\t\"image.description\": \"Ubuntu focal amd64 (20210820_07:42)\",\n\t\t\t\t\t\"image.os\": \"Ubuntu\",\n\t\t\t\t\t\"image.release\": \"focal\",\n\t\t\t\t\t\"image.serial\": \"20210820_07:42\",\n\t\t\t\t\t\"image.type\": \"squashfs\",\n\t\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\t\"volatile.base_image\": \"25bc836a6adef83da97aa68893d0232d2c3809084e60afa094e33bbf664303a7\",\n\t\t\t\t\t\"volatile.eth0.hwaddr\": \"00:16:3e:8f:3d:93\",\n\t\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.idmap\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.power\": \"RUNNING\",\n\t\t\t\t\t\"volatile.uuid\": \"b3804b87-f638-488d-a4fc-b05e6f87c741\"\n\t\t\t\t},\n\t\t\t\t\"devices\": {},\n\t\t\t\t\"ephemeral\": false,\n\t\t\t\t\"profiles\": [\n\t\t\t\t\t\"default\"\n\t\t\t\t],\n\t\t\t\t\"stateful\": false,\n\t\t\t\t\"description\": \"\",\n\t\t\t\t\"created_at\": \"2021-08-20T11:26:27.817189289-06:00\",\n\t\t\t\t\"expanded_config\": {\n\t\t\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\t\t\"image.description\": \"Ubuntu focal amd64 (20210820_07:42)\",\n\t\t\t\t\t\"image.os\": \"Ubuntu\",\n\t\t\t\t\t\"image.release\": \"focal\",\n\t\t\t\t\t\"image.serial\": \"20210820_07:42\",\n\t\t\t\t\t\"image.type\": \"squashfs\",\n\t\t\t\t\t\"image.variant\": \"default\",\n\t\t\t\t\t\"volatile.base_image\": \"25bc836a6adef83da97aa68893d0232d2c3809084e60afa094e33bbf664303a7\",\n\t\t\t\t\t\"volatile.eth0.hwaddr\": \"00:16:3e:8f:3d:93\",\n\t\t\t\t\t\"volatile.idmap.base\": \"0\",\n\t\t\t\t\t\"volatile.idmap.current\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.idmap.next\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.idmap\": \"[{\\\"Isuid\\\":true,\\\"Isgid\\\":false,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536},{\\\"Isuid\\\":false,\\\"Isgid\\\":true,\\\"Hostid\\\":1000000,\\\"Nsid\\\":0,\\\"Maprange\\\":65536}]\",\n\t\t\t\t\t\"volatile.last_state.power\": \"RUNNING\",\n\t\t\t\t\t\"volatile.uuid\": \"b3804b87-f638-488d-a4fc-b05e6f87c741\"\n\t\t\t\t},\n\t\t\t\t\"expanded_devices\": {\n\t\t\t\t\t\"eth0\": {\n\t\t\t\t\t\t\"name\": \"eth0\",\n\t\t\t\t\t\t\"network\": \"lxdbr0\",\n\t\t\t\t\t\t\"type\": \"nic\"\n\t\t\t\t\t},\n\t\t\t\t\t\"root\": {\n\t\t\t\t\t\t\"path\": \"/\",\n\t\t\t\t\t\t\"pool\": \"lxd_zfs\",\n\t\t\t\t\t\t\"type\": \"disk\"\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"name\": \"ubuntu-PG-C01\",\n\t\t\t\t\"status\": \"Stopped\",\n\t\t\t\t\"status_code\": 102,\n\t\t\t\t\"last_used_at\": \"2021-12-16T17:16:22.018302391-07:00\",\n\t\t\t\t\"location\": \"none\",\n\t\t\t\t\"type\": \"container\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"backups\": null,\n\t\t\t\t\"state\": {\n\t\t\t\t\t\"status\": \"Stopped\",\n\t\t\t\t\t\"status_code\": 102,\n\t\t\t\t\t\"disk\": {\n\t\t\t\t\t\t\"root\": {\n\t\t\t\t\t\t\t\"usage\": 1246341120\n\t\t\t\t\t\t}\n\t\t\t\t\t},\n\t\t\t\t\t\"memory\": {\n\t\t\t\t\t\t\"usage\": 0,\n\t\t\t\t\t\t\"usage_peak\": 0,\n\t\t\t\t\t\t\"swap_usage\": 0,\n\t\t\t\t\t\t\"swap_usage_peak\": 0\n\t\t\t\t\t},\n\t\t\t\t\t\"network\": null,\n\t\t\t\t\t\"pid\": 0,\n\t\t\t\t\t\"processes\": 0,\n\t\t\t\t\t\"cpu\": {\n\t\t\t\t\t\t\"usage\": 0\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t\"snapshots\": null\n\t\t\t}\n\t\t]\n\t}" http_code=200
t=2021-12-24T15:37:55-0700 lvl=dbug msg="Handling API request" ip=@ method=PUT protocol=unix url="/internal/shutdown?force=false" username=sfbosch
t=2021-12-24T15:37:55-0700 lvl=info msg="Asked to shutdown by API" force=false
t=2021-12-24T15:37:55-0700 lvl=info msg="Starting shutdown sequence" signal="power failure"
t=2021-12-24T15:37:55-0700 lvl=info msg="Waiting for operations to finish" 
t=2021-12-24T15:37:56-0700 lvl=info msg="All running operations finished, shutting down" 
t=2021-12-24T15:37:56-0700 lvl=info msg="Stopping daemon storage volumes" 
t=2021-12-24T15:37:56-0700 lvl=info msg="Stopping instances" 
t=2021-12-24T15:37:56-0700 lvl=info msg="Stopping networks" 
t=2021-12-24T15:37:56-0700 lvl=dbug msg=Stop driver=bridge network=lxdbr0 project=default
t=2021-12-24T15:37:56-0700 lvl=dbug msg="Deleting firewall" driver=bridge network=lxdbr0 project=default
t=2021-12-24T15:37:56-0700 lvl=info msg="Stopping storage pools" 
t=2021-12-24T15:37:56-0700 lvl=dbug msg="Unmount started" driver=zfs pool=lxd_zfs
t=2021-12-24T15:37:56-0700 lvl=dbug msg="Unmount finished" driver=zfs pool=lxd_zfs
t=2021-12-24T15:37:56-0700 lvl=eror msg="Unable to unmount storage pool" err="Failed to run: zpool export zfs_lxd: cannot export 'zfs_lxd': pool is busy" pool=lxd_zfs
t=2021-12-24T15:37:56-0700 lvl=dbug msg="Cancel ongoing or future gRPC connection attempts" 
t=2021-12-24T15:37:56-0700 lvl=info msg="Closing the database" 
t=2021-12-24T15:37:56-0700 lvl=info msg="Stop database gateway" 
t=2021-12-24T15:37:56-0700 lvl=info msg="Stopping REST API handler:" 
t=2021-12-24T15:37:56-0700 lvl=info msg=" - closing socket" socket=/var/lib/lxd/unix.socket
t=2021-12-24T15:37:56-0700 lvl=info msg="Stopping /dev/lxd handler:" 
t=2021-12-24T15:37:56-0700 lvl=info msg=" - closing socket" socket=/var/lib/lxd/devlxd/sock
t=2021-12-24T15:37:56-0700 lvl=info msg="Unmounting temporary filesystems" 
t=2021-12-24T15:37:56-0700 lvl=info msg="Done unmounting temporary filesystems" 
t=2021-12-24T15:37:56-0700 lvl=dbug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" http_code=200

cemzafer · December 25, 2021, 7:43am

Hi,
Humm, in your first post it looks like your zfs pool name is zfs_lxd, but the error you post lately looking for the zfs name lxd_zfs.
Regards.

pajot · December 25, 2021, 9:06am

That’s really odd. If you look at my original post, you’ll see that the pool internal label was zfs_lxd, and all I did was reimport it using that one.

Still, it seems mad that this should cause lxd to hang so hard. If the pool doesn’t exist, then I would expect lxd to tell me so gracefully and give up.

EDIT: I don’t think this was the problem. I renamed the pool to lxd_zfs and tried to start lxd, and this is what I get in the log output:

t=2021-12-25T02:15:25-0700 lvl=info msg="Initializing storage pools" 
t=2021-12-25T02:15:25-0700 lvl=dbug msg="Initializing and checking storage pool" pool=lxd_zfs
t=2021-12-25T02:15:25-0700 lvl=dbug msg="Mount started" driver=zfs pool=lxd_zfs
t=2021-12-25T02:15:25-0700 lvl=dbug msg="Mount finished" driver=zfs pool=lxd_zfs
t=2021-12-25T02:15:25-0700 lvl=eror msg="Failed to start the daemon" err="Failed initializing storage pool \"lxd_zfs\": Failed to run: zpool import zfs_lxd: cannot import 'zfs_lxd': no such pool available"
t=2021-12-25T02:15:25-0700 lvl=info msg="Starting shutdown sequence" signal=interrupt
t=2021-12-25T02:15:25-0700 lvl=dbug msg="Cancel ongoing or future gRPC connection attempts" 
t=2021-12-25T02:15:25-0700 lvl=info msg="Closing the database" 
t=2021-12-25T02:15:25-0700 lvl=info msg="Stop database gateway" 
t=2021-12-25T02:15:25-0700 lvl=info msg="Stopping REST API handler:" 
t=2021-12-25T02:15:25-0700 lvl=info msg=" - closing socket" socket=/var/lib/lxd/unix.socket
t=2021-12-25T02:15:25-0700 lvl=info msg="Stopping /dev/lxd handler:" 
t=2021-12-25T02:15:25-0700 lvl=info msg=" - closing socket" socket=/var/lib/lxd/devlxd/sock
t=2021-12-25T02:15:25-0700 lvl=info msg="Unmounting temporary filesystems" 
t=2021-12-25T02:15:25-0700 lvl=info msg="Done unmounting temporary filesystems" 
t=2021-12-25T02:15:25-0700 lvl=info msg="Daemon stopped" 
Error: Failed initializing storage pool "lxd_zfs": Failed to run: zpool import zfs_lxd: cannot import 'zfs_lxd': no such pool available

Which is the behaviour I would expect. ‘zfs_lxd’ was the name of the original pool. ‘lxd_zfs’ must be a label/identifier for something else, perhaps the lxd internal name for the pool.

cemzafer · December 25, 2021, 9:43am

Can you post the lxc storage ls and lxc storage show lxd_zfs?
Regards

pajot · December 25, 2021, 9:48am

Sure, here it is:

$ lxc storage ls
±--------±-------------±-------±--------±--------+
| NAME | BESCHREIBUNG | DRIVER | SOURCE | USED BY |
±--------±-------------±-------±--------±--------+
| lxd_zfs | | zfs | zfs_lxd | 6 |
±--------±-------------±-------±--------±--------+

$ lxc storage show lxd_zfs
config:
  source: zfs_lxd
  volatile.initial_source: zfs_lxd
  zfs.pool_name: zfs_lxd
description: ""
name: lxd_zfs
driver: zfs
used_by:
- /1.0/images/874c26d35039097edd5b2063b679d839988d1dcdb2c039ba9aa9c41a532ab2df
- /1.0/images/8d87094883e69a195a8af8f988ba517d1ccc02b90b4ff984c97dc1e76bbe199d
- /1.0/instances/gentoo-PG-C01
- /1.0/instances/gentoo-WS-C01
- /1.0/instances/ubuntu-PG-C01
- /1.0/profiles/default
status: Created
locations:
- none

Here you can see what is happening. The zfs pool name and the lxd storage name are different.

One problem is that none of the debug output we have looked is providing any clues as to why lxd is blocking on start up and consuming all that CPU.

Here’s a listing of the file descriptors in proc, perhaps there’s one I could attach to to get some insight:

# LANG=C ls -liah /proc/16425/fd/   
total 0
79765 dr-x------ 2 root root  0 Dec 25 03:00 .
65274 dr-xr-xr-x 9 root root  0 Dec 25 02:43 ..
79995 lrwx------ 1 root root 64 Dec 25 03:00 0 -> /dev/pts/3
79996 l-wx------ 1 root root 64 Dec 25 03:00 1 -> 'pipe:[63403]'
80005 l-wx------ 1 root root 64 Dec 25 03:00 10 -> 'pipe:[64511]'
80006 lr-x------ 1 root root 64 Dec 25 03:00 11 -> 'pipe:[64512]'
80007 l-wx------ 1 root root 64 Dec 25 03:00 12 -> 'pipe:[64512]'
80008 lrwx------ 1 root root 64 Dec 25 03:00 13 -> 'anon_inode:[eventfd]'
80009 lrwx------ 1 root root 64 Dec 25 03:00 14 -> 'socket:[66562]'
80010 lr-x------ 1 root root 64 Dec 25 03:00 15 -> /dev/null
80011 lrwx------ 1 root root 64 Dec 25 03:00 16 -> 'socket:[66564]'
80012 lrwx------ 1 root root 64 Dec 25 03:00 17 -> 'socket:[66565]'
80013 lrwx------ 1 root root 64 Dec 25 03:00 18 -> 'socket:[66567]'
80014 lrwx------ 1 root root 64 Dec 25 03:00 19 -> 'socket:[65999]'
79997 l-wx------ 1 root root 64 Dec 25 03:00 2 -> 'pipe:[63403]'
80015 l-wx------ 1 root root 64 Dec 25 03:00 20 -> /var/lib/lxd/database/global/open-1
80016 lrwx------ 1 root root 64 Dec 25 03:00 21 -> 'anon_inode:[eventfd]'
80017 l-wx------ 1 root root 64 Dec 25 03:00 22 -> /var/lib/lxd/database/global/open-2
80018 l-wx------ 1 root root 64 Dec 25 03:00 23 -> /var/lib/lxd/database/global/open-3
80019 lr-x------ 1 root root 64 Dec 25 03:00 24 -> anon_inode:inotify
80020 lrwx------ 1 root root 64 Dec 25 03:00 25 -> 'socket:[66228]'
80021 lrwx------ 1 root root 64 Dec 25 03:00 26 -> 'anon_inode:[eventpoll]'
80022 lr-x------ 1 root root 64 Dec 25 03:00 27 -> 'pipe:[66674]'
80023 l-wx------ 1 root root 64 Dec 25 03:00 28 -> 'pipe:[66674]'
79998 lrwx------ 1 root root 64 Dec 25 03:00 3 -> /var/lib/lxd/database/local.db
80024 lrwx------ 1 root root 64 Dec 25 03:00 32 -> 'socket:[65480]'
80025 l-wx------ 1 root root 64 Dec 25 03:00 34 -> /var/log/lxd/gentoo-PG-C01/lxc.log.old
79999 lrwx------ 1 root root 64 Dec 25 03:00 4 -> 'anon_inode:[eventpoll]'
80000 lr-x------ 1 root root 64 Dec 25 03:00 5 -> 'pipe:[65996]'
80001 lr-x------ 1 root root 64 Dec 25 03:00 6 -> /dev/null
80002 l-wx------ 1 root root 64 Dec 25 03:00 7 -> 'pipe:[65996]'
80003 lrwx------ 1 root root 64 Dec 25 03:00 8 -> 'anon_inode:[eventpoll]'
80004 lr-x------ 1 root root 64 Dec 25 03:00 9 -> 'pipe:[64511]'

cemzafer · December 25, 2021, 11:58am

Are you sure that process is lxd? Can you check it out with the top command?
Regards.

pajot · December 27, 2021, 8:47am

Hi Cem, yes, I am sure it’s lxd:

top - 01:45:26 up 11:11,  6 users,  load average: 1.07, 0.86, 0.71
Tasks: 255 total,   1 running, 254 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.6 us, 25.0 sy,  0.0 ni, 73.4 id,  0.0 wa,  0.0 hi,  0.9 si,  0.0 st
MiB Mem :  31985.2 total,  22282.9 free,   2268.4 used,   7433.9 buff/cache
MiB Swap:  32044.0 total,  32044.0 free,      0.0 used.  29087.1 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                        
20195 root      20   0 1890284 160120  30160 S 100.0   0.5   2:28.73 lxd

If I can’t fix this soon, I will have to reinit my lxd, provided that I can figure out how to rescue my containers.

cemzafer · December 27, 2021, 9:02am

If you access all your containers then export it and import again after reinitialize the lxd.
lxc export --optimized-storage --instance-only --compression=lzma <container> <filename>
Regards.

pajot · December 27, 2021, 5:29pm

@cemzafer Unfortunately, that only works if lxd is running. Right now, my lxd runs, but consumes 100% CPU. While I can list the containers, so far I haven’t been able to do anything else.

4.21 version does not include the patch to provide a progress indicator (see https://github.com/lxc/lxd/issues/9711) and exports take a long time. I will have no way to check whether the export is proceeding as it should.

@stgraber I have tried following the advice given here (“Lxc commands hang. Containers are not running”) and the query of the config table yields no output. I fear my lxd installation has been damaged. How can I recover it? EDIT: Okay, I see that that only applies to clusters; this was never run as a cluster. My next question would be: How can I recover the instances without having to run lxd?

(I should mention that this install originally used 4.0.8, the long-term support version. In an effort to get things working I “upgraded” to 4.21; that hasn’t solved anything so I’ve since reverted to 4.0.8. I’ve now backed up the entire lxd directory (with lxd stopped) using rsync so that I can troubleshoot safely.)

pajot · December 27, 2021, 9:38pm

I was advised by @stgraber on IRC that processes which do not respond properly to SIGKILL are almost always an indication of a kernel problem.

I tried launching lxd through strace, like so:

strace -o lxd_trace -ff lxd --debug --group lxd

To my surprise, lxd started normally and all the containers did as well. Why? I don’t know for certain, but between yesterday and today, I rebuilt the kernel once and updated (as part of a normal OS update) zfs and the zfs kernel module.

Now everything works again. I’m sorry I don’t have a more specific cause to report.

cemzafer · December 28, 2021, 8:23pm

Hi @pajot,

By the way I’m just curious which kernel and zfs versions used before and after the problem.
Thanks.

pajot · December 28, 2021, 10:42pm

Hi Cem -

The kernel hasn’t changed, it was always 5.4.156:

Linux t470 5.4.156-gentoo #7 SMP Mon Dec 27 14:33:44 MST 2021 x86_64 Intel(R) Core(TM) i7-7600U CPU @ 2.80GHz GenuineIntel GNU/Linux

However, I’ve made incremental changes to the config and rebuilt it a number of times while trying to fix this problem. Unfortunately I wasn’t keeping careful track of the changes so I can’t give more information here.

The zfs userspace utilities and kernel modules were version 2.0.6 with the kernel blocking problem, and 2.0.7 currently (working).

I am pretty confident that the problem was caused by zfs.

cemzafer · December 29, 2021, 8:54am

Thanks for the information.