Error: LXD is shutting down - Not closing bash in an instance

michacassola · October 23, 2022, 5:52pm

I have scripts to launch and delete containers as I need to do custom stuff.

In both scripts I do the following, because I implement and remove NAT rules:

nft -f /etc/nftables.conf
systemctl reload snap.lxd.daemon  # To make lxd load the rules into nftables

On a host machine (where I run my compile-scripts which run in LXD in LXD) I just had the following error after closing a secondary SSH connection to the host were lxc exec new2 bash was running:

On the other SSH connection to the host:

lxc exec new1 bash
Error: LXD is shutting down

Then I did this:

lxc operation list
+--------------------------------------+-----------+-------------------+---------+------------+----------------------+
|                  ID                  |   TYPE    |    DESCRIPTION    | STATUS  | CANCELABLE |       CREATED        |
+--------------------------------------+-----------+-------------------+---------+------------+----------------------+
| 5c52886a-8259-4a2c-8ff8-2c62e5bed535 | WEBSOCKET | Executing command | RUNNING | NO         | 2022/10/23 17:06 UTC |
+--------------------------------------+-----------+-------------------+---------+------------+----------------------+

And this:

lxc operation show 5c52886a-8259-4a2c-8ff8-2c62e5bed535

id: 5c52886a-8259-4a2c-8ff8-2c62e5bed535
class: websocket
description: Executing command
created_at: 2022-10-23T17:06:44.135603561Z
updated_at: 2022-10-23T17:06:44.135603561Z
status: Running
status_code: 103
resources:
  containers:
  - /1.0/containers/new2
  instances:
  - /1.0/instances/new2
metadata:
  command:
  - bash
  environment:
    HOME: /root
    LANG: C.UTF-8
    PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
    TERM: xterm-256color
    USER: root
  fds:
    "0": 4f22278552fd4f88f4d6d9f844551c944a690506c2fac4abee3cff42724bda94
    control: 6a85db681a7c4b9bc5a639636ce2408aca70af540d118c99c5a5206657f1c1c9
  interactive: true
may_cancel: false
err: ""
location: none

And below is the output of doing

sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon
sudo tail -f /var/snap/lxd/common/lxd/logs/lxd.log

and then running lxc exec new1 bash.

Is there a better way so I don’t get these minutes long outages?
Because after some minutes it resovles on it’s own.

time="2022-10-23T17:06:26Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
time="2022-10-23T17:06:26Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
time="2022-10-23T17:32:46Z" level=warning msg="Ignoring signal, shutdown already in progress" signal=terminated
time="2022-10-23T17:35:51Z" level=warning msg="Ignoring signal, shutdown already in progress" signal=terminated
tail: /var/snap/lxd/common/lxd/logs/lxd.log: file truncated
time="2022-10-23T17:36:23Z" level=info msg="LXD is starting" mode=normal path=/var/snap/lxd/common/lxd version=5.6
time="2022-10-23T17:36:23Z" level=info msg="Kernel uid/gid map:"
time="2022-10-23T17:36:23Z" level=info msg=" - u 0 0 4294967295"
time="2022-10-23T17:36:23Z" level=info msg=" - g 0 0 4294967295"
time="2022-10-23T17:36:23Z" level=info msg="Configured LXD uid/gid map:"
time="2022-10-23T17:36:23Z" level=info msg=" - u 0 1000000 1000000000"
time="2022-10-23T17:36:23Z" level=info msg=" - g 0 1000000 1000000000"
time="2022-10-23T17:36:23Z" level=info msg="Kernel features:"
time="2022-10-23T17:36:23Z" level=info msg=" - closing multiple file descriptors efficiently: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - netnsid-based network retrieval: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - pidfds: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - core scheduling: no"
time="2022-10-23T17:36:23Z" level=info msg=" - uevent injection: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - seccomp listener: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - seccomp listener continue syscalls: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - seccomp listener add file descriptors: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - attach to namespaces via pidfds: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - safe native terminal allocation : yes"
time="2022-10-23T17:36:23Z" level=info msg=" - unprivileged file capabilities: yes"
time="2022-10-23T17:36:23Z" level=info msg=" - cgroup layout: cgroup2"
time="2022-10-23T17:36:23Z" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
time="2022-10-23T17:36:23Z" level=info msg=" - shiftfs support: disabled"
time="2022-10-23T17:36:23Z" level=info msg=" - idmapped mounts kernel support: yes"
time="2022-10-23T17:36:23Z" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
time="2022-10-23T17:36:23Z" level=info msg="Initializing local database"
time="2022-10-23T17:36:23Z" level=debug msg="Refreshing local trusted certificate cache"
time="2022-10-23T17:36:23Z" level=info msg="Set client certificate to server certificate" fingerprint=6391a616baa0dda1f217b235e974bf17d96603e446cf400d1ce412363bd52c77
time="2022-10-23T17:36:23Z" level=debug msg="Initializing database gateway"
time="2022-10-23T17:36:23Z" level=info msg="Starting database node" id=1 local=1 role=voter
time="2022-10-23T17:36:25Z" level=info msg="Loading daemon configuration"
time="2022-10-23T17:36:25Z" level=info msg="Binding socket" socket=/var/snap/lxd/common/lxd/devlxd/sock type="devlxd socket"
time="2022-10-23T17:36:25Z" level=info msg="Binding socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket type="REST API Unix socket"
time="2022-10-23T17:36:25Z" level=info msg="Binding socket" socket="[::]:8444" type="REST API TCP socket"
time="2022-10-23T17:36:25Z" level=info msg="Initializing global database"
time="2022-10-23T17:36:25Z" level=info msg="Connecting to global database"
time="2022-10-23T17:36:25Z" level=debug msg="Dqlite: attempt 1: server 1: connected"
time="2022-10-23T17:36:25Z" level=info msg="Connected to global database"
time="2022-10-23T17:36:25Z" level=info msg="Initialized global database"
time="2022-10-23T17:36:25Z" level=info msg="Firewall loaded driver" driver=nftables
time="2022-10-23T17:36:25Z" level=info msg="Initializing storage pools"
time="2022-10-23T17:36:25Z" level=debug msg="Initializing storage pool" pool=new2-pool
time="2022-10-23T17:36:25Z" level=debug msg="Mount started" driver=zfs pool=new2-pool
time="2022-10-23T17:36:25Z" level=debug msg="Mount finished" driver=zfs pool=new2-pool
time="2022-10-23T17:36:25Z" level=info msg="Initialized storage pool" pool=new2-pool
time="2022-10-23T17:36:25Z" level=debug msg="Initializing storage pool" pool=base-pool
time="2022-10-23T17:36:25Z" level=debug msg="Mount started" driver=zfs pool=base-pool
time="2022-10-23T17:36:26Z" level=debug msg="Mount finished" driver=zfs pool=base-pool
time="2022-10-23T17:36:26Z" level=info msg="Initialized storage pool" pool=base-pool
time="2022-10-23T17:36:26Z" level=debug msg="Initializing storage pool" pool=noco2-pool
time="2022-10-23T17:36:26Z" level=debug msg="Mount started" driver=zfs pool=noco2-pool
time="2022-10-23T17:36:26Z" level=debug msg="Mount finished" driver=zfs pool=noco2-pool
time="2022-10-23T17:36:26Z" level=info msg="Initialized storage pool" pool=noco2-pool
time="2022-10-23T17:36:26Z" level=debug msg="Initializing storage pool" pool=qyc-auto-pool
time="2022-10-23T17:36:26Z" level=debug msg="Mount started" driver=zfs pool=qyc-auto-pool
time="2022-10-23T17:36:27Z" level=debug msg="Mount finished" driver=zfs pool=qyc-auto-pool
time="2022-10-23T17:36:27Z" level=info msg="Initialized storage pool" pool=qyc-auto-pool
time="2022-10-23T17:36:27Z" level=debug msg="Initializing storage pool" pool=new1-pool
time="2022-10-23T17:36:27Z" level=debug msg="Mount started" driver=zfs pool=new1-pool
time="2022-10-23T17:36:27Z" level=debug msg="Mount finished" driver=zfs pool=new1-pool
time="2022-10-23T17:36:27Z" level=info msg="Initialized storage pool" pool=new1-pool
time="2022-10-23T17:36:27Z" level=info msg="All storage pools initialized"
time="2022-10-23T17:36:27Z" level=info msg="Initializing daemon storage mounts"
time="2022-10-23T17:36:27Z" level=info msg="Initializing networks"
time="2022-10-23T17:36:27Z" level=debug msg=Start driver=bridge network=lxdbr0 project=default
time="2022-10-23T17:36:27Z" level=debug msg="Setting up network" driver=bridge network=lxdbr0 project=default
time="2022-10-23T17:36:27Z" level=debug msg="Stable MAC generated" driver=bridge hwAddr="00:16:3e:9a:45:8e" network=lxdbr0 project=default seed=6391a616baa0dda1f217b235e974bf17d96603e446cf400d1ce412363bd52c77.0.1
time="2022-10-23T17:36:27Z" level=debug msg="Clearing firewall" driver=bridge network=lxdbr0 project=default
time="2022-10-23T17:36:28Z" level=debug msg="Setting up firewall" driver=bridge network=lxdbr0 project=default
time="2022-10-23T17:36:28Z" level=info msg="Initialized network" name=lxdbr0 project=default
time="2022-10-23T17:36:28Z" level=info msg="All networks initialized"
time="2022-10-23T17:36:28Z" level=debug msg="New operation" class=task description="Pruning leftover image files" operation=4985a979-c35d-42f0-ab71-52105272b9eb project=
time="2022-10-23T17:36:28Z" level=debug msg="Acquiring image task lock"
time="2022-10-23T17:36:28Z" level=debug msg="Acquired image task lock"
time="2022-10-23T17:36:28Z" level=info msg="Pruning leftover image files"
time="2022-10-23T17:36:28Z" level=debug msg="Started operation" class=task description="Pruning leftover image files" operation=4985a979-c35d-42f0-ab71-52105272b9eb project=
time="2022-10-23T17:36:28Z" level=debug msg="Success for operation" class=task description="Pruning leftover image files" operation=4985a979-c35d-42f0-ab71-52105272b9eb project=
time="2022-10-23T17:36:28Z" level=info msg="Done pruning leftover image files"
time="2022-10-23T17:36:28Z" level=info msg="Starting device monitor"
time="2022-10-23T17:36:28Z" level=debug msg="Initialized filesystem monitor" path=/dev
time="2022-10-23T17:36:28Z" level=debug msg="Registering running instances"
time="2022-10-23T17:36:28Z" level=debug msg="Initialising mounted disk ref counter" device=root driver=disk instance=new1 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountInstance started" instance=new1 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountInstance finished" instance=new1 project=default
time="2022-10-23T17:36:28Z" level=debug msg="Initialising mounted disk ref counter" device=mysql-disk driver=disk instance=new1 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountCustomVolume started" project=default volName=mysql-volume
time="2022-10-23T17:36:28Z" level=debug msg="MountCustomVolume finished" project=default volName=mysql-volume
time="2022-10-23T17:36:28Z" level=debug msg="Initialising mounted disk ref counter" device=root driver=disk instance=new2 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountInstance started" instance=new2 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountInstance finished" instance=new2 project=default
time="2022-10-23T17:36:28Z" level=debug msg="Initialising mounted disk ref counter" device=mysql-disk driver=disk instance=new2 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountCustomVolume started" project=default volName=mysql-volume
time="2022-10-23T17:36:28Z" level=debug msg="MountCustomVolume finished" project=default volName=mysql-volume
time="2022-10-23T17:36:28Z" level=debug msg="Initialising mounted disk ref counter" device=root driver=disk instance=noco2 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountInstance started" instance=noco2 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountInstance finished" instance=noco2 project=default
time="2022-10-23T17:36:28Z" level=debug msg="Initialising mounted disk ref counter" device=mysql-disk driver=disk instance=noco2 project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountCustomVolume started" project=default volName=mysql-volume
time="2022-10-23T17:36:28Z" level=debug msg="MountCustomVolume finished" project=default volName=mysql-volume
time="2022-10-23T17:36:28Z" level=debug msg="Initialising mounted disk ref counter" device=root driver=disk instance=qyc-auto project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountInstance started" instance=qyc-auto project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountInstance finished" instance=qyc-auto project=default
time="2022-10-23T17:36:28Z" level=debug msg="Initialising mounted disk ref counter" device=mysql-disk driver=disk instance=qyc-auto project=default
time="2022-10-23T17:36:28Z" level=debug msg="MountCustomVolume started" project=default volName=mysql-volume
time="2022-10-23T17:36:28Z" level=debug msg="MountCustomVolume finished" project=default volName=mysql-volume
time="2022-10-23T17:36:28Z" level=info msg="Started seccomp handler" path=/var/snap/lxd/common/lxd/seccomp.socket
time="2022-10-23T17:36:28Z" level=debug msg="Refreshing trusted certificate cache"
time="2022-10-23T17:36:28Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=root
time="2022-10-23T17:36:28Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"core.https_address\": \"[::]:8444\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\",\n\t\t\t\t\"instance_get_full\",\n\t\t\t\t\"qemu_metrics\",\n\t\t\t\t\"gpu_mig_uuid\",\n\t\t\t\t\"event_project\",\n\t\t\t\t\"clustering_evacuation_live\",\n\t\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\t\"network_state_ovn\",\n\t\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\t\"image_restrictions\",\n\t\t\t\t\"storage_zfs_export\",\n\t\t\t\t\"network_dns_records\",\n\t\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\t\"network_acl_log\",\n\t\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\t\"instance_snapshot_never\",\n\t\t\t\t\"certificate_token\",\n\t\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\t\"event_hub\",\n\t\t\t\t\"agent_nic_config\",\n\t\t\t\t\"projects_restricted_intercept\",\n\t\t\t\t\"metrics_authentication\",\n\t\t\t\t\"images_target_project\",\n\t\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\t\"storage_volume_state_total\",\n\t\t\t\t\"instance_file_head\",\n\t\t\t\t\"instances_nic_host_name\",\n\t\t\t\t\"image_copy_profile\",\n\t\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\t\"resources_pci_vpd\",\n\t\t\t\t\"qemu_raw_conf\",\n\t\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\t\"network_load_balancer\",\n\t\t\t\t\"vsock_api\",\n\t\t\t\t\"instance_ready_state\",\n\t\t\t\t\"network_bgp_holdtime\",\n\t\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\t\"storage_buckets\",\n\t\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\t\"storage_buckets_local\",\n\t\t\t\t\"loki\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"95.111.234.170:8444\",\n\t\t\t\t\t\"[2a02:c207:3005:8614::1]:8444\",\n\t\t\t\t\t\"10.0.0.1:8444\",\n\t\t\t\t\t\"[2a02:c207:3005:8614:9999:0:2:1]:8444\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nTHISHASBEENALTERED\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"6391a616baa0dda1f217b235e974bf17d96603e446cf400d1ce412363bd52c77\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"5.0.1\",\n\t\t\t\t\"firewall\": \"nftables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"idmapped_mounts\": \"true\",\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"true\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"5.15.0-52-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Ubuntu\",\n\t\t\t\t\"os_version\": \"22.04\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\t\"server_name\": \"s2\",\n\t\t\t\t\"server_pid\": 6284,\n\t\t\t\t\"server_version\": \"5.6\",\n\t\t\t\t\"storage\": \"zfs\",\n\t\t\t\t\"storage_version\": \"2.1.4-0ubuntu0.1\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"cephobject\",\n\t\t\t\t\t\t\"Version\": \"15.2.16\",\n\t\t\t\t\t\t\"Remote\": true\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"zfs\",\n\t\t\t\t\t\t\"Version\": \"2.1.4-0ubuntu0.1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"btrfs\",\n\t\t\t\t\t\t\"Version\": \"5.4.1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"ceph\",\n\t\t\t\t\t\t\"Version\": \"15.2.16\",\n\t\t\t\t\t\t\"Remote\": true\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"cephfs\",\n\t\t\t\t\t\t\"Version\": \"15.2.16\",\n\t\t\t\t\t\t\"Remote\": true\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
time="2022-10-23T17:36:28Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/internal/ready username=root
time="2022-10-23T17:36:28Z" level=debug msg="Error Response\n\t{\n\t\t\"type\": \"error\",\n\t\t\"status\": \"\",\n\t\t\"status_code\": 0,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 503,\n\t\t\"error\": \"LXD daemon not ready yet\",\n\t\t\"metadata\": null\n\t}" http_code=503
time="2022-10-23T17:36:28Z" level=debug msg="New operation" class=task description="Cleaning up expired images" operation=e993d7f2-8b89-4a90-926a-ea261214704c project=
time="2022-10-23T17:36:28Z" level=debug msg="Acquiring image task lock"
time="2022-10-23T17:36:28Z" level=debug msg="Acquired image task lock"
time="2022-10-23T17:36:28Z" level=info msg="Pruning expired images"
time="2022-10-23T17:36:28Z" level=debug msg="Started operation" class=task description="Cleaning up expired images" operation=e993d7f2-8b89-4a90-926a-ea261214704c project=
time="2022-10-23T17:36:28Z" level=debug msg="Success for operation" class=task description="Cleaning up expired images" operation=e993d7f2-8b89-4a90-926a-ea261214704c project=
time="2022-10-23T17:36:28Z" level=info msg="Done pruning expired images"
time="2022-10-23T17:36:28Z" level=debug msg="New operation" class=task description="Cleaning up expired instance backups" operation=598df721-70b5-4983-be67-e1f233980ed8 project=
time="2022-10-23T17:36:28Z" level=info msg="Pruning expired instance backups"
time="2022-10-23T17:36:28Z" level=debug msg="Started operation" class=task description="Cleaning up expired instance backups" operation=598df721-70b5-4983-be67-e1f233980ed8 project=
time="2022-10-23T17:36:28Z" level=debug msg="Success for operation" class=task description="Cleaning up expired instance backups" operation=598df721-70b5-4983-be67-e1f233980ed8 project=
time="2022-10-23T17:36:28Z" level=info msg="Done pruning expired instance backups"
time="2022-10-23T17:36:28Z" level=debug msg="New operation" class=task description="Updating images" operation=9a8e8974-7abc-421e-ad7f-8fb5d237347f project=
time="2022-10-23T17:36:28Z" level=debug msg="Acquiring image task lock"
time="2022-10-23T17:36:28Z" level=debug msg="Acquired image task lock"
time="2022-10-23T17:36:28Z" level=info msg="Updating images"
time="2022-10-23T17:36:28Z" level=debug msg="Started operation" class=task description="Updating images" operation=9a8e8974-7abc-421e-ad7f-8fb5d237347f project=
time="2022-10-23T17:36:28Z" level=debug msg="New operation" class=task description="Pruning resolved warnings" operation=b0457fe2-afb5-43bb-ab11-5140c40988f4 project=
time="2022-10-23T17:36:28Z" level=info msg="Pruning resolved warnings"
time="2022-10-23T17:36:28Z" level=debug msg="Started operation" class=task description="Pruning resolved warnings" operation=b0457fe2-afb5-43bb-ab11-5140c40988f4 project=
time="2022-10-23T17:36:28Z" level=info msg="Done pruning resolved warnings"
time="2022-10-23T17:36:28Z" level=debug msg="Success for operation" class=task description="Pruning resolved warnings" operation=b0457fe2-afb5-43bb-ab11-5140c40988f4 project=
time="2022-10-23T17:36:28Z" level=debug msg="New operation" class=task description="Expiring log files" operation=b260806c-2084-4cf6-b2ec-fe65aa2999fd project=
time="2022-10-23T17:36:28Z" level=info msg="Expiring log files"
time="2022-10-23T17:36:28Z" level=debug msg="Started operation" class=task description="Expiring log files" operation=b260806c-2084-4cf6-b2ec-fe65aa2999fd project=
time="2022-10-23T17:36:28Z" level=debug msg="Success for operation" class=task description="Expiring log files" operation=b260806c-2084-4cf6-b2ec-fe65aa2999fd project=
time="2022-10-23T17:36:28Z" level=info msg="Done expiring log files"
time="2022-10-23T17:36:28Z" level=debug msg="New operation" class=task description="Updating instance types" operation=bb549e9f-2dcb-4371-a5a8-96ab4b019953 project=
time="2022-10-23T17:36:28Z" level=info msg="Updating instance types"
time="2022-10-23T17:36:28Z" level=debug msg="Started operation" class=task description="Updating instance types" operation=bb549e9f-2dcb-4371-a5a8-96ab4b019953 project=
time="2022-10-23T17:36:28Z" level=debug msg="Processing image" alias=20.04 fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69 protocol=simplestreams server="https://cloud-images.ubuntu.com/releases"
time="2022-10-23T17:36:28Z" level=debug msg="Connecting to a remote simplestreams server" URL="https://cloud-images.ubuntu.com/releases"
time="2022-10-23T17:36:28Z" level=info msg="Daemon started"
time="2022-10-23T17:36:29Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/1.0 username=root
time="2022-10-23T17:36:29Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"config\": {\n\t\t\t\t\"core.https_address\": \"[::]:8444\",\n\t\t\t\t\"core.trust_password\": true\n\t\t\t},\n\t\t\t\"api_extensions\": [\n\t\t\t\t\"storage_zfs_remove_snapshots\",\n\t\t\t\t\"container_host_shutdown_timeout\",\n\t\t\t\t\"container_stop_priority\",\n\t\t\t\t\"container_syscall_filtering\",\n\t\t\t\t\"auth_pki\",\n\t\t\t\t\"container_last_used_at\",\n\t\t\t\t\"etag\",\n\t\t\t\t\"patch\",\n\t\t\t\t\"usb_devices\",\n\t\t\t\t\"https_allowed_credentials\",\n\t\t\t\t\"image_compression_algorithm\",\n\t\t\t\t\"directory_manipulation\",\n\t\t\t\t\"container_cpu_time\",\n\t\t\t\t\"storage_zfs_use_refquota\",\n\t\t\t\t\"storage_lvm_mount_options\",\n\t\t\t\t\"network\",\n\t\t\t\t\"profile_usedby\",\n\t\t\t\t\"container_push\",\n\t\t\t\t\"container_exec_recording\",\n\t\t\t\t\"certificate_update\",\n\t\t\t\t\"container_exec_signal_handling\",\n\t\t\t\t\"gpu_devices\",\n\t\t\t\t\"container_image_properties\",\n\t\t\t\t\"migration_progress\",\n\t\t\t\t\"id_map\",\n\t\t\t\t\"network_firewall_filtering\",\n\t\t\t\t\"network_routes\",\n\t\t\t\t\"storage\",\n\t\t\t\t\"file_delete\",\n\t\t\t\t\"file_append\",\n\t\t\t\t\"network_dhcp_expiry\",\n\t\t\t\t\"storage_lvm_vg_rename\",\n\t\t\t\t\"storage_lvm_thinpool_rename\",\n\t\t\t\t\"network_vlan\",\n\t\t\t\t\"image_create_aliases\",\n\t\t\t\t\"container_stateless_copy\",\n\t\t\t\t\"container_only_migration\",\n\t\t\t\t\"storage_zfs_clone_copy\",\n\t\t\t\t\"unix_device_rename\",\n\t\t\t\t\"storage_lvm_use_thinpool\",\n\t\t\t\t\"storage_rsync_bwlimit\",\n\t\t\t\t\"network_vxlan_interface\",\n\t\t\t\t\"storage_btrfs_mount_options\",\n\t\t\t\t\"entity_description\",\n\t\t\t\t\"image_force_refresh\",\n\t\t\t\t\"storage_lvm_lv_resizing\",\n\t\t\t\t\"id_map_base\",\n\t\t\t\t\"file_symlinks\",\n\t\t\t\t\"container_push_target\",\n\t\t\t\t\"network_vlan_physical\",\n\t\t\t\t\"storage_images_delete\",\n\t\t\t\t\"container_edit_metadata\",\n\t\t\t\t\"container_snapshot_stateful_migration\",\n\t\t\t\t\"storage_driver_ceph\",\n\t\t\t\t\"storage_ceph_user_name\",\n\t\t\t\t\"resource_limits\",\n\t\t\t\t\"storage_volatile_initial_source\",\n\t\t\t\t\"storage_ceph_force_osd_reuse\",\n\t\t\t\t\"storage_block_filesystem_btrfs\",\n\t\t\t\t\"resources\",\n\t\t\t\t\"kernel_limits\",\n\t\t\t\t\"storage_api_volume_rename\",\n\t\t\t\t\"macaroon_authentication\",\n\t\t\t\t\"network_sriov\",\n\t\t\t\t\"console\",\n\t\t\t\t\"restrict_devlxd\",\n\t\t\t\t\"migration_pre_copy\",\n\t\t\t\t\"infiniband\",\n\t\t\t\t\"maas_network\",\n\t\t\t\t\"devlxd_events\",\n\t\t\t\t\"proxy\",\n\t\t\t\t\"network_dhcp_gateway\",\n\t\t\t\t\"file_get_symlink\",\n\t\t\t\t\"network_leases\",\n\t\t\t\t\"unix_device_hotplug\",\n\t\t\t\t\"storage_api_local_volume_handling\",\n\t\t\t\t\"operation_description\",\n\t\t\t\t\"clustering\",\n\t\t\t\t\"event_lifecycle\",\n\t\t\t\t\"storage_api_remote_volume_handling\",\n\t\t\t\t\"nvidia_runtime\",\n\t\t\t\t\"container_mount_propagation\",\n\t\t\t\t\"container_backup\",\n\t\t\t\t\"devlxd_images\",\n\t\t\t\t\"container_local_cross_pool_handling\",\n\t\t\t\t\"proxy_unix\",\n\t\t\t\t\"proxy_udp\",\n\t\t\t\t\"clustering_join\",\n\t\t\t\t\"proxy_tcp_udp_multi_port_handling\",\n\t\t\t\t\"network_state\",\n\t\t\t\t\"proxy_unix_dac_properties\",\n\t\t\t\t\"container_protection_delete\",\n\t\t\t\t\"unix_priv_drop\",\n\t\t\t\t\"pprof_http\",\n\t\t\t\t\"proxy_haproxy_protocol\",\n\t\t\t\t\"network_hwaddr\",\n\t\t\t\t\"proxy_nat\",\n\t\t\t\t\"network_nat_order\",\n\t\t\t\t\"container_full\",\n\t\t\t\t\"candid_authentication\",\n\t\t\t\t\"backup_compression\",\n\t\t\t\t\"candid_config\",\n\t\t\t\t\"nvidia_runtime_config\",\n\t\t\t\t\"storage_api_volume_snapshots\",\n\t\t\t\t\"storage_unmapped\",\n\t\t\t\t\"projects\",\n\t\t\t\t\"candid_config_key\",\n\t\t\t\t\"network_vxlan_ttl\",\n\t\t\t\t\"container_incremental_copy\",\n\t\t\t\t\"usb_optional_vendorid\",\n\t\t\t\t\"snapshot_scheduling\",\n\t\t\t\t\"snapshot_schedule_aliases\",\n\t\t\t\t\"container_copy_project\",\n\t\t\t\t\"clustering_server_address\",\n\t\t\t\t\"clustering_image_replication\",\n\t\t\t\t\"container_protection_shift\",\n\t\t\t\t\"snapshot_expiry\",\n\t\t\t\t\"container_backup_override_pool\",\n\t\t\t\t\"snapshot_expiry_creation\",\n\t\t\t\t\"network_leases_location\",\n\t\t\t\t\"resources_cpu_socket\",\n\t\t\t\t\"resources_gpu\",\n\t\t\t\t\"resources_numa\",\n\t\t\t\t\"kernel_features\",\n\t\t\t\t\"id_map_current\",\n\t\t\t\t\"event_location\",\n\t\t\t\t\"storage_api_remote_volume_snapshots\",\n\t\t\t\t\"network_nat_address\",\n\t\t\t\t\"container_nic_routes\",\n\t\t\t\t\"rbac\",\n\t\t\t\t\"cluster_internal_copy\",\n\t\t\t\t\"seccomp_notify\",\n\t\t\t\t\"lxc_features\",\n\t\t\t\t\"container_nic_ipvlan\",\n\t\t\t\t\"network_vlan_sriov\",\n\t\t\t\t\"storage_cephfs\",\n\t\t\t\t\"container_nic_ipfilter\",\n\t\t\t\t\"resources_v2\",\n\t\t\t\t\"container_exec_user_group_cwd\",\n\t\t\t\t\"container_syscall_intercept\",\n\t\t\t\t\"container_disk_shift\",\n\t\t\t\t\"storage_shifted\",\n\t\t\t\t\"resources_infiniband\",\n\t\t\t\t\"daemon_storage\",\n\t\t\t\t\"instances\",\n\t\t\t\t\"image_types\",\n\t\t\t\t\"resources_disk_sata\",\n\t\t\t\t\"clustering_roles\",\n\t\t\t\t\"images_expiry\",\n\t\t\t\t\"resources_network_firmware\",\n\t\t\t\t\"backup_compression_algorithm\",\n\t\t\t\t\"ceph_data_pool_name\",\n\t\t\t\t\"container_syscall_intercept_mount\",\n\t\t\t\t\"compression_squashfs\",\n\t\t\t\t\"container_raw_mount\",\n\t\t\t\t\"container_nic_routed\",\n\t\t\t\t\"container_syscall_intercept_mount_fuse\",\n\t\t\t\t\"container_disk_ceph\",\n\t\t\t\t\"virtual-machines\",\n\t\t\t\t\"image_profiles\",\n\t\t\t\t\"clustering_architecture\",\n\t\t\t\t\"resources_disk_id\",\n\t\t\t\t\"storage_lvm_stripes\",\n\t\t\t\t\"vm_boot_priority\",\n\t\t\t\t\"unix_hotplug_devices\",\n\t\t\t\t\"api_filtering\",\n\t\t\t\t\"instance_nic_network\",\n\t\t\t\t\"clustering_sizing\",\n\t\t\t\t\"firewall_driver\",\n\t\t\t\t\"projects_limits\",\n\t\t\t\t\"container_syscall_intercept_hugetlbfs\",\n\t\t\t\t\"limits_hugepages\",\n\t\t\t\t\"container_nic_routed_gateway\",\n\t\t\t\t\"projects_restrictions\",\n\t\t\t\t\"custom_volume_snapshot_expiry\",\n\t\t\t\t\"volume_snapshot_scheduling\",\n\t\t\t\t\"trust_ca_certificates\",\n\t\t\t\t\"snapshot_disk_usage\",\n\t\t\t\t\"clustering_edit_roles\",\n\t\t\t\t\"container_nic_routed_host_address\",\n\t\t\t\t\"container_nic_ipvlan_gateway\",\n\t\t\t\t\"resources_usb_pci\",\n\t\t\t\t\"resources_cpu_threads_numa\",\n\t\t\t\t\"resources_cpu_core_die\",\n\t\t\t\t\"api_os\",\n\t\t\t\t\"container_nic_routed_host_table\",\n\t\t\t\t\"container_nic_ipvlan_host_table\",\n\t\t\t\t\"container_nic_ipvlan_mode\",\n\t\t\t\t\"resources_system\",\n\t\t\t\t\"images_push_relay\",\n\t\t\t\t\"network_dns_search\",\n\t\t\t\t\"container_nic_routed_limits\",\n\t\t\t\t\"instance_nic_bridged_vlan\",\n\t\t\t\t\"network_state_bond_bridge\",\n\t\t\t\t\"usedby_consistency\",\n\t\t\t\t\"custom_block_volumes\",\n\t\t\t\t\"clustering_failure_domains\",\n\t\t\t\t\"resources_gpu_mdev\",\n\t\t\t\t\"console_vga_type\",\n\t\t\t\t\"projects_limits_disk\",\n\t\t\t\t\"network_type_macvlan\",\n\t\t\t\t\"network_type_sriov\",\n\t\t\t\t\"container_syscall_intercept_bpf_devices\",\n\t\t\t\t\"network_type_ovn\",\n\t\t\t\t\"projects_networks\",\n\t\t\t\t\"projects_networks_restricted_uplinks\",\n\t\t\t\t\"custom_volume_backup\",\n\t\t\t\t\"backup_override_name\",\n\t\t\t\t\"storage_rsync_compression\",\n\t\t\t\t\"network_type_physical\",\n\t\t\t\t\"network_ovn_external_subnets\",\n\t\t\t\t\"network_ovn_nat\",\n\t\t\t\t\"network_ovn_external_routes_remove\",\n\t\t\t\t\"tpm_device_type\",\n\t\t\t\t\"storage_zfs_clone_copy_rebase\",\n\t\t\t\t\"gpu_mdev\",\n\t\t\t\t\"resources_pci_iommu\",\n\t\t\t\t\"resources_network_usb\",\n\t\t\t\t\"resources_disk_address\",\n\t\t\t\t\"network_physical_ovn_ingress_mode\",\n\t\t\t\t\"network_ovn_dhcp\",\n\t\t\t\t\"network_physical_routes_anycast\",\n\t\t\t\t\"projects_limits_instances\",\n\t\t\t\t\"network_state_vlan\",\n\t\t\t\t\"instance_nic_bridged_port_isolation\",\n\t\t\t\t\"instance_bulk_state_change\",\n\t\t\t\t\"network_gvrp\",\n\t\t\t\t\"instance_pool_move\",\n\t\t\t\t\"gpu_sriov\",\n\t\t\t\t\"pci_device_type\",\n\t\t\t\t\"storage_volume_state\",\n\t\t\t\t\"network_acl\",\n\t\t\t\t\"migration_stateful\",\n\t\t\t\t\"disk_state_quota\",\n\t\t\t\t\"storage_ceph_features\",\n\t\t\t\t\"projects_compression\",\n\t\t\t\t\"projects_images_remote_cache_expiry\",\n\t\t\t\t\"certificate_project\",\n\t\t\t\t\"network_ovn_acl\",\n\t\t\t\t\"projects_images_auto_update\",\n\t\t\t\t\"projects_restricted_cluster_target\",\n\t\t\t\t\"images_default_architecture\",\n\t\t\t\t\"network_ovn_acl_defaults\",\n\t\t\t\t\"gpu_mig\",\n\t\t\t\t\"project_usage\",\n\t\t\t\t\"network_bridge_acl\",\n\t\t\t\t\"warnings\",\n\t\t\t\t\"projects_restricted_backups_and_snapshots\",\n\t\t\t\t\"clustering_join_token\",\n\t\t\t\t\"clustering_description\",\n\t\t\t\t\"server_trusted_proxy\",\n\t\t\t\t\"clustering_update_cert\",\n\t\t\t\t\"storage_api_project\",\n\t\t\t\t\"server_instance_driver_operational\",\n\t\t\t\t\"server_supported_storage_drivers\",\n\t\t\t\t\"event_lifecycle_requestor_address\",\n\t\t\t\t\"resources_gpu_usb\",\n\t\t\t\t\"clustering_evacuation\",\n\t\t\t\t\"network_ovn_nat_address\",\n\t\t\t\t\"network_bgp\",\n\t\t\t\t\"network_forward\",\n\t\t\t\t\"custom_volume_refresh\",\n\t\t\t\t\"network_counters_errors_dropped\",\n\t\t\t\t\"metrics\",\n\t\t\t\t\"image_source_project\",\n\t\t\t\t\"clustering_config\",\n\t\t\t\t\"network_peer\",\n\t\t\t\t\"linux_sysctl\",\n\t\t\t\t\"network_dns\",\n\t\t\t\t\"ovn_nic_acceleration\",\n\t\t\t\t\"certificate_self_renewal\",\n\t\t\t\t\"instance_project_move\",\n\t\t\t\t\"storage_volume_project_move\",\n\t\t\t\t\"cloud_init\",\n\t\t\t\t\"network_dns_nat\",\n\t\t\t\t\"database_leader\",\n\t\t\t\t\"instance_all_projects\",\n\t\t\t\t\"clustering_groups\",\n\t\t\t\t\"ceph_rbd_du\",\n\t\t\t\t\"instance_get_full\",\n\t\t\t\t\"qemu_metrics\",\n\t\t\t\t\"gpu_mig_uuid\",\n\t\t\t\t\"event_project\",\n\t\t\t\t\"clustering_evacuation_live\",\n\t\t\t\t\"instance_allow_inconsistent_copy\",\n\t\t\t\t\"network_state_ovn\",\n\t\t\t\t\"storage_volume_api_filtering\",\n\t\t\t\t\"image_restrictions\",\n\t\t\t\t\"storage_zfs_export\",\n\t\t\t\t\"network_dns_records\",\n\t\t\t\t\"storage_zfs_reserve_space\",\n\t\t\t\t\"network_acl_log\",\n\t\t\t\t\"storage_zfs_blocksize\",\n\t\t\t\t\"metrics_cpu_seconds\",\n\t\t\t\t\"instance_snapshot_never\",\n\t\t\t\t\"certificate_token\",\n\t\t\t\t\"instance_nic_routed_neighbor_probe\",\n\t\t\t\t\"event_hub\",\n\t\t\t\t\"agent_nic_config\",\n\t\t\t\t\"projects_restricted_intercept\",\n\t\t\t\t\"metrics_authentication\",\n\t\t\t\t\"images_target_project\",\n\t\t\t\t\"cluster_migration_inconsistent_copy\",\n\t\t\t\t\"cluster_ovn_chassis\",\n\t\t\t\t\"container_syscall_intercept_sched_setscheduler\",\n\t\t\t\t\"storage_lvm_thinpool_metadata_size\",\n\t\t\t\t\"storage_volume_state_total\",\n\t\t\t\t\"instance_file_head\",\n\t\t\t\t\"instances_nic_host_name\",\n\t\t\t\t\"image_copy_profile\",\n\t\t\t\t\"container_syscall_intercept_sysinfo\",\n\t\t\t\t\"clustering_evacuation_mode\",\n\t\t\t\t\"resources_pci_vpd\",\n\t\t\t\t\"qemu_raw_conf\",\n\t\t\t\t\"storage_cephfs_fscache\",\n\t\t\t\t\"network_load_balancer\",\n\t\t\t\t\"vsock_api\",\n\t\t\t\t\"instance_ready_state\",\n\t\t\t\t\"network_bgp_holdtime\",\n\t\t\t\t\"storage_volumes_all_projects\",\n\t\t\t\t\"metrics_memory_oom_total\",\n\t\t\t\t\"storage_buckets\",\n\t\t\t\t\"storage_buckets_create_credentials\",\n\t\t\t\t\"metrics_cpu_effective_total\",\n\t\t\t\t\"projects_networks_restricted_access\",\n\t\t\t\t\"storage_buckets_local\",\n\t\t\t\t\"loki\"\n\t\t\t],\n\t\t\t\"api_status\": \"stable\",\n\t\t\t\"api_version\": \"1.0\",\n\t\t\t\"auth\": \"trusted\",\n\t\t\t\"public\": false,\n\t\t\t\"auth_methods\": [\n\t\t\t\t\"tls\"\n\t\t\t],\n\t\t\t\"environment\": {\n\t\t\t\t\"addresses\": [\n\t\t\t\t\t\"95.111.234.170:8444\",\n\t\t\t\t\t\"[2a02:c207:3005:8614::1]:8444\",\n\t\t\t\t\t\"10.0.0.1:8444\",\n\t\t\t\t\t\"[2a02:c207:3005:8614:9999:0:2:1]:8444\"\n\t\t\t\t],\n\t\t\t\t\"architectures\": [\n\t\t\t\t\t\"x86_64\",\n\t\t\t\t\t\"i686\"\n\t\t\t\t],\n\t\t\t\t\"certificate\": \"-----BEGIN CERTIFICATE-----\\nTHISHASBEENCHANGED\\n-----END CERTIFICATE-----\\n\",\n\t\t\t\t\"certificate_fingerprint\": \"THISHASBEENCHANGED\",\n\t\t\t\t\"driver\": \"lxc\",\n\t\t\t\t\"driver_version\": \"5.0.1\",\n\t\t\t\t\"firewall\": \"nftables\",\n\t\t\t\t\"kernel\": \"Linux\",\n\t\t\t\t\"kernel_architecture\": \"x86_64\",\n\t\t\t\t\"kernel_features\": {\n\t\t\t\t\t\"idmapped_mounts\": \"true\",\n\t\t\t\t\t\"netnsid_getifaddrs\": \"true\",\n\t\t\t\t\t\"seccomp_listener\": \"true\",\n\t\t\t\t\t\"seccomp_listener_continue\": \"true\",\n\t\t\t\t\t\"shiftfs\": \"false\",\n\t\t\t\t\t\"uevent_injection\": \"true\",\n\t\t\t\t\t\"unpriv_fscaps\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"kernel_version\": \"5.15.0-52-generic\",\n\t\t\t\t\"lxc_features\": {\n\t\t\t\t\t\"cgroup2\": \"true\",\n\t\t\t\t\t\"core_scheduling\": \"true\",\n\t\t\t\t\t\"devpts_fd\": \"true\",\n\t\t\t\t\t\"idmapped_mounts_v2\": \"true\",\n\t\t\t\t\t\"mount_injection_file\": \"true\",\n\t\t\t\t\t\"network_gateway_device_route\": \"true\",\n\t\t\t\t\t\"network_ipvlan\": \"true\",\n\t\t\t\t\t\"network_l2proxy\": \"true\",\n\t\t\t\t\t\"network_phys_macvlan_mtu\": \"true\",\n\t\t\t\t\t\"network_veth_router\": \"true\",\n\t\t\t\t\t\"pidfd\": \"true\",\n\t\t\t\t\t\"seccomp_allow_deny_syntax\": \"true\",\n\t\t\t\t\t\"seccomp_notify\": \"true\",\n\t\t\t\t\t\"seccomp_proxy_send_notify_fd\": \"true\"\n\t\t\t\t},\n\t\t\t\t\"os_name\": \"Ubuntu\",\n\t\t\t\t\"os_version\": \"22.04\",\n\t\t\t\t\"project\": \"default\",\n\t\t\t\t\"server\": \"lxd\",\n\t\t\t\t\"server_clustered\": false,\n\t\t\t\t\"server_event_mode\": \"full-mesh\",\n\t\t\t\t\"server_name\": \"s2\",\n\t\t\t\t\"server_pid\": 6284,\n\t\t\t\t\"server_version\": \"5.6\",\n\t\t\t\t\"storage\": \"zfs\",\n\t\t\t\t\"storage_version\": \"2.1.4-0ubuntu0.1\",\n\t\t\t\t\"storage_supported_drivers\": [\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"cephobject\",\n\t\t\t\t\t\t\"Version\": \"15.2.16\",\n\t\t\t\t\t\t\"Remote\": true\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"dir\",\n\t\t\t\t\t\t\"Version\": \"1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"lvm\",\n\t\t\t\t\t\t\"Version\": \"2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"zfs\",\n\t\t\t\t\t\t\"Version\": \"2.1.4-0ubuntu0.1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"btrfs\",\n\t\t\t\t\t\t\"Version\": \"5.4.1\",\n\t\t\t\t\t\t\"Remote\": false\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"ceph\",\n\t\t\t\t\t\t\"Version\": \"15.2.16\",\n\t\t\t\t\t\t\"Remote\": true\n\t\t\t\t\t},\n\t\t\t\t\t{\n\t\t\t\t\t\t\"Name\": \"cephfs\",\n\t\t\t\t\t\t\"Version\": \"15.2.16\",\n\t\t\t\t\t\t\"Remote\": true\n\t\t\t\t\t}\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t}" http_code=200
time="2022-10-23T17:36:29Z" level=debug msg="Handling API request" ip=@ method=GET protocol=unix url=/internal/ready username=root
time="2022-10-23T17:36:29Z" level=debug msg="WriteJSON\n\t{\n\t\t\"type\": \"sync\",\n\t\t\"status\": \"Success\",\n\t\t\"status_code\": 200,\n\t\t\"operation\": \"\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {}\n\t}" http_code=200
time="2022-10-23T17:36:29Z" level=debug msg="Acquiring lock for image" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69
time="2022-10-23T17:36:29Z" level=debug msg="Lock acquired for image" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69
time="2022-10-23T17:36:29Z" level=debug msg="Image already exists in the DB" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69
time="2022-10-23T17:36:29Z" level=debug msg="Image already exists on storage pool" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69 pool=base-pool
time="2022-10-23T17:36:29Z" level=debug msg="Image already up to date" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69
time="2022-10-23T17:36:29Z" level=debug msg="Connecting to a remote simplestreams server" URL="https://cloud-images.ubuntu.com/releases"
time="2022-10-23T17:36:30Z" level=debug msg="Acquiring lock for image" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69
time="2022-10-23T17:36:30Z" level=debug msg="Lock acquired for image" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69
time="2022-10-23T17:36:30Z" level=debug msg="Image already exists in the DB" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69
time="2022-10-23T17:36:30Z" level=debug msg="Image already exists on storage pool" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69 pool=new2-pool
time="2022-10-23T17:36:30Z" level=debug msg="Image already up to date" fingerprint=4ca869945db4265e397925faf31fc93dac5cbbde0435aa6d3d12b15e2d85fe69
time="2022-10-23T17:36:30Z" level=debug msg="Processing image" alias=22.04 fingerprint=85e44c519b2a1e81f76a388ef48ab7f3308acc5fe10dd61ab7f394179d746bf6 protocol=simplestreams server="https://cloud-images.ubuntu.com/releases"
time="2022-10-23T17:36:30Z" level=debug msg="Connecting to a remote simplestreams server" URL="https://cloud-images.ubuntu.com/releases"
time="2022-10-23T17:36:30Z" level=debug msg="Success for operation" class=task description="Updating instance types" operation=bb549e9f-2dcb-4371-a5a8-96ab4b019953 project=
time="2022-10-23T17:36:30Z" level=info msg="Done updating instance types"
time="2022-10-23T17:36:30Z" level=debug msg="Acquiring lock for image" fingerprint=85e44c519b2a1e81f76a388ef48ab7f3308acc5fe10dd61ab7f394179d746bf6
time="2022-10-23T17:36:30Z" level=debug msg="Lock acquired for image" fingerprint=85e44c519b2a1e81f76a388ef48ab7f3308acc5fe10dd61ab7f394179d746bf6
time="2022-10-23T17:36:30Z" level=debug msg="Image already exists in the DB" fingerprint=85e44c519b2a1e81f76a388ef48ab7f3308acc5fe10dd61ab7f394179d746bf6
time="2022-10-23T17:36:30Z" level=debug msg="Image already exists on storage pool" fingerprint=85e44c519b2a1e81f76a388ef48ab7f3308acc5fe10dd61ab7f394179d746bf6 pool=new1-pool
time="2022-10-23T17:36:30Z" level=debug msg="Image already up to date" fingerprint=85e44c519b2a1e81f76a388ef48ab7f3308acc5fe10dd61ab7f394179d746bf6
time="2022-10-23T17:36:30Z" level=debug msg="Success for operation" class=task description="Updating images" operation=9a8e8974-7abc-421e-ad7f-8fb5d237347f project=
time="2022-10-23T17:36:30Z" level=info msg="Done updating images"

tomp · October 25, 2022, 10:15am

So are you saying that you had an lxc exec session running while you tried to reload LXD, and you’re asking why LXD doesn’t kill it?

michacassola · October 25, 2022, 10:36am

While spawning and deleting container instances I used to use systemctl reload snap.lxd.daemon which didn’t complete I guess while working inside another container. If I then tried to do stuff after deleting or spawning I got Error: LXD is shutting down and I was wondering if there was a better way, and I guess not reloading is the answer.
Now I do nft delete ... additionally to deleting it from the conf file. I guess this is better anyways and I should have done it this way and not invoke nft -f /etc/nftables.conf and systemctl reload snap.lxd.daemon.

Why can’t systemctl reload snap.lxd.daemon do it’s thing while I am still connected to a container with lxc exec new1 bash? And even after I disconnect with exit it takes a long time for things to go back to normal…

tomp · October 25, 2022, 10:41am

I think there is something fishy going on, as I can reproduce something similar by doing:

lxc launch images:ubuntu/jammy c1
lxc shell c1

Then in a different window doing:

lxc stop c1

The original lxc shell command ends, but lxc stop takes a long time between the exec session ending and the lxc stop command completing:

DEBUG  [2022-10-25T11:38:41+01:00] Success for operation                         class=websocket description="Executing command" operation=8d06b5ed-24ad-4673-af77-8adc5a98071d project=default
DEBUG  [2022-10-25T11:38:41+01:00] Event listener server handler stopped         listener=bb30ae4b-bd59-4186-ab68-3dd503523dc7 local=/var/lib/lxd/unix.socket remote=@
DEBUG  [2022-10-25T11:40:09+01:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/cbuild/onstopns?netns=%2Fproc%2F30984%2Ffd%2F4&project=default&target=stop" username=root

Here there is a gap between 11:38:41 and 11:40:09.

Are you seeing the same thing?

michacassola · October 25, 2022, 10:46am

I was stopping other instances or starting them, but I guess I am experiencing the same underlying issue?!

tomp · October 26, 2022, 5:26am

I think my experience is different and specific to a certain container I had as its not happening with a fresh Jammy container.

However I did notice that the core.shutdown_timeout setting was not being respected when doing lxd shutdown.

This fixes it:

Did you try lowering core.shutdown_timeout as the default is 5 minutes if not specified?

Because reloading LXD will have to kill any active lxc exec sessions (but not restart the instances themselves), so by default when systemctl reload snap.lxd.daemon is initiated we wait up to 5 minutes before killing the sessions forcefully.

michacassola · October 26, 2022, 11:54am

No, I was not aware of the thing with exec. And I was not aware of the core.shutdown_timeout. But my execs were killed by me by exiting the bash shell of the exec command. And still it waited until killing them, although they should have been dead already… So maybe there is something else going on also?

tomp · October 26, 2022, 11:56am

I’m not really following you, can you show full reproducer steps I can take (including creating the instance)?

michacassola · October 26, 2022, 12:03pm

Reproducible setps:

Create two instances
lxc exec instance bash into one
Reload the daemon
Exit the bash in the instance
Try to stop and delete the other

The bash process will be running in the background although it is not necessary anymore and make the LXd shutdown error occur.

tomp · October 26, 2022, 1:21pm

So when I start the LXD reload by sending TERM signal to LXD process, the LXD stop is held up due The bash process will be running in the background although it is not necessary anymore and make the LXd shutdown error occur.to the active lxc exec session. Then once the lxc exec session ends the LXD daemon immediately finishes on my system.

It maybe on your system that the LXD process stop takes some time after the exec session has ended, which is why you can then not stop/delete the other instance.

Can you run lxc monitor --pretty then start the lxc exec session and then trigger the reload and capture what the monitor screen outputs, as that should show how far LXD is getting during shutdown.

michacassola · October 27, 2022, 9:02am

Hm, did not work trying to reproduce:

DEBUG  [2022-10-27T08:59:58Z] Event listener server handler started         id=e6a8ab3c-aba8-4d5e-a24e-0359705892ff local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-10-27T09:00:05Z] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2022-10-27T09:00:05Z] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=root
DEBUG  [2022-10-27T09:00:05Z] Handling API request                          ip=@ method=POST protocol=unix url=/1.0/instances/git/exec username=root
DEBUG  [2022-10-27T09:00:05Z] Event listener server handler started         id=bdf7c524-817d-42cc-bfe8-e4e38633c7d5 local=/var/snap/lxd/common/lxd/unix.socket remote=@
DEBUG  [2022-10-27T09:00:05Z] Started operation                             class=websocket description="Executing command" operation=083b925e-1e91-4d3c-b715-cc4947ed5cf6 project=default
DEBUG  [2022-10-27T09:00:05Z] New operation                                 class=websocket description="Executing command" operation=083b925e-1e91-4d3c-b715-cc4947ed5cf6 project=default
INFO   [2022-10-27T09:00:05Z] ID: 083b925e-1e91-4d3c-b715-cc4947ed5cf6, Class: websocket, Description: Executing command  CreatedAt="2022-10-27 09:00:05.35639183 +0000 UTC" Err= Location=none MayCancel=false Metadata="map[command:[bash] environment:map[HOME:/root LANG:C.UTF-8 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin TERM:xterm-256color USER:root] fds:map[0:5ac7e48dc4c183018e23705bf6351b9ef8147b46546b2e169372b9f0b8972070 control:095567950f50ca0a7e4180b4a407d5ab3196f6cf678112cb6e63ef35b89e6916] interactive:true]" Resources="map[containers:[/1.0/containers/git] instances:[/1.0/instances/git]]" Status=Pending StatusCode=Pending UpdatedAt="2022-10-27 09:00:05.35639183 +0000 UTC"
INFO   [2022-10-27T09:00:05Z] ID: 083b925e-1e91-4d3c-b715-cc4947ed5cf6, Class: websocket, Description: Executing command  CreatedAt="2022-10-27 09:00:05.35639183 +0000 UTC" Err= Location=none MayCancel=false Metadata="map[command:[bash] environment:map[HOME:/root LANG:C.UTF-8 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin TERM:xterm-256color USER:root] fds:map[0:5ac7e48dc4c183018e23705bf6351b9ef8147b46546b2e169372b9f0b8972070 control:095567950f50ca0a7e4180b4a407d5ab3196f6cf678112cb6e63ef35b89e6916] interactive:true]" Resources="map[containers:[/1.0/containers/git] instances:[/1.0/instances/git]]" Status=Running StatusCode=Running UpdatedAt="2022-10-27 09:00:05.35639183 +0000 UTC"
DEBUG  [2022-10-27T09:00:05Z] Waiting for exec websockets to connect       
DEBUG  [2022-10-27T09:00:05Z] Connecting to operation                       class=websocket description="Executing command" operation=083b925e-1e91-4d3c-b715-cc4947ed5cf6 project=default
DEBUG  [2022-10-27T09:00:05Z] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/083b925e-1e91-4d3c-b715-cc4947ed5cf6/websocket?secret=095567950f50ca0a7e4180b4a407d5ab3196f6cf678112cb6e63ef35b89e6916" username=root
DEBUG  [2022-10-27T09:00:05Z] Connected to operation                        class=websocket description="Executing command" operation=083b925e-1e91-4d3c-b715-cc4947ed5cf6 project=default
DEBUG  [2022-10-27T09:00:05Z] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/operations/083b925e-1e91-4d3c-b715-cc4947ed5cf6/websocket?secret=5ac7e48dc4c183018e23705bf6351b9ef8147b46546b2e169372b9f0b8972070" username=root
DEBUG  [2022-10-27T09:00:05Z] Connecting to operation                       class=websocket description="Executing command" operation=083b925e-1e91-4d3c-b715-cc4947ed5cf6 project=default
DEBUG  [2022-10-27T09:00:05Z] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/083b925e-1e91-4d3c-b715-cc4947ed5cf6 username=root
DEBUG  [2022-10-27T09:00:05Z] Connected to operation                        class=websocket description="Executing command" operation=083b925e-1e91-4d3c-b715-cc4947ed5cf6 project=default
DEBUG  [2022-10-27T09:00:05Z] Retrieved PID of executing child process      attachedPid=49227 instance=git instanceType=container project=default
DEBUG  [2022-10-27T09:00:05Z] Exec mirror websocket started                 PID=49227 instance=git interactive=true number=0 project="{{map[features.images:true features.networks:true features.profiles:true features.storage.buckets:true features.storage.volumes:true] Default LXD project} default []}"
DEBUG  [2022-10-27T09:00:05Z] Exec control handler started                  PID=49227 instance=git interactive=true project="{{map[features.images:true features.networks:true features.profiles:true features.storage.buckets:true features.storage.volumes:true] Default LXD project} default []}"
INFO   [2022-10-27T09:00:05Z] Action: instance-exec, Source: /1.0/instances/git  command="[bash]"
DEBUG  [2022-10-27T09:00:05Z] Instance process started                      PID=49227 instance=git interactive=true project="{{map[features.images:true features.networks:true features.profiles:true features.storage.buckets:true features.storage.volumes:true] Default LXD project} default []}"
INFO   [2022-10-27T09:00:37Z] Received signal                               signal=terminated
INFO   [2022-10-27T09:00:37Z] Starting shutdown sequence                    signal=terminated
INFO   [2022-10-27T09:00:37Z] Waiting for operations to finish             
DEBUG  [2022-10-27T09:00:45Z] Sending write barrier                        
DEBUG  [2022-10-27T09:00:45Z] Instance process stopped                      PID=49227 err="<nil>" exitStatus=0 instance=git interactive=true project="{{map[features.images:true features.networks:true features.profiles:true features.storage.buckets:true features.storage.volumes:true] Default LXD project} default []}"
DEBUG  [2022-10-27T09:00:45Z] Detected poll(POLLHUP) event: exiting        
DEBUG  [2022-10-27T09:00:45Z] Detected poll(POLLNVAL) event                
DEBUG  [2022-10-27T09:00:45Z] Exec control handler finished                 PID=49227 instance=git interactive=true project="{{map[features.images:true features.networks:true features.profiles:true features.storage.buckets:true features.storage.volumes:true] Default LXD project} default []}"
DEBUG  [2022-10-27T09:00:45Z] DefaultWriter got error getting next reader   err="websocket: close 1006 (abnormal closure): unexpected EOF"
DEBUG  [2022-10-27T09:00:45Z] Exec mirror websocket finished                PID=49227 instance=git interactive=true number=0 project="{{map[features.images:true features.networks:true features.profiles:true features.storage.buckets:true features.storage.volumes:true] Default LXD project} default []}"
DEBUG  [2022-10-27T09:00:45Z] Updated metadata for operation                class=websocket description="Executing command" operation=083b925e-1e91-4d3c-b715-cc4947ed5cf6 project=default
DEBUG  [2022-10-27T09:00:45Z] Success for operation                         class=websocket description="Executing command" operation=083b925e-1e91-4d3c-b715-cc4947ed5cf6 project=default
INFO   [2022-10-27T09:00:45Z] ID: 083b925e-1e91-4d3c-b715-cc4947ed5cf6, Class: websocket, Description: Executing command  CreatedAt="2022-10-27 09:00:05.35639183 +0000 UTC" Err= Location=none MayCancel=false Metadata="map[command:[bash] environment:map[HOME:/root LANG:C.UTF-8 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin TERM:xterm-256color USER:root] fds:map[0:5ac7e48dc4c183018e23705bf6351b9ef8147b46546b2e169372b9f0b8972070 control:095567950f50ca0a7e4180b4a407d5ab3196f6cf678112cb6e63ef35b89e6916] interactive:true return:0]" Resources="map[containers:[/1.0/containers/git] instances:[/1.0/instances/git]]" Status=Running StatusCode=Running UpdatedAt="2022-10-27 09:00:45.152495323 +0000 UTC"
INFO   [2022-10-27T09:00:45Z] ID: 083b925e-1e91-4d3c-b715-cc4947ed5cf6, Class: websocket, Description: Executing command  CreatedAt="2022-10-27 09:00:05.35639183 +0000 UTC" Err= Location=none MayCancel=false Metadata="map[command:[bash] environment:map[HOME:/root LANG:C.UTF-8 PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin TERM:xterm-256color USER:root] fds:map[0:5ac7e48dc4c183018e23705bf6351b9ef8147b46546b2e169372b9f0b8972070 control:095567950f50ca0a7e4180b4a407d5ab3196f6cf678112cb6e63ef35b89e6916] interactive:true return:0]" Resources="map[containers:[/1.0/containers/git] instances:[/1.0/instances/git]]" Status=Success StatusCode=Success UpdatedAt="2022-10-27 09:00:45.152495323 +0000 UTC"
DEBUG  [2022-10-27T09:00:45Z] Event listener server handler stopped         listener=bdf7c524-817d-42cc-bfe8-e4e38633c7d5 local=/var/snap/lxd/common/lxd/unix.socket remote=@
INFO   [2022-10-27T09:00:45Z] All running operations finished, shutting down 
INFO   [2022-10-27T09:00:45Z] Stopping daemon storage volumes              
DEBUG  [2022-10-27T09:00:45Z] Cancel ongoing or future gRPC connection attempts 
INFO   [2022-10-27T09:00:45Z] Closing the database                         
INFO   [2022-10-27T09:00:46Z] Stop database gateway                        
INFO   [2022-10-27T09:00:46Z] Closing socket                                socket="[::]:8444" type="REST API TCP socket"
INFO   [2022-10-27T09:00:46Z] Closing socket                                socket=/var/snap/lxd/common/lxd/devlxd/sock type="devlxd socket"
INFO   [2022-10-27T09:00:46Z] Closing socket                                socket=/var/snap/lxd/common/lxd/unix.socket type="REST API Unix socket"
INFO   [2022-10-27T09:00:46Z] Not unmounting temporary filesystems (instances are still running) 
Error: websocket: close 1006 (abnormal closure): unexpected EOF

tomp · October 28, 2022, 8:33am

Yes the reload doesn’t seem to be being held up by operations.