Fail to start VM / saving config failed (disk full)

Hello,

After a a failed snapshot & restore, I am having trouble getting the main VM backon jammy that is using 4.0.9 as LTS thru standard Canonical’s snap.

admin@host:~$ lxc start gooo --debug
DBUG[11-28|09:27:19] Connecting to a local LXD over a Unix socket
DBUG[11-28|09:27:19] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
DBUG[11-28|09:27:19] Got response struct from LXD
DBUG[11-28|09:27:19]
        {
                "config": {
                        "core.https_address": "[::]:8443",
                        "core.trust_password": true
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "snapshot_schedule_aliases",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "resources_system",
                        "usedby_consistency",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "storage_rsync_compression",
                        "gpu_mdev",
                        "resources_pci_iommu",
                        "resources_network_usb",
                        "resources_disk_address",
                        "network_state_vlan",
                        "gpu_sriov",
                        "migration_stateful",
                        "disk_state_quota",
                        "storage_ceph_features",
                        "gpu_mig",
                        "clustering_join_token",
                        "clustering_description",
                        "server_trusted_proxy",
                        "clustering_update_cert",
                        "storage_api_project",
                        "server_instance_driver_operational",
                        "server_supported_storage_drivers",
                        "event_lifecycle_requestor_address",
                        "resources_gpu_usb",
                        "network_counters_errors_dropped",
                        "image_source_project",
                        "database_leader",
                        "instance_all_projects",
                        "ceph_rbd_du",
                        "qemu_metrics",
                        "gpu_mig_uuid",
                        "event_project",
                        "instance_allow_inconsistent_copy",
                        "image_restrictions"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "192.168.1.58:8443",
                                "[2a01:e0a:208:c400:8c29:24ff:fe76:854d]:8443",
                                "10.226.78.1:8443",
                                "[fd42:998:926f:7af4::1]:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----<removed>\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "<removed>",
                        "driver": "lxc | qemu",
                        "driver_version": "4.0.12 | 6.1.1",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "5.15.0-53-generic",
                        "lxc_features": {
                                "cgroup2": "true",
                                "core_scheduling": "true",
                                "devpts_fd": "true",
                                "idmapped_mounts_v2": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "Ubuntu",
                        "os_version": "22.04",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_name": "jarjar1",
                        "server_pid": 2274,
                        "server_version": "4.0.9",
                        "storage": "btrfs",
                        "storage_version": "5.4.1",
                        "storage_supported_drivers": [
                                {
                                        "Name": "cephfs",
                                        "Version": "15.2.14",
                                        "Remote": true
                                },
                                {
                                        "Name": "dir",
                                        "Version": "1",
                                        "Remote": false
                                },
                                {
                                        "Name": "lvm",
                                        "Version": "2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0",
                                        "Remote": false
                                },
                                {
                                        "Name": "zfs",
                                        "Version": "2.1.4-0ubuntu0.1",
                                        "Remote": false
                                },
                                {
                                        "Name": "ceph",
                                        "Version": "15.2.14",
                                        "Remote": true
                                },
                                {
                                        "Name": "btrfs",
                                        "Version": "5.4.1",
                                        "Remote": false
                                }
                        ]
                }
        }
DBUG[11-28|09:27:19] Sending request to LXD                   method=GET url=http://unix.socket/1.0/instances/gooo etag=
DBUG[11-28|09:27:19] Got response struct from LXD
DBUG[11-28|09:27:19]
        {
                "architecture": "x86_64",
                "config": {
                        "image.architecture": "amd64",
                        "image.description": "ubuntu 18.04 LTS amd64 (release) (20180808)",
                        "image.label": "release",
                        "image.os": "ubuntu",
                        "image.release": "bionic",
                        "image.serial": "20180808",
                        "image.version": "18.04",
                        "security.nesting": "true",
                        "volatile.base_image": "7e8633da9dfc800230c7330cf04e9f284e82e26ddbc1757448c29c25db80f1e4",
                        "volatile.eth0.hwaddr": "08:00:27:bc:91:63",
                        "volatile.idmap.base": "0",
                        "volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
                        "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
                        "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
                        "volatile.last_state.power": "STOPPED",
                        "volatile.uuid": "8162435d-aa67-4630-bb7f-63528e4b697c"
                },
                "devices": {},
                "ephemeral": false,
                "profiles": [
                        "default",
                        "mse-lan-profile"
                ],
                "stateful": false,
                "description": "",
                "created_at": "2018-08-13T19:59:19+02:00",
                "expanded_config": {
                        "image.architecture": "amd64",
                        "image.description": "ubuntu 18.04 LTS amd64 (release) (20180808)",
                        "image.label": "release",
                        "image.os": "ubuntu",
                        "image.release": "bionic",
                        "image.serial": "20180808",
                        "image.version": "18.04",
                        "security.nesting": "true",
                        "volatile.base_image": "7e8633da9dfc800230c7330cf04e9f284e82e26ddbc1757448c29c25db80f1e4",
                        "volatile.eth0.hwaddr": "08:00:27:bc:91:63",
                        "volatile.idmap.base": "0",
                        "volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
                        "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
                        "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
                        "volatile.last_state.power": "STOPPED",
                        "volatile.uuid": "8162435d-aa67-4630-bb7f-63528e4b697c"
                },
                "expanded_devices": {
                        "eth0": {
                                "name": "eth0",
                                "nictype": "bridged",
                                "parent": "br0",
                                "type": "nic"
                        },
                        "root": {
                                "path": "/",
                                "pool": "Medggg",
                                "type": "disk"
                        }
                },
                "name": "gooo",
                "status": "Stopped",
                "status_code": 102,
                "last_used_at": "2022-11-19T00:35:41.808118014Z",
                "location": "none",
                "type": "container",
                "project": "default"
        }
DBUG[11-28|09:27:19] Connected to the websocket: ws://unix.socket/1.0/events
DBUG[11-28|09:27:19] Sending request to LXD                   method=PUT url=http://unix.socket/1.0/instances/gooo/state etag=
DBUG[11-28|09:27:19]
        {
                "action": "start",
                "timeout": 0,
                "force": false,
                "stateful": false
        }
DBUG[11-28|09:27:19] Got operation from LXD
DBUG[11-28|09:27:19]
        {
                "id": "7adf57f6-53ca-4a73-b682-8fae7b8a7421",
                "class": "task",
                "description": "Starting instance",
                "created_at": "2022-11-28T09:27:19.590660465+01:00",
                "updated_at": "2022-11-28T09:27:19.590660465+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "instances": [
                                "/1.0/instances/gooo"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
DBUG[11-28|09:27:19] Sending request to LXD                   method=GET url=http://unix.socket/1.0/operations/7adf57f6-53ca-4a73-b682-8fae7b8a7421 etag=
DBUG[11-28|09:27:19] Got response struct from LXD
DBUG[11-28|09:27:19]
        {
                "id": "7adf57f6-53ca-4a73-b682-8fae7b8a7421",
                "class": "task",
                "description": "Starting instance",
                "created_at": "2022-11-28T09:27:19.590660465+01:00",
                "updated_at": "2022-11-28T09:27:19.590660465+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "instances": [
                                "/1.0/instances/gooo"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
Error: saving config file for the container failed
Try `lxc info --show-log gooo` for more info

Nothing more on the issue. Previous cases I found did not help too. Any idea what is blocking lxc there ?

Update 1 : Tried snap refreshing from 4.0.9-8e2046b to latest 4.0.9-eb5e237 did not helped.

Please show lxc config show <instance> --expanded as well as lxc storage show <pool>.
Please can you also describe more about this failed snapshot restored? What happened, what was the error?

Here it is

admin@host:~$ lxc config show gooo
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20180808)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20180808"
  image.version: "18.04"
  security.nesting: "true"
  volatile.base_image: 7e8633da9dfc800230c7330cf04e9f284e82e26ddbc1757448c29c25db80f1e4
  volatile.eth0.hwaddr: 08:00:27:bc:91:63
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 8162435d-aa67-4630-bb7f-63528e4b697c
devices: {}
ephemeral: false
profiles:
- default
- mse-lan-profile
stateful: false
description: ""

This was a indeed bionic when installed, but it has been upgraded to jammy which the command is not reflecting.

admin@host:~$  lxc storage show Medggg
config:
  size: 15GB
  source: /var/snap/lxd/common/lxd/disks/Medggg.img
description: ""
name: Medggg
driver: btrfs
used_by:
- /1.0/instances/gooo
- /1.0/instances/gooo/snap-2021-02-25
- /1.0/profiles/default
status: Created
locations:
- none
admin@host:~$ lxc info gooo --verbose
Name: gooo
Location: none
Remote: unix://
Architecture: x86_64
Created: 2018/08/13 19:59 CEST
Status: Stopped
Type: container
Profiles: default, mse-lan-profile
Snapshots:
  snap-2021-02-25 (taken at 2021/02/25 00:29 CET) (stateless)
  snap-2022-11-18 (taken at 2022/11/18 23:41 CET) (stateless)
  snap-2022-11-25 (taken at 2022/11/25 23:05 CET) (stateless)
  snap-2022-11-26 (taken at 2022/11/26 01:44 CET) (stateless)

I had some issue after a software upgrade inside the gooo VM so I’ve decided to restore to a previous snapshot state and see later as it had to be live back. But before I’ve tried to have a snap-2022-11-27 to see later if I can get a fix on the situation but it was dog slow so I attempted to stop it. I did not saw that it was still in progress (I though it was stuck as there was no progress bar). I tried to restore to snap-2022-11-26 and it went thru after several hours indicating that it was not beeing enable to rename some file and I unfortunatly was not able to copy their name straight away (GNU screen scrolled). I am not finding those errors in the logs anymore.

Please show lxc config show <instance> --expanded output.

Also, have you tried rebooting to clear any errant mounts?

Here it is

admin@host:~$ lxc config show gooo --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20180808)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20180808"
  image.version: "18.04"
  security.nesting: "true"
  volatile.base_image: 7e8633da9dfc800230c7330cf04e9f284e82e26ddbc1757448c29c25db80f1e4
  volatile.eth0.hwaddr: 08:00:27:bc:91:63
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 8162435d-aa67-4630-bb7f-63528e4b697c
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: Medggg
    type: disk
ephemeral: false
profiles:
- default
- mse-lan-profile
stateful: false
description: ""

I did not tried to reboot as it is a live server. I will schedule this ASAP.

Thanks, let me know.

Also can you show lxc info <instance> too.

I was able to get a reboot slot from the users. Still the same issues.

admin**@host:~$ lxc info gooo**
**Name: gooo**
**Location: none**
**Remote: unix://**
**Architecture: x86_64**
**Created: 2018/08/13 19:59 CEST**
**Status: Stopped**
**Type: container**
**Profiles: default, mse-lan-profile**
**Snapshots:**
**  snap-2021-02-25 (taken at 2021/02/25 00:29 CET) (stateless)**
**  snap-2022-11-18 (taken at 2022/11/18 23:41 CET) (stateless)**
**  snap-2022-11-25 (taken at 2022/11/25 23:05 CET) (stateless)**
**  snap-2022-11-26 (taken at 2022/11/26 01:44 CET) (stateless)**

Still wondering why I don’t see any message/error other than this saving config problem.

Have you looked in /var/snap/lxd/common/lxd/logs/lxd.log

Also anything in sudo dmesg?

Lets take a look at lxc storage info <pool> too to get an idea of sizings.

Nothing special, after reboot

t=2022-11-28T11:01:03+0100 lvl=info msg="LXD is starting" mode=normal path=/var/snap/lxd/common/lxd version=4.0.9
t=2022-11-28T11:01:03+0100 lvl=info msg="Kernel uid/gid map:"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - u 0 0 4294967295"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - g 0 0 4294967295"
t=2022-11-28T11:01:03+0100 lvl=info msg="Configured LXD uid/gid map:"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - u 0 1000000 1000000000"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - g 0 1000000 1000000000"
t=2022-11-28T11:01:03+0100 lvl=info msg="Kernel features:"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - closing multiple file descriptors efficiently: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - netnsid-based network retrieval: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - pidfds: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - core scheduling: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - uevent injection: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - seccomp listener: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - seccomp listener continue syscalls: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - seccomp listener add file descriptors: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - attach to namespaces via pidfds: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - safe native terminal allocation : yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - unprivileged file capabilities: yes"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - cgroup layout: cgroup2"
t=2022-11-28T11:01:03+0100 lvl=warn msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
t=2022-11-28T11:01:03+0100 lvl=info msg=" - shiftfs support: disabled"
t=2022-11-28T11:01:04+0100 lvl=info msg="Initializing local database"
t=2022-11-28T11:01:04+0100 lvl=info msg="Set client certificate to server certificate" fingerprint=6b08b9b7b909c92e92bbf1c8b5efcc649d7ddade33c74dfb06e87542ceb94e2e
t=2022-11-28T11:01:04+0100 lvl=info msg="Starting database node" id=1 local=1 role=voter
t=2022-11-28T11:01:04+0100 lvl=info msg="Starting /dev/lxd handler:"
t=2022-11-28T11:01:04+0100 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2022-11-28T11:01:04+0100 lvl=info msg="REST API daemon:"
t=2022-11-28T11:01:04+0100 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2022-11-28T11:01:04+0100 lvl=info msg=" - binding TCP socket" socket=[::]:8443
t=2022-11-28T11:01:04+0100 lvl=info msg="Initializing global database"
t=2022-11-28T11:01:04+0100 lvl=info msg="Connecting to global database"
t=2022-11-28T11:01:04+0100 lvl=info msg="Connected to global database"
t=2022-11-28T11:01:04+0100 lvl=info msg="Initialized global database"
t=2022-11-28T11:01:05+0100 lvl=info msg="Firewall loaded driver" driver=nftables
t=2022-11-28T11:01:05+0100 lvl=info msg="Initializing storage pools"
t=2022-11-28T11:01:05+0100 lvl=info msg="Initializing daemon storage mounts"
t=2022-11-28T11:01:05+0100 lvl=info msg="Loading daemon configuration"
t=2022-11-28T11:01:05+0100 lvl=info msg="Initializing networks"
t=2022-11-28T11:01:06+0100 lvl=info msg="Pruning leftover image files"
t=2022-11-28T11:01:06+0100 lvl=info msg="Done pruning leftover image files"
t=2022-11-28T11:01:06+0100 lvl=info msg="Starting device monitor"
t=2022-11-28T11:01:06+0100 lvl=info msg="Started seccomp handler" path=/var/snap/lxd/common/lxd/seccomp.socket
t=2022-11-28T11:01:06+0100 lvl=info msg="Pruning expired images"
t=2022-11-28T11:01:06+0100 lvl=info msg="Done pruning expired images"
t=2022-11-28T11:01:06+0100 lvl=info msg="Pruning expired instance backups"
t=2022-11-28T11:01:06+0100 lvl=info msg="Done pruning expired instance backups"
t=2022-11-28T11:01:06+0100 lvl=info msg="Updating images"
t=2022-11-28T11:01:06+0100 lvl=info msg="Expiring log files"
t=2022-11-28T11:01:06+0100 lvl=info msg="Done updating images"
t=2022-11-28T11:01:06+0100 lvl=info msg="Done expiring log files"
t=2022-11-28T11:01:06+0100 lvl=info msg="Updating instance types"
t=2022-11-28T11:01:06+0100 lvl=info msg="Done updating instance types"
t=2022-11-28T11:01:06+0100 lvl=info msg="Daemon started"

And before reboot is was not much more …

And dmesg has not much too. Nothing but a simple

[ 41.708215] audit: type=1400 audit(1669629665.585:116): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd_dnsmasq-lxdbr0_</var/snap/lxd/common/lxd>" pid=2839 comm="apparmor_parser"

Odd…

Is that including trying to start the container?

Does lxc info --show-log gooo show anything useful after a start failure?

Looking at Containers cannot start: saving config file for the container failed · Issue #6118 · lxc/lxd · GitHub and Error: Common start logic: saving config file for the container failed · Issue #7406 · lxc/lxd · GitHub it suggests you may be out of pool storage space.

Nope, nothing in the show-log, nothing in the log, just the error at the start without any other indication.

admin@host:~$ lxc start gooo
Error: saving config file for the container failed
Try `lxc info --show-log gooo` for more info
admin@host:~$

and

admin@host:~$ lxc info --show-log gooo
Name: gooo
Location: none
Remote: unix://
Architecture: x86_64
Created: 2018/08/13 19:59 CEST
Status: Stopped
Type: container
Profiles: default, mse-lan-profile
Snapshots:
  snap-2021-02-25 (taken at 2021/02/25 00:29 CET) (stateless)
  snap-2022-11-18 (taken at 2022/11/18 23:41 CET) (stateless)
  snap-2022-11-25 (taken at 2022/11/25 23:05 CET) (stateless)
  snap-2022-11-26 (taken at 2022/11/26 01:44 CET) (stateless)

Log:


admin@host:~$

I had a space issue after the last snapshot and upgrade but I increased the LVM volume quota and perform the required resize2fs on the LVM volume. I fear that the storage group might not have seen the size upgrade. But I have not found so far how to fix this as well.

admin@host:~$ lxc storage info Meddd
info:
  description: ""
  driver: btrfs
  name: Meddd
  space used: 39.74GiB
  total space: 41.00GiB
used by:
  instances:
  - gooo
  - gooo
  profiles:
  - default

I added 50GB to the LVM Volume so I was expeting the total space to be close to 100GB.

I suspect you’ll need to resize the btrfs ontop of the LVM, I don’t think resize2fs will do that.

Problem is that it is a btrfs file img. There seem to be no link between the BTRFS file sizes on the LVM extfs partition (real or theorical) and the btrfs device size from that same file once mounted. I have tried a fallocate for the img file but it does not sync with the btrfs device size. So quite logically any attempt such as “btrfs filesystem resize max” fail.

Surprisingly most of documentation on btrfs resizing only applies to a regular FS not a file img FS.

Still digging …

Specifically the last command:

sudo btrfs filesystem resize max <LXD_lib_dir>/storage-pools/<pool_name>/

Once you’ve grown the LVM and the image file.

LVM grown and image file grown thru but resize not working

admin@host:~# sudo btrfs filesystem resize max /var/snap/lxd/common/lxd/storage-pools/Medd/
ERROR: not a btrfs filesystem: /var/snap/lxd/common/lxd/storage-pools/Medd/

Which is quite logical as this is not any mounted point.

I tried also adding the
sudo losetup -c <loop_device>
with the same result.