Loading config file for the container failed (2022)

I am getting the above error when using Manjaro and Ubuntu jammy snap 2.57 version of LXD 5.4. I finally managed to get the nodes to join a cluster and adding a VM works but a CT fails to start.

+------+---------+-------------------------+------+-----------------+-----------+----------+
| NAME |  STATE  |          IPV4           | IPV6 |      TYPE       | SNAPSHOTS | LOCATION |
+------+---------+-------------------------+------+-----------------+-----------+----------+
| ctu1 | STOPPED |                         |      | CONTAINER       | 0         | lxd1     |
+------+---------+-------------------------+------+-----------------+-----------+----------+
| vmu1 | RUNNING | 192.168.20.154 (enp5s0) |      | VIRTUAL-MACHINE | 0         | mbox     |
+------+---------+-------------------------+------+-----------------+-----------+----------+
lxd1 ~ lxc start ctu1
Error: Failed to load config file "/tmp/lxd_config_894494813": loading config file for the container failed
Try `lxc info --show-log ctu1` for more info
Name: ctu1
Status: STOPPED
Type: container
Architecture: x86_64
Location: lxd1
Created: 2022/08/09 16:05 AEST

Log:
lxd1 ~ ll /tmp/snap.lxd/tmp/lxd_config_894494813
-rw------- 1 root root 25 Aug 11 12:25 /tmp/snap.lxd/tmp/lxd_config_894494813
lxd1 ~ cat /tmp/snap.lxd/tmp/lxd_config_894494813   
lxc.aa_profile=unconfined
lxd1 ~ lxc config show ctu1 --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20220807_07:42)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20220807_07:42"
  image.type: squashfs
  image.variant: default
  raw.lxc: lxc.aa_profile=unconfined
  volatile.apply_template: create
  volatile.base_image: 63c1913c93b50fbf0b3790186598dba34bc0b9f4edfa1ff145a438cc65837342
  volatile.cloud-init.instance-id: a95bbab0-3d7f-44eb-8a74-c1b537500deb
  volatile.eth0.hwaddr: 00:16:3e:3e:a2:07
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.uuid: a289743a-e746-4bf7-852f-184ded838c98
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: local
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
lxd1 ~ lxc info
config:
  cluster.https_address: 192.168.20.169:8443
  core.https_address: 192.168.20.169:8443
  core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 192.168.20.169:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIB/zCCAYSgAwIBAgIRAM2AGJTJS78LtRPKlsgT31cwCgYIKoZIzj0EAwMwMjEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEBtYm94
    MB4XDTIyMDgwOTAyNDI0NloXDTMyMDgwNjAyNDI0NlowMjEcMBoGA1UEChMTbGlu
    dXhjb250YWluZXJzLm9yZzESMBAGA1UEAwwJcm9vdEBtYm94MHYwEAYHKoZIzj0C
    AQYFK4EEACIDYgAE7bve/+iI5XBl6aSAYOFsY0Okp5IDYyD+TCo7t4heOWIRL6gS
    UrQOBcFTsYaKt8EAdvWfHVxdlsyWfhj6S6jCjiFqY4p4ZbcP3shH0oPQ5zeIsXXe
    yryN2iyqhsEFd5uFo14wXDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYB
    BQUHAwEwDAYDVR0TAQH/BAIwADAnBgNVHREEIDAeggRtYm94hwR/AAABhxAAAAAA
    AAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQDasYSP7p+t9WgyYvhWNmJg
    /GfcKx8N1MvepAXO312R1GDE+nVd6x9R/vwdhRyCm4ACMQDlmr3Z/YH50FP0CSoD
    4ffDps3+l6Iy0aqNUOZgZD7IpxoKhVJltTUlwr5UBDm4ru4=
    -----END CERTIFICATE-----
  certificate_fingerprint: 9a3c7a600c53f29dae250231454db75f07d49cd3e8e848c2f60ea6a2bb0e90db
  driver: lxc
  driver_version: 5.0.0
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.15.0-43-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "22.04"
  project: default
  server: lxd
  server_clustered: true
  server_event_mode: full-mesh
  server_name: lxd1
  server_pid: 10558
  server_version: "5.4"
  storage: dir
  storage_version: "1"
  storage_supported_drivers:
  - name: zfs
    version: 2.1.4-0ubuntu0.1
    remote: false
  - name: ceph
    version: 15.2.16
    remote: true
  - name: btrfs
    version: 5.4.1
    remote: false
  - name: cephfs
    version: 15.2.16
    remote: true
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0
    remote: false

I thought I would try again with the official ubuntu:22.04 image but I just get the same error so I can’t actually start the container. There are very few google results for this error so if anyone has a clue as to what I can look at to figure this out then it would be greatly appreciated.

lbox ~ lxc launch ubuntu:22.04 ctu2 -c security.nesting=true
Creating ctu2
Starting ctu2                                 
Error: Failed to load config file "/tmp/lxd_config_2227921804": loading config file for the container failed
Try `lxc info --show-log local:ctu2` for more info

lbox ~ lxc info --show-log local:ctu2
Name: ctu2
Status: STOPPED
Type: container
Architecture: x86_64
Location: lbox
Created: 2022/08/12 15:49 AEST

Log:

lbox ~ ll /tmp/snap.lxd/tmp/
total 4
-rw------- 1 root root 25 Aug 12 15:49 lxd_config_2227921804

lbox ~ ll /tmp/snap.lxd/tmp/lxd_config_2227921804
-rw------- 1 root root 25 Aug 12 15:49 /tmp/snap.lxd/tmp/lxd_config_2227921804

lbox ~ cat /tmp/snap.lxd/tmp/lxd_config_2227921804
lxc.aa_profile=unconfined

Hmm, I notice as a normal user in the lxd group that I cannot access that config file.

lbox ~ groups
sys network power lxd lp kvm wheel markc

lbox ~ ll /tmp/snap.lxd/tmp/lxd_config_2227921804
ls: cannot access '/tmp/snap.lxd/tmp/lxd_config_2227921804': Permission denied

So as a longshot I modified the owner perms of /tmp/snap.lxd to :lxd 755 so that I can list that config file as my normal user but “lxc start ctu2” still won’t work.

lbox ~ ll /tmp/snap.lxd -d
drwxrwxr-x 3 root lxd 60 Aug  9 16:02 /tmp/snap.lxd/

lbox ~ ll /tmp/snap.lxd/tmp/lxd_config_2227921804
-rw------- 1 root root 25 Aug 12 15:49 /tmp/snap.lxd/tmp/lxd_config_2227921804

lbox ~ lxc start ctu2
Error: Failed to load config file "/tmp/lxd_config_69303279": loading config file for the container failed
Try `lxc info --show-log ctu2` for more info

This sounds similar to https://github.com/lxc/lxd/issues/10771#issuecomment-1212183389

Is something clearing /tmp?

Clearing /tmp? Not according to this output, the config file failing to load is still there in /tmp/snap.lxd/tmp/. If there is meant to be a temporary symlink from /tmp/lxd_config_2227921804 to /tmp/snap.lxd/tmp/lxd_config_2227921804 then that could be deleted. The other concern is that lxc info is not showing any logging output, so maybe there is a problem with the snap install on Manjaro, but I have completely removed both lxd and snap and done a complete reinstallation of both after rebooting each time. It’s also strange that a VM installs and starts up okay, but a CT will not start.

lbox ~ ll /tmp/snap.lxd/tmp/lxd_config_2227921804
-rw------- 1 root root 25 Aug 12 15:49 /tmp/snap.lxd/tmp/lxd_config_2227921804

Well, I have to give up. This is a 3rd time in 5 years I’ve gone all in on LXD only to bump into some silly problem that wastes so much of my time I just can’t continue. It took me 3 hours to set up a 3 node Proxmox cluster which is… still working after a week!

Yes its a strange one, not something I’ve come across before, although I am not familiar with Manjaro.

One thing that I would consider trying is LXD Manjaro package, as that way it would at least give an additional data point. I.e. if it works then it could point to an issue with the snap packaging or snapd, and if it still doesn’t work then it would rule out an issue with snap altogether.