Cannot lxc copy --refresh with option -d (device)?

Hi
Despite doc reading and searchs, I cannot find the syntax to refresh a remote container copy when the initial copy was performed while specifying a device value change. Can someone help?

Initial copy was done with:
lxc copy gitrepo backupserver: --stateless -d sharedrepos,source=/srv/repos
(on source containter, the “sharedrepos” device points to /srv/dev-disk-by-label-ssdct/repos, and on the backupserver’s container, it points to /srv/repos thanks to the -d option)

I tried to refresh the remote container with both lines below but all return the same error:
lxc copy gitrepo backupserver: --refresh --stateless
lxc copy gitrepo backupserver: --refresh --stateless -d sharedrepos,source=/srv/repos
The error is:
Error transferring instance data: Invalid devices: Device validation failed for "sharedrepos": Missing source "/srv/dev-disk-by-label-ssdct/repos" for disk "sharedrepos"

thank you!

Hmm, can you show the --debug output?

Here it is below
Command used:
lxc copy gitrepo backupserver: -d sharedrepos,source=/srv/repos --stateless --refresh --debug

I didn’t mention it , but ‘copy --refresh’ works fine for other containers for which there is no device to remap.

Log
DBUG[11-10|15:18:09] Connecting to a local LXD over a Unix socket
DBUG[11-10|15:18:09] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
DBUG[11-10|15:18:09] Got response struct from LXD
DBUG[11-10|15:18:09]
        {
                "config": {
                        "core.https_address": "0.0.0.0:8443",
                        "core.trust_password": true,
                        "storage.backups_volume": "default/backups",
                        "storage.images_volume": "default/images"
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "10.4.0.2:8443",
                                "192.168.22.2:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\n[....]\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "5bd674088d67774566d7d39e79eca069c5fbe5f93b23614859d7d5cf84e48e5b",
                        "driver": "lxc",
                        "driver_version": "4.0.5",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "5.8.0-0.bpo.2-amd64",
                        "lxc_features": {
                                "cgroup2": "true",
                                "devpts_fd": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "Debian GNU/Linux",
                        "os_version": "10",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_name": "liveserver",
                        "server_pid": 13341,
                        "server_version": "4.7",
                        "storage": "btrfs",
                        "storage_version": "4.15.1"
                }
        }
DBUG[11-10|15:18:09] Connecting to a remote LXD over HTTPs
DBUG[11-10|15:18:09] Sending request to LXD                   method=GET url=https://192.168.22.3:8443/1.0 etag=
DBUG[11-10|15:18:09] Got response struct from LXD
DBUG[11-10|15:18:09]
        {
                "config": {
                        "core.https_address": "192.168.22.3:8443",
                        "core.trust_password": true
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key",
                        "network_vxlan_ttl",
                        "container_incremental_copy",
                        "usb_optional_vendorid",
                        "snapshot_scheduling",
                        "container_copy_project",
                        "clustering_server_address",
                        "clustering_image_replication",
                        "container_protection_shift",
                        "snapshot_expiry",
                        "container_backup_override_pool",
                        "snapshot_expiry_creation",
                        "network_leases_location",
                        "resources_cpu_socket",
                        "resources_gpu",
                        "resources_numa",
                        "kernel_features",
                        "id_map_current",
                        "event_location",
                        "storage_api_remote_volume_snapshots",
                        "network_nat_address",
                        "container_nic_routes",
                        "rbac",
                        "cluster_internal_copy",
                        "seccomp_notify",
                        "lxc_features",
                        "container_nic_ipvlan",
                        "network_vlan_sriov",
                        "storage_cephfs",
                        "container_nic_ipfilter",
                        "resources_v2",
                        "container_exec_user_group_cwd",
                        "container_syscall_intercept",
                        "container_disk_shift",
                        "storage_shifted",
                        "resources_infiniband",
                        "daemon_storage",
                        "instances",
                        "image_types",
                        "resources_disk_sata",
                        "clustering_roles",
                        "images_expiry",
                        "resources_network_firmware",
                        "backup_compression_algorithm",
                        "ceph_data_pool_name",
                        "container_syscall_intercept_mount",
                        "compression_squashfs",
                        "container_raw_mount",
                        "container_nic_routed",
                        "container_syscall_intercept_mount_fuse",
                        "container_disk_ceph",
                        "virtual-machines",
                        "image_profiles",
                        "clustering_architecture",
                        "resources_disk_id",
                        "storage_lvm_stripes",
                        "vm_boot_priority",
                        "unix_hotplug_devices",
                        "api_filtering",
                        "instance_nic_network",
                        "clustering_sizing",
                        "firewall_driver",
                        "projects_limits",
                        "container_syscall_intercept_hugetlbfs",
                        "limits_hugepages",
                        "container_nic_routed_gateway",
                        "projects_restrictions",
                        "custom_volume_snapshot_expiry",
                        "volume_snapshot_scheduling",
                        "trust_ca_certificates",
                        "snapshot_disk_usage",
                        "clustering_edit_roles",
                        "container_nic_routed_host_address",
                        "container_nic_ipvlan_gateway",
                        "resources_usb_pci",
                        "resources_cpu_threads_numa",
                        "resources_cpu_core_die",
                        "api_os",
                        "container_nic_routed_host_table",
                        "container_nic_ipvlan_host_table",
                        "container_nic_ipvlan_mode",
                        "resources_system",
                        "images_push_relay",
                        "network_dns_search",
                        "container_nic_routed_limits",
                        "instance_nic_bridged_vlan",
                        "network_state_bond_bridge",
                        "usedby_consistency",
                        "custom_block_volumes",
                        "clustering_failure_domains",
                        "resources_gpu_mdev",
                        "console_vga_type",
                        "projects_limits_disk",
                        "network_type_macvlan",
                        "network_type_sriov",
                        "container_syscall_intercept_bpf_devices",
                        "network_type_ovn",
                        "projects_networks",
                        "projects_networks_restricted_uplinks",
                        "custom_volume_backup",
                        "backup_override_name",
                        "storage_rsync_compression",
                        "network_type_physical",
                        "network_ovn_external_subnets",
                        "network_ovn_nat",
                        "network_ovn_external_routes_remove"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "192.168.22.3:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\n[....]\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "98e03f957c41f8dbcc093a7499ead1e3dc5c06872b60aa83bb8de5a1ea39f99a",
                        "driver": "lxc",
                        "driver_version": "4.0.5",
                        "firewall": "nftables",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_features": {
                                "netnsid_getifaddrs": "true",
                                "seccomp_listener": "true",
                                "seccomp_listener_continue": "true",
                                "shiftfs": "false",
                                "uevent_injection": "true",
                                "unpriv_fscaps": "true"
                        },
                        "kernel_version": "5.8.0-0.bpo.2-amd64",
                        "lxc_features": {
                                "cgroup2": "true",
                                "devpts_fd": "true",
                                "mount_injection_file": "true",
                                "network_gateway_device_route": "true",
                                "network_ipvlan": "true",
                                "network_l2proxy": "true",
                                "network_phys_macvlan_mtu": "true",
                                "network_veth_router": "true",
                                "pidfd": "true",
                                "seccomp_allow_deny_syntax": "true",
                                "seccomp_notify": "true",
                                "seccomp_proxy_send_notify_fd": "true"
                        },
                        "os_name": "Debian GNU/Linux",
                        "os_version": "10",
                        "project": "default",
                        "server": "lxd",
                        "server_clustered": false,
                        "server_name": "backupserver",
                        "server_pid": 24387,
                        "server_version": "4.7",
                        "storage": "btrfs",
                        "storage_version": "4.15.1"
                }
        }
DBUG[11-10|15:18:09] Sending request to LXD                   method=GET url=http://unix.socket/1.0/instances/gitrepo etag=
DBUG[11-10|15:18:09] Got response struct from LXD
DBUG[11-10|15:18:09]
        {
                "architecture": "x86_64",
                "config": {
                        "boot.autostart": "true",
                        "image.architecture": "amd64",
                        "image.description": "Debian buster amd64 (20191102_05:24)",
                        "image.os": "Debian",
                        "image.release": "buster",
                        "image.serial": "20191102_05:24",
                        "image.type": "squashfs",
                        "raw.idmap": "uid 1007 109\ngid 100 117",
                        "security.nesting": "true",
                        "volatile.base_image": "64e27f01cc55b2b9fc4761fe430884bce51c7df0d61678c802866e3b333d479d",
                        "volatile.eth0.host_name": "veth1233999d",
                        "volatile.eth0.hwaddr": "00:16:3e:f7:77:46",
                        "volatile.idmap.base": "0",
                        "volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.last_state.power": "RUNNING"
                },
                "devices": {
                        "sharedrepos": {
                                "path": "/srv/gitolite/repositories",
                                "source": "/srv/dev-disk-by-label-ssdct/repos",
                                "type": "disk"
                        }
                },
                "ephemeral": false,
                "profiles": [
                        "publicCT",
                        "backupDevice4To"
                ],
                "stateful": false,
                "description": "",
                "created_at": "2020-10-19T12:37:47.622406732+02:00",
                "expanded_config": {
                        "boot.autostart": "true",
                        "environment.TZ": "Europe/Paris",
                        "image.architecture": "amd64",
                        "image.description": "Debian buster amd64 (20191102_05:24)",
                        "image.os": "Debian",
                        "image.release": "buster",
                        "image.serial": "20191102_05:24",
                        "image.type": "squashfs",
                        "raw.idmap": "uid 1007 109\ngid 100 117",
                        "security.nesting": "true",
                        "user.comment": "Stockage sur disque 4To",
                        "volatile.base_image": "64e27f01cc55b2b9fc4761fe430884bce51c7df0d61678c802866e3b333d479d",
                        "volatile.eth0.host_name": "veth1233999d",
                        "volatile.eth0.hwaddr": "00:16:3e:f7:77:46",
                        "volatile.idmap.base": "0",
                        "volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.last_state.power": "RUNNING"
                },
                "expanded_devices": {
                        "eth0": {
                                "name": "eth0",
                                "nictype": "bridged",
                                "parent": "lxdbr0",
                                "type": "nic"
                        },
                        "root": {
                                "path": "/",
                                "pool": "default",
                                "type": "disk"
                        },
                        "sharedrepos": {
                                "path": "/srv/gitolite/repositories",
                                "source": "/srv/dev-disk-by-label-ssdct/repos",
                                "type": "disk"
                        }
                },
                "name": "gitrepo",
                "status": "Running",
                "status_code": 103,
                "last_used_at": "2020-10-30T09:40:08.492196847+01:00",
                "location": "none",
                "type": "container"
        }
DBUG[11-10|15:18:09] Connected to the websocket: ws://unix.socket/1.0/events
DBUG[11-10|15:18:09] Sending request to LXD                   method=POST url=http://unix.socket/1.0/instances/gitrepo etag=
DBUG[11-10|15:18:09]
        {
                "name": "",
                "migration": true,
                "live": false,
                "instance_only": false,
                "container_only": false,
                "target": null
        }
DBUG[11-10|15:18:09] Got operation from LXD
DBUG[11-10|15:18:09]
        {
                "id": "288924ef-b335-4a27-b2e6-8f7782986345",
                "class": "websocket",
                "description": "Migrating container",
                "created_at": "2020-11-10T15:18:09.467979344+01:00",
                "updated_at": "2020-11-10T15:18:09.467979344+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/gitrepo"
                        ],
                        "instances": [
                                "/1.0/instances/gitrepo"
                        ]
                },
                "metadata": {
                        "control": "06ed8e92aa5f97814233ea1b3ea3a7574c8a0524b2f685f03ac77abba8573103",
                        "fs": "3540cbcf3654b2e8445fed26a68a687964c5cf2034833ea9ea11c76ff139be8b"
                },
                "may_cancel": true,
                "err": "",
                "location": "none"
        }
DBUG[11-10|15:18:09] Connected to the websocket: wss://192.168.22.3:8443/1.0/events
DBUG[11-10|15:18:09] Sending request to LXD                   method=POST url=https://192.168.22.3:8443/1.0/instances etag=
DBUG[11-10|15:18:09]
        {
                "architecture": "x86_64",
                "config": {
                        "boot.autostart": "true",
                        "image.architecture": "amd64",
                        "image.description": "Debian buster amd64 (20191102_05:24)",
                        "image.os": "Debian",
                        "image.release": "buster",
                        "image.serial": "20191102_05:24",
                        "image.type": "squashfs",
                        "raw.idmap": "uid 1007 109\ngid 100 117",
                        "security.nesting": "true",
                        "volatile.base_image": "64e27f01cc55b2b9fc4761fe430884bce51c7df0d61678c802866e3b333d479d",
                        "volatile.eth0.host_name": "veth1233999d",
                        "volatile.eth0.hwaddr": "00:16:3e:f7:77:46",
                        "volatile.idmap.base": "0",
                        "volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]"
                },
                "devices": {
                        "sharedrepos": {
                                "path": "/srv/gitolite/repositories",
                                "source": "/srv/repos",
                                "type": "disk"
                        }
                },
                "ephemeral": false,
                "profiles": [
                        "publicCT",
                        "backupDevice4To"
                ],
                "stateful": false,
                "description": "",
                "name": "gitrepo",
                "source": {
                        "type": "migration",
                        "certificate": "-----BEGIN CERTIFICATE-----\n[...]\n-----END CERTIFICATE-----\n",
                        "base-image": "64e27f01cc55b2b9fc4761fe430884bce51c7df0d61678c802866e3b333d479d",
                        "mode": "pull",
                        "operation": "https://10.4.0.2:8443/1.0/operations/288924ef-b335-4a27-b2e6-8f7782986345",
                        "secrets": {
                                "control": "06ed8e92aa5f97814233ea1b3ea3a7574c8a0524b2f685f03ac77abba8573103",
                                "fs": "3540cbcf3654b2e8445fed26a68a687964c5cf2034833ea9ea11c76ff139be8b"
                        },
                        "refresh": true
                },
                "instance_type": "",
                "type": "container"
        }
DBUG[11-10|15:18:09] Got operation from LXD
DBUG[11-10|15:18:09]
        {
                "id": "20ef6dfd-b6e2-4118-a05e-fef0e8a6c7f1",
                "class": "task",
                "description": "Creating container",
                "created_at": "2020-11-10T15:18:09.769247208+01:00",
                "updated_at": "2020-11-10T15:18:09.769247208+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/gitrepo"
                        ],
                        "instances": [
                                "/1.0/instances/gitrepo"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
DBUG[11-10|15:18:09] Sending request to LXD                   method=GET url=https://192.168.22.3:8443/1.0/operations/20ef6dfd-b6e2-4118-a05e-fef0e8a6c7f1 etag=
DBUG[11-10|15:18:09] Got response struct from LXD
DBUG[11-10|15:18:09]
        {
                "id": "20ef6dfd-b6e2-4118-a05e-fef0e8a6c7f1",
                "class": "task",
                "description": "Creating container",
                "created_at": "2020-11-10T15:18:09.769247208+01:00",
                "updated_at": "2020-11-10T15:18:09.769247208+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/gitrepo"
                        ],
                        "instances": [
                                "/1.0/instances/gitrepo"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
DBUG[11-10|15:18:19] Sending request to LXD                   method=POST url=https://192.168.22.3:8443/1.0/instances etag=
DBUG[11-10|15:18:19]
        {
                "architecture": "x86_64",
                "config": {
                        "boot.autostart": "true",
                        "image.architecture": "amd64",
                        "image.description": "Debian buster amd64 (20191102_05:24)",
                        "image.os": "Debian",
                        "image.release": "buster",
                        "image.serial": "20191102_05:24",
                        "image.type": "squashfs",
                        "raw.idmap": "uid 1007 109\ngid 100 117",
                        "security.nesting": "true",
                        "volatile.base_image": "64e27f01cc55b2b9fc4761fe430884bce51c7df0d61678c802866e3b333d479d",
                        "volatile.eth0.host_name": "veth1233999d",
                        "volatile.eth0.hwaddr": "00:16:3e:f7:77:46",
                        "volatile.idmap.base": "0",
                        "volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]",
                        "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":109},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1007,\"Nsid\":109,\"Maprange\":1},{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000110,\"Nsid\":110,\"Maprange\":999999890},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":117},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100,\"Nsid\":117,\"Maprange\":1},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000118,\"Nsid\":118,\"Maprange\":999999882}]"
                },
                "devices": {
                        "sharedrepos": {
                                "path": "/srv/gitolite/repositories",
                                "source": "/srv/repos",
                                "type": "disk"
                        }
                },
                "ephemeral": false,
                "profiles": [
                        "publicCT",
                        "backupDevice4To"
                ],
                "stateful": false,
                "description": "",
                "name": "gitrepo",
                "source": {
                        "type": "migration",
                        "certificate": "-----BEGIN CERTIFICATE-----\n[...]\n-----END CERTIFICATE-----\n",
                        "base-image": "64e27f01cc55b2b9fc4761fe430884bce51c7df0d61678c802866e3b333d479d",
                        "mode": "pull",
                        "operation": "https://192.168.22.2:8443/1.0/operations/288924ef-b335-4a27-b2e6-8f7782986345",
                        "secrets": {
                                "control": "06ed8e92aa5f97814233ea1b3ea3a7574c8a0524b2f685f03ac77abba8573103",
                                "fs": "3540cbcf3654b2e8445fed26a68a687964c5cf2034833ea9ea11c76ff139be8b"
                        },
                        "refresh": true
                },
                "instance_type": "",
                "type": "container"
        }
DBUG[11-10|15:18:20] Got operation from LXD
DBUG[11-10|15:18:20]
        {
                "id": "0f3dece7-b9c5-4b66-a4c3-c11b671134bc",
                "class": "task",
                "description": "Creating container",
                "created_at": "2020-11-10T15:18:19.985111308+01:00",
                "updated_at": "2020-11-10T15:18:19.985111308+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/gitrepo"
                        ],
                        "instances": [
                                "/1.0/instances/gitrepo"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
DBUG[11-10|15:18:20] Sending request to LXD                   method=GET url=https://192.168.22.3:8443/1.0/operations/0f3dece7-b9c5-4b66-a4c3-c11b671134bc etag=
DBUG[11-10|15:18:20] Got response struct from LXD
DBUG[11-10|15:18:20]
        {
                "id": "0f3dece7-b9c5-4b66-a4c3-c11b671134bc",
                "class": "task",
                "description": "Creating container",
                "created_at": "2020-11-10T15:18:19.985111308+01:00",
                "updated_at": "2020-11-10T15:18:19.985111308+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/gitrepo"
                        ],
                        "instances": [
                                "/1.0/instances/gitrepo"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": "",
                "location": "none"
        }
DBUG[11-10|15:18:20] Sending request to LXD                   method=DELETE url=http://unix.socket/1.0/operations/288924ef-b335-4a27-b2e6-8f7782986345 etag=
Error: Failed instance creation:
 - https://10.4.0.2:8443: Error transferring instance data: Unable to connect to: 10.4.0.2:8443
 - https://192.168.22.2:8443: Error transferring instance data: Invalid devices: Device validation failed for "sharedrepos": Missing source "/srv/dev-disk-by-label-ssdct/repos" for disk "sharedrepos"

There are two error messages. First one is for LAN address (10.4.x.x), second one for vpn address (192.168.x.x.).

The override clearly appears in the POST /1.0/instances API call above so the client seems to be doing the right thing, so this is most likely a validation issue somewhere on the receiving side somehow.

@tomp can you take this one?

Yep I will take a look.

Thank you @tomp . If you need further tests on my own servers, please tell me.

I’ve just tried this now on LXD edge channel and I cannot reproduce.

Here is my scenario. Two nodes; v1 and v2.

On v1:

apt install snapd -y; snap install lxd --edge
lxd init --auto
lxc config set core.https_address [::]:8443
lxc config set core.trust_password pw

Create container on v1 and add disk linked to /mnt/test1:

lxc init images:alpine/3.12 c1
mkdir /mnt/test1
lxc config device add c1 vol1 disk source=/mnt/test1 path=/mnt
lxc start c1

On v2:

apt install snapd -y; snap install lxd --edge
lxd init --auto
lxc config set core.https_address [::]:8443
lxc config set core.trust_password pw
mkdir /mnt/test2

On v1:

lxc remote add v2 10.163.71.70

# Expected to fail as no /mnt/test1 on target machine.
lxc copy c1 v2: --stateless
Error: Failed instance creation: Invalid devices: Device validation failed for "vol1": Missing source "/mnt/test1" for disk "vol1"

# Completes OK
lxc copy c1 v2: --refresh --stateless -d vol1,source=/mnt/test2
lxc config show v2:c1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Alpine 3.12 amd64 (20201113_13:00)
  image.os: Alpine
  image.release: "3.12"
  image.serial: "20201113_13:00"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 59de5eb5e9a468eebdd49703fa2bbc218d73ed0b11ad00cc010850e327f4da48
  volatile.eth0.host_name: veth6a76c0b7
  volatile.eth0.hwaddr: 00:16:3e:97:78:c5
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
devices:
  vol1:
    path: /mnt/
    source: /mnt/test2
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

This is with a dir pool, so will try on LXD 4.7 and BTRFS to align with report.

Yeah it works fine on LXD 4.7 with BTRFS.

Can you show where the original disk device is defined, is it in the containers config or the profiles?

Can you show output of lxc config show gitrepo and lxc config show gitrepo --expanded so I can see the difference.

@tomp: on my two nodes, the initial copy works fine, it’s the updates afterward which fail. Your command lxc copy c1 v2: --refresh --stateless -d vol1,source=/mnt/test2 succeeded but that was actually an initial copy and not an update even if the ‘refresh’ keyword was there. Do you see what I mean? If you retry your command to make a true update, does it work?

To answer your question, the original disk device is defined in the container config as you can see below.

Here are the configs:

lxc config show gitrepo

architecture: x86_64
config:
boot.autostart: “true”
image.architecture: amd64
image.description: Debian buster amd64 (20191102_05:24)
image.os: Debian
image.release: buster
image.serial: “20191102_05:24”
image.type: squashfs
raw.idmap: |-
uid 1007 109
gid 100 117
security.nesting: “true”
volatile.base_image: 64e27f01cc55b2b9fc4761fe430884bce51c7df0d61678c802866e3b333d479d
volatile.eth0.host_name: veth1233999d
volatile.eth0.hwaddr: 00:16:3e:f7:77:46
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:109},{“Isuid”:true,“Isgid”:false,“Hostid”:1007,“Nsid”:109,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:1000110,“Nsid”:110,“Maprange”:999999890},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:117},{“Isuid”:false,“Isgid”:true,“Hostid”:100,“Nsid”:117,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:1000118,“Nsid”:118,“Maprange”:999999882}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:109},{“Isuid”:true,“Isgid”:false,“Hostid”:1007,“Nsid”:109,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:1000110,“Nsid”:110,“Maprange”:999999890},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:117},{“Isuid”:false,“Isgid”:true,“Hostid”:100,“Nsid”:117,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:1000118,“Nsid”:118,“Maprange”:999999882}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:109},{“Isuid”:true,“Isgid”:false,“Hostid”:1007,“Nsid”:109,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:1000110,“Nsid”:110,“Maprange”:999999890},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:117},{“Isuid”:false,“Isgid”:true,“Hostid”:100,“Nsid”:117,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:1000118,“Nsid”:118,“Maprange”:999999882}]’
volatile.last_state.power: RUNNING
devices:
sharedrepos:
path: /srv/gitolite/repositories
source: /srv/dev-disk-by-label-ssdct/repos
type: disk
ephemeral: false
profiles:

  • publicCT
  • backupDevice4To
    stateful: false
    description: “”
lxc config show gitrepo --expanded

architecture: x86_64
config:
boot.autostart: “true”
environment.TZ: Europe/Paris
image.architecture: amd64
image.description: Debian buster amd64 (20191102_05:24)
image.os: Debian
image.release: buster
image.serial: “20191102_05:24”
image.type: squashfs
raw.idmap: |-
uid 1007 109
gid 100 117
security.nesting: “true”
user.comment: Stockage sur disque 4To
volatile.base_image: 64e27f01cc55b2b9fc4761fe430884bce51c7df0d61678c802866e3b333d479d
volatile.eth0.host_name: veth1233999d
volatile.eth0.hwaddr: 00:16:3e:f7:77:46
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:109},{“Isuid”:true,“Isgid”:false,“Hostid”:1007,“Nsid”:109,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:1000110,“Nsid”:110,“Maprange”:999999890},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:117},{“Isuid”:false,“Isgid”:true,“Hostid”:100,“Nsid”:117,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:1000118,“Nsid”:118,“Maprange”:999999882}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:109},{“Isuid”:true,“Isgid”:false,“Hostid”:1007,“Nsid”:109,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:1000110,“Nsid”:110,“Maprange”:999999890},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:117},{“Isuid”:false,“Isgid”:true,“Hostid”:100,“Nsid”:117,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:1000118,“Nsid”:118,“Maprange”:999999882}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:109},{“Isuid”:true,“Isgid”:false,“Hostid”:1007,“Nsid”:109,“Maprange”:1},{“Isuid”:true,“Isgid”:false,“Hostid”:1000110,“Nsid”:110,“Maprange”:999999890},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:117},{“Isuid”:false,“Isgid”:true,“Hostid”:100,“Nsid”:117,“Maprange”:1},{“Isuid”:false,“Isgid”:true,“Hostid”:1000118,“Nsid”:118,“Maprange”:999999882}]’
volatile.last_state.power: RUNNING
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
sharedrepos:
path: /srv/gitolite/repositories
source: /srv/dev-disk-by-label-ssdct/repos
type: disk
ephemeral: false
profiles:

  • publicCT
  • backupDevice4To
    stateful: false
    description: “”

The command runs successfully on each subsequent run with --refresh.

However I have managed to recreate the issue I believe, it is to do with copying snapshots.

Does your container have any snapshots? lxc info <container>?

@stgraber so this issue is caused by the snapshot’s config not being overridden by the -d argument.

This does make some sense, as a snapshot is supposed to be read-only, and there’s no guarantee that the device exists in the snapshot to be overridden anyway.

However, if I relax the validation to exclude snapshots, then the migration succeeds, but later when lxc restore c1 snap0 is run the invalid device in the snapshot prevents the snapshot from being restored:

lxc restore c1 snap0
Error: Failed to add device "vol1": Missing source "/mnt/test1" for disk "vol1"

So this makes the argument for using the supplied device overrides during migration to modify the snapshots being sent.

Presumably, we would need to check whether the snapshot actually has that device, and whether the device type matches before modifying it.

The other approach we could use is to allow the snapshot’s config to be modified using lxc config edit c1/snap0. Today this appears to work, but silently discards any device config made (which in itself seems to me a bug/bad user experience).

Another approach could be to add the -d flag to the lxc restore command allowing for a snapshot’s config to be modified as its restored.

What do you think?

I think there’s a simpler solution. A snapshot can only ever be restored on the same storage pool as its parent instance. So the root device must be mangled if it’s incorrect.

For all other disks, we don’t do anything and if they’ve changed or are now incorrect, startup will fail and the user can fix it.

Fully agree with both of you.

What’s important is that the remote copy and refresh can be done.

For the restore, provided that the user gets a clear enough message about why the restore fails and what device is concerned, it’s almost easy to fix it by hand.

Additionally a -d flag in lxc restore like @tomp describes would be cool, as it would ease things and make them scriptable also.

I think you’re saying we skip that particular check when validating a disk device without a ‘real’ instance, which is trivially done.

However in my tests, once I add that exception, the migration succeeds, but the snapshot restore later fails because it calls the Update() function on the instance to apply the snapshot’s config, and that in turn attempts to load each device (in order to call it’s Add() function to perform any “create time” setup of the device) which then fails device validation.

We have decided to modify the error validation to only check whether the path exists when an instance is loaded, rather than during profile or creation validation (which includes the migration process).

This will allow the migration to complete.

For the snapshot restore issue, LXD already has the concept of a “user requested update” (i.e lxc config ...) vs a “non-user requested update” (such as doing snapshot restore that modifies the config of the existing instance).

For non-user requested updates, if the device add process fails we will not fail the whole process, but instead just log the error and skip that device’s setup .

This will allow the snapshot restoration to succeed and then the container will not be able to be started, at which point the config can be fixed using the normal container config commands lxc config ....

I’ve also improved the error messages so that if a device is failing validation during migration due to some other snaphot config issue the error reported to the user will include the snapshot name, making it clearer to see what the problem is.

The changes are included in this PR:

In the meantime until this gets applied to a release, if you ensure the directory exists on the target node you should be able to complete the migration (and the migrated container will still use the desired directory specified).

Perfect. Thank you.
Have you an idea of which release it will be part of? Next one?

Yes it will be in LXD 4.9. It may be in sooner in the snap package as part of @stgraber cherry-picking of fixes inbetween releases.

If you want to test it you can switch the snap to the edge channel, although be aware that if there is a database schema change or patch between 4.8 and edge then you won’t be able to downgrade again.

sudo snap refresh lxd --channel=latest/edge

I’ll wait for 4.9, and in the meantime play with the advised hack. Thank you again for your time & effort.

Yeah, the fixes will hit 4.8 probably early next week when I go through all our current fixes and backport them.