How can I copy an instance between projects?

Incus copy has the following usage description:

pargo@bastion:~$ incus copy
Description:
  Copy instances within or in between servers

  Transfer modes (--mode):
   - pull: Target server pulls the data from the source server (source must listen on network)
   - push: Source server pushes the data to the target server (target must listen on network)
   - relay: The CLI connects to both source and server and proxies the data (both source and target must listen on network)

  The pull transfer mode is the default as it is compatible with all server versions.

Usage:
  incus copy [<remote>:]<source>[/<snapshot>] [[<remote>:]<destination>] [flags]

Aliases:
  copy, cp

Flags:
      --allow-inconsistent   Ignore copy errors for volatile files
  -c, --config               Config key/value to apply to the new instance
  -d, --device               New key/value to apply to a specific device
  -e, --ephemeral            Ephemeral instance
      --instance-only        Copy the instance without its snapshots
      --mode                 Transfer mode. One of pull, push or relay (default "pull")
      --no-profiles          Create the instance with no profiles applied
  -p, --profile              Profile to apply to the new instance
      --refresh              Perform an incremental copy
      --stateless            Copy a stateful instance stateless
  -s, --storage              Storage pool name
      --target               Cluster member name
      --target-project       Copy to a project different from the source

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

According to the usage description, incus copy <source> --target-project <target project> should work but I get the following.

pargo@bastion:~$ incus list
+-------------------+---------+--------------------+------+-----------+-----------+------------+
|       NAME        |  STATE  |        IPV4        | IPV6 |   TYPE    | SNAPSHOTS |  LOCATION  |
+-------------------+---------+--------------------+------+-----------+-----------+------------+
| pedrojorge-PAPCMC | STOPPED |                    |      | CONTAINER | 0         | dedicado01 |
+-------------------+---------+--------------------+------+-----------+-----------+------------+
| robertty-exp      | RUNNING | 10.11.21.70 (eth0) |      | CONTAINER | 0         | dedicado02 |
+-------------------+---------+--------------------+------+-----------+-----------+------------+
| robertty-exp2     | RUNNING | 10.11.21.71 (eth0) |      | CONTAINER | 0         | dedicado03 |
+-------------------+---------+--------------------+------+-----------+-----------+------------+
pargo@bastion:~$ incus project list
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
|        NAME        | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES |      DESCRIPTION      | USED BY |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| compartilhado      | YES    | YES      | YES             | YES             | NO       | NO            |                       | 2       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| dedicado (current) | YES    | YES      | YES             | YES             | NO       | NO            |                       | 4       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| default            | YES    | YES      | YES             | YES             | YES      | YES           | Default Incus project | 8       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| sandbox            | YES    | YES      | YES             | YES             | NO       | NO            |                       | 5       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
| services           | YES    | YES      | YES             | YES             | NO       | NO            |                       | 7       |
+--------------------+--------+----------+-----------------+-----------------+----------+---------------+-----------------------+---------+
pargo@bastion:~$ incus copy pedrojorge-PAPCMC --target-project compartilhado
Error: You must specify a destination instance name

That’s slighly odd since I thought the brackets meant the destination instance name was optional.

With a destination instance name I get the following.

pargo@bastion:~$ incus copy pedrojorge-PAPCMC pedrojorge-PAPCMC2 --target-project compartilhado
Error: Failed to get address of instance's member: Instance not found

I can´t get the copy command to do what it was supposed to do.

Some information on what I’m running. It’s a cluster and I’ll post the information about both projects.

pargo@bastion:~$ incus cluster list
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
|      NAME       |           URL            |      ROLES       | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATE  |      MESSAGE      |
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| compartilhado01 | https://10.11.16.11:8443 | database         | x86_64       | default        |             | ONLINE | Fully operational |
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| compartilhado02 | https://10.11.16.12:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| compartilhado03 | https://10.11.16.13:8443 | database         | x86_64       | default        |             | ONLINE | Fully operational |
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| compartilhado04 | https://10.11.16.14:8443 | database-leader  | x86_64       | default        |             | ONLINE | Fully operational |
|                 |                          | database         |              |                |             |        |                   |
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| dedicado01      | https://10.11.16.31:8443 |                  | x86_64       | default        |             | ONLINE | Fully operational |
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| dedicado02      | https://10.11.16.32:8443 | database-standby | x86_64       | default        |             | ONLINE | Fully operational |
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
| dedicado03      | https://10.11.16.33:8443 |                  | x86_64       | default        |             | ONLINE | Fully operational |
+-----------------+--------------------------+------------------+--------------+----------------+-------------+--------+-------------------+
pargo@bastion:~$ incus cluster group show compartilhado
description: ""
members:
- compartilhado01
- compartilhado02
- compartilhado03
- compartilhado04
name: compartilhado
pargo@bastion:~$ incus cluster group show dedicado
description: ""
members:
- dedicado01
- dedicado02
- dedicado03
name: dedicado
pargo@bastion:~$ incus project show compartilhado
config:
  features.images: "true"
  features.profiles: "true"
  features.storage.buckets: "true"
  features.storage.volumes: "true"
  limits.instances: "4"
  restricted: "true"
  restricted.backups: allow
  restricted.cluster.groups: compartilhado
  restricted.cluster.target: allow
  restricted.containers.nesting: allow
  restricted.devices.nic: allow
  restricted.snapshots: allow
description: ""
name: compartilhado
used_by:
- /1.0/profiles/default?project=compartilhado
- /1.0/images/77f67d56a4ba8e351e1b0d22034b65700a5e02ea28d0ae7f04d268c7a6a2bd2a?project=compartilhado
pargo@bastion:~$ incus project show dedicado
config:
  features.images: "true"
  features.profiles: "true"
  features.storage.buckets: "true"
  features.storage.volumes: "true"
  limits.instances: "3"
  restricted: "true"
  restricted.backups: allow
  restricted.cluster.groups: dedicado
  restricted.cluster.target: allow
  restricted.containers.nesting: allow
  restricted.devices.nic: allow
  restricted.snapshots: allow
description: ""
name: dedicado
used_by:
- /1.0/instances/pedrojorge-PAPCMC?project=dedicado
- /1.0/instances/robertty-exp?project=dedicado
- /1.0/instances/robertty-exp2?project=dedicado
- /1.0/profiles/default?project=dedicado

What am I doing wrong?

stgraber@castiana:~$ incus create images:alpine/edge a1
Creating a1
stgraber@castiana:~$ incus copy a1 a2 --target-project demo
stgraber@castiana:~$ 

So works fine locally, I suspect it may be a cluster issue.

Can you post the incus copy output with --debug?

pargo@bastion:~$ incus copy pedrojorge-PAPCMC pedrojorge-PAPCMC2 --target-project compartilhado --debug
DEBUG  [2024-02-14T20:56:24-03:00] Connecting to a remote Incus over HTTPS       url="https://10.11.16.11:8443"
DEBUG  [2024-02-14T20:56:24-03:00] Sending request to Incus                      etag= method=GET url="https://10.11.16.11:8443/1.0"
DEBUG  [2024-02-14T20:56:24-03:00] Got response struct from Incus               
DEBUG  [2024-02-14T20:56:24-03:00] 
	{
		"config": {
			"cluster.https_address": "10.11.16.11:8443",
			"core.https_address": "10.11.16.11:8443"
		},
		"api_extensions": [
			"storage_zfs_remove_snapshots",
			"container_host_shutdown_timeout",
			"container_stop_priority",
			"container_syscall_filtering",
			"auth_pki",
			"container_last_used_at",
			"etag",
			"patch",
			"usb_devices",
			"https_allowed_credentials",
			"image_compression_algorithm",
			"directory_manipulation",
			"container_cpu_time",
			"storage_zfs_use_refquota",
			"storage_lvm_mount_options",
			"network",
			"profile_usedby",
			"container_push",
			"container_exec_recording",
			"certificate_update",
			"container_exec_signal_handling",
			"gpu_devices",
			"container_image_properties",
			"migration_progress",
			"id_map",
			"network_firewall_filtering",
			"network_routes",
			"storage",
			"file_delete",
			"file_append",
			"network_dhcp_expiry",
			"storage_lvm_vg_rename",
			"storage_lvm_thinpool_rename",
			"network_vlan",
			"image_create_aliases",
			"container_stateless_copy",
			"container_only_migration",
			"storage_zfs_clone_copy",
			"unix_device_rename",
			"storage_lvm_use_thinpool",
			"storage_rsync_bwlimit",
			"network_vxlan_interface",
			"storage_btrfs_mount_options",
			"entity_description",
			"image_force_refresh",
			"storage_lvm_lv_resizing",
			"id_map_base",
			"file_symlinks",
			"container_push_target",
			"network_vlan_physical",
			"storage_images_delete",
			"container_edit_metadata",
			"container_snapshot_stateful_migration",
			"storage_driver_ceph",
			"storage_ceph_user_name",
			"resource_limits",
			"storage_volatile_initial_source",
			"storage_ceph_force_osd_reuse",
			"storage_block_filesystem_btrfs",
			"resources",
			"kernel_limits",
			"storage_api_volume_rename",
			"network_sriov",
			"console",
			"restrict_dev_incus",
			"migration_pre_copy",
			"infiniband",
			"dev_incus_events",
			"proxy",
			"network_dhcp_gateway",
			"file_get_symlink",
			"network_leases",
			"unix_device_hotplug",
			"storage_api_local_volume_handling",
			"operation_description",
			"clustering",
			"event_lifecycle",
			"storage_api_remote_volume_handling",
			"nvidia_runtime",
			"container_mount_propagation",
			"container_backup",
			"dev_incus_images",
			"container_local_cross_pool_handling",
			"proxy_unix",
			"proxy_udp",
			"clustering_join",
			"proxy_tcp_udp_multi_port_handling",
			"network_state",
			"proxy_unix_dac_properties",
			"container_protection_delete",
			"unix_priv_drop",
			"pprof_http",
			"proxy_haproxy_protocol",
			"network_hwaddr",
			"proxy_nat",
			"network_nat_order",
			"container_full",
			"backup_compression",
			"nvidia_runtime_config",
			"storage_api_volume_snapshots",
			"storage_unmapped",
			"projects",
			"network_vxlan_ttl",
			"container_incremental_copy",
			"usb_optional_vendorid",
			"snapshot_scheduling",
			"snapshot_schedule_aliases",
			"container_copy_project",
			"clustering_server_address",
			"clustering_image_replication",
			"container_protection_shift",
			"snapshot_expiry",
			"container_backup_override_pool",
			"snapshot_expiry_creation",
			"network_leases_location",
			"resources_cpu_socket",
			"resources_gpu",
			"resources_numa",
			"kernel_features",
			"id_map_current",
			"event_location",
			"storage_api_remote_volume_snapshots",
			"network_nat_address",
			"container_nic_routes",
			"cluster_internal_copy",
			"seccomp_notify",
			"lxc_features",
			"container_nic_ipvlan",
			"network_vlan_sriov",
			"storage_cephfs",
			"container_nic_ipfilter",
			"resources_v2",
			"container_exec_user_group_cwd",
			"container_syscall_intercept",
			"container_disk_shift",
			"storage_shifted",
			"resources_infiniband",
			"daemon_storage",
			"instances",
			"image_types",
			"resources_disk_sata",
			"clustering_roles",
			"images_expiry",
			"resources_network_firmware",
			"backup_compression_algorithm",
			"ceph_data_pool_name",
			"container_syscall_intercept_mount",
			"compression_squashfs",
			"container_raw_mount",
			"container_nic_routed",
			"container_syscall_intercept_mount_fuse",
			"container_disk_ceph",
			"virtual-machines",
			"image_profiles",
			"clustering_architecture",
			"resources_disk_id",
			"storage_lvm_stripes",
			"vm_boot_priority",
			"unix_hotplug_devices",
			"api_filtering",
			"instance_nic_network",
			"clustering_sizing",
			"firewall_driver",
			"projects_limits",
			"container_syscall_intercept_hugetlbfs",
			"limits_hugepages",
			"container_nic_routed_gateway",
			"projects_restrictions",
			"custom_volume_snapshot_expiry",
			"volume_snapshot_scheduling",
			"trust_ca_certificates",
			"snapshot_disk_usage",
			"clustering_edit_roles",
			"container_nic_routed_host_address",
			"container_nic_ipvlan_gateway",
			"resources_usb_pci",
			"resources_cpu_threads_numa",
			"resources_cpu_core_die",
			"api_os",
			"container_nic_routed_host_table",
			"container_nic_ipvlan_host_table",
			"container_nic_ipvlan_mode",
			"resources_system",
			"images_push_relay",
			"network_dns_search",
			"container_nic_routed_limits",
			"instance_nic_bridged_vlan",
			"network_state_bond_bridge",
			"usedby_consistency",
			"custom_block_volumes",
			"clustering_failure_domains",
			"resources_gpu_mdev",
			"console_vga_type",
			"projects_limits_disk",
			"network_type_macvlan",
			"network_type_sriov",
			"container_syscall_intercept_bpf_devices",
			"network_type_ovn",
			"projects_networks",
			"projects_networks_restricted_uplinks",
			"custom_volume_backup",
			"backup_override_name",
			"storage_rsync_compression",
			"network_type_physical",
			"network_ovn_external_subnets",
			"network_ovn_nat",
			"network_ovn_external_routes_remove",
			"tpm_device_type",
			"storage_zfs_clone_copy_rebase",
			"gpu_mdev",
			"resources_pci_iommu",
			"resources_network_usb",
			"resources_disk_address",
			"network_physical_ovn_ingress_mode",
			"network_ovn_dhcp",
			"network_physical_routes_anycast",
			"projects_limits_instances",
			"network_state_vlan",
			"instance_nic_bridged_port_isolation",
			"instance_bulk_state_change",
			"network_gvrp",
			"instance_pool_move",
			"gpu_sriov",
			"pci_device_type",
			"storage_volume_state",
			"network_acl",
			"migration_stateful",
			"disk_state_quota",
			"storage_ceph_features",
			"projects_compression",
			"projects_images_remote_cache_expiry",
			"certificate_project",
			"network_ovn_acl",
			"projects_images_auto_update",
			"projects_restricted_cluster_target",
			"images_default_architecture",
			"network_ovn_acl_defaults",
			"gpu_mig",
			"project_usage",
			"network_bridge_acl",
			"warnings",
			"projects_restricted_backups_and_snapshots",
			"clustering_join_token",
			"clustering_description",
			"server_trusted_proxy",
			"clustering_update_cert",
			"storage_api_project",
			"server_instance_driver_operational",
			"server_supported_storage_drivers",
			"event_lifecycle_requestor_address",
			"resources_gpu_usb",
			"clustering_evacuation",
			"network_ovn_nat_address",
			"network_bgp",
			"network_forward",
			"custom_volume_refresh",
			"network_counters_errors_dropped",
			"metrics",
			"image_source_project",
			"clustering_config",
			"network_peer",
			"linux_sysctl",
			"network_dns",
			"ovn_nic_acceleration",
			"certificate_self_renewal",
			"instance_project_move",
			"storage_volume_project_move",
			"cloud_init",
			"network_dns_nat",
			"database_leader",
			"instance_all_projects",
			"clustering_groups",
			"ceph_rbd_du",
			"instance_get_full",
			"qemu_metrics",
			"gpu_mig_uuid",
			"event_project",
			"clustering_evacuation_live",
			"instance_allow_inconsistent_copy",
			"network_state_ovn",
			"storage_volume_api_filtering",
			"image_restrictions",
			"storage_zfs_export",
			"network_dns_records",
			"storage_zfs_reserve_space",
			"network_acl_log",
			"storage_zfs_blocksize",
			"metrics_cpu_seconds",
			"instance_snapshot_never",
			"certificate_token",
			"instance_nic_routed_neighbor_probe",
			"event_hub",
			"agent_nic_config",
			"projects_restricted_intercept",
			"metrics_authentication",
			"images_target_project",
			"cluster_migration_inconsistent_copy",
			"cluster_ovn_chassis",
			"container_syscall_intercept_sched_setscheduler",
			"storage_lvm_thinpool_metadata_size",
			"storage_volume_state_total",
			"instance_file_head",
			"instances_nic_host_name",
			"image_copy_profile",
			"container_syscall_intercept_sysinfo",
			"clustering_evacuation_mode",
			"resources_pci_vpd",
			"qemu_raw_conf",
			"storage_cephfs_fscache",
			"network_load_balancer",
			"vsock_api",
			"instance_ready_state",
			"network_bgp_holdtime",
			"storage_volumes_all_projects",
			"metrics_memory_oom_total",
			"storage_buckets",
			"storage_buckets_create_credentials",
			"metrics_cpu_effective_total",
			"projects_networks_restricted_access",
			"storage_buckets_local",
			"loki",
			"acme",
			"internal_metrics",
			"cluster_join_token_expiry",
			"remote_token_expiry",
			"init_preseed",
			"storage_volumes_created_at",
			"cpu_hotplug",
			"projects_networks_zones",
			"network_txqueuelen",
			"cluster_member_state",
			"instances_placement_scriptlet",
			"storage_pool_source_wipe",
			"zfs_block_mode",
			"instance_generation_id",
			"disk_io_cache",
			"amd_sev",
			"storage_pool_loop_resize",
			"migration_vm_live",
			"ovn_nic_nesting",
			"oidc",
			"network_ovn_l3only",
			"ovn_nic_acceleration_vdpa",
			"cluster_healing",
			"instances_state_total",
			"auth_user",
			"security_csm",
			"instances_rebuild",
			"numa_cpu_placement",
			"custom_volume_iso",
			"network_allocations",
			"zfs_delegate",
			"storage_api_remote_volume_snapshot_copy",
			"operations_get_query_all_projects",
			"metadata_configuration",
			"syslog_socket",
			"event_lifecycle_name_and_project",
			"instances_nic_limits_priority",
			"disk_initial_volume_configuration",
			"operation_wait",
			"image_restriction_privileged",
			"cluster_internal_custom_volume_copy",
			"disk_io_bus",
			"storage_cephfs_create_missing",
			"instance_move_config",
			"ovn_ssl_config",
			"certificate_description",
			"disk_io_bus_virtio_blk",
			"loki_config_instance",
			"instance_create_start",
			"clustering_evacuation_stop_options",
			"boot_host_shutdown_action",
			"agent_config_drive"
		],
		"api_status": "stable",
		"api_version": "1.0",
		"auth": "trusted",
		"public": false,
		"auth_methods": [
			"tls"
		],
		"auth_user_name": "554ed3318498b6c10700630c18dabffe9b083cc3a65020d742def3f23b4df676",
		"auth_user_method": "tls",
		"environment": {
			"addresses": [
				"10.11.16.11:8443"
			],
			"architectures": [
				"x86_64",
				"i686"
			],
			"certificate": "-----BEGIN CERTIFICATE-----<REDACTED>\n-----END CERTIFICATE-----\n",
			"certificate_fingerprint": "26c636f0f56c8fb9cf4fa986c8190fd1b75b4714b0465d939dec3297eac2dd62",
			"driver": "qemu | lxc",
			"driver_version": "8.2.1 | 5.0.3",
			"firewall": "nftables",
			"kernel": "Linux",
			"kernel_architecture": "x86_64",
			"kernel_features": {
				"idmapped_mounts": "true",
				"netnsid_getifaddrs": "true",
				"seccomp_listener": "true",
				"seccomp_listener_continue": "true",
				"uevent_injection": "true",
				"unpriv_fscaps": "true"
			},
			"kernel_version": "6.1.0-17-amd64",
			"lxc_features": {
				"cgroup2": "true",
				"core_scheduling": "true",
				"devpts_fd": "true",
				"idmapped_mounts_v2": "true",
				"mount_injection_file": "true",
				"network_gateway_device_route": "true",
				"network_ipvlan": "true",
				"network_l2proxy": "true",
				"network_phys_macvlan_mtu": "true",
				"network_veth_router": "true",
				"pidfd": "true",
				"seccomp_allow_deny_syntax": "true",
				"seccomp_notify": "true",
				"seccomp_proxy_send_notify_fd": "true"
			},
			"os_name": "Debian GNU/Linux",
			"os_version": "12",
			"project": "default",
			"server": "incus",
			"server_clustered": true,
			"server_event_mode": "full-mesh",
			"server_name": "compartilhado01",
			"server_pid": 124875,
			"server_version": "0.5.1",
			"storage": "btrfs",
			"storage_version": "6.2",
			"storage_supported_drivers": [
				{
					"Name": "btrfs",
					"Version": "6.2",
					"Remote": false
				},
				{
					"Name": "dir",
					"Version": "1",
					"Remote": false
				}
			]
		}
	} 
DEBUG  [2024-02-14T20:56:24-03:00] Sending request to Incus                      etag= method=GET url="https://10.11.16.11:8443/1.0/instances/pedrojorge-PAPCMC?project=dedicado"
DEBUG  [2024-02-14T20:56:24-03:00] Got response struct from Incus               
DEBUG  [2024-02-14T20:56:24-03:00] 
	{
		"architecture": "x86_64",
		"config": {
			"image.architecture": "amd64",
			"image.description": "Ubuntu jammy amd64 (20231128_07:42)",
			"image.os": "Ubuntu",
			"image.release": "jammy",
			"image.serial": "20231128_07:42",
			"image.type": "squashfs",
			"image.variant": "cloud",
			"volatile.base_image": "81b028b2bb98a8c11d9c03d49ea67b432b7fbb5549997beb7af0f4d139125df7",
			"volatile.cloud-init.instance-id": "001931da-9b6d-4c22-9c1a-e834e74ff4a4",
			"volatile.eth0.host_name": "veth876daf36",
			"volatile.eth0.hwaddr": "00:16:3e:d7:07:86",
			"volatile.idmap.base": "0",
			"volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]",
			"volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]",
			"volatile.last_state.idmap": "[]",
			"volatile.last_state.power": "RUNNING",
			"volatile.last_state.ready": "false",
			"volatile.uuid": "2e6d0cfc-c7e0-4b3e-bbea-ff408c724590",
			"volatile.uuid.generation": "2e6d0cfc-c7e0-4b3e-bbea-ff408c724590"
		},
		"devices": {
			"root": {
				"path": "/",
				"pool": "local",
				"type": "disk"
			}
		},
		"ephemeral": false,
		"profiles": [
			"default"
		],
		"stateful": false,
		"description": "",
		"created_at": "2024-02-08T17:01:34.694452371Z",
		"expanded_config": {
			"cloud-init.vendor-data": "#cloud-config\npackage_update: true\npackage_upgrade: true\ntimezone: America/Fortaleza",
			"image.architecture": "amd64",
			"image.description": "Ubuntu jammy amd64 (20231128_07:42)",
			"image.os": "Ubuntu",
			"image.release": "jammy",
			"image.serial": "20231128_07:42",
			"image.type": "squashfs",
			"image.variant": "cloud",
			"security.nesting": "true",
			"volatile.base_image": "81b028b2bb98a8c11d9c03d49ea67b432b7fbb5549997beb7af0f4d139125df7",
			"volatile.cloud-init.instance-id": "001931da-9b6d-4c22-9c1a-e834e74ff4a4",
			"volatile.eth0.host_name": "veth876daf36",
			"volatile.eth0.hwaddr": "00:16:3e:d7:07:86",
			"volatile.idmap.base": "0",
			"volatile.idmap.current": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]",
			"volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]",
			"volatile.last_state.idmap": "[]",
			"volatile.last_state.power": "RUNNING",
			"volatile.last_state.ready": "false",
			"volatile.uuid": "2e6d0cfc-c7e0-4b3e-bbea-ff408c724590",
			"volatile.uuid.generation": "2e6d0cfc-c7e0-4b3e-bbea-ff408c724590"
		},
		"expanded_devices": {
			"eth0": {
				"name": "eth0",
				"nictype": "bridged",
				"parent": "br0",
				"type": "nic"
			},
			"root": {
				"path": "/",
				"pool": "local",
				"type": "disk"
			}
		},
		"name": "pedrojorge-PAPCMC",
		"status": "Running",
		"status_code": 103,
		"last_used_at": "2024-02-14T22:19:46.459572608Z",
		"location": "dedicado01",
		"type": "container",
		"project": "dedicado"
	} 
DEBUG  [2024-02-14T20:56:24-03:00] Connected to the websocket: wss://10.11.16.11:8443/1.0/events?project=compartilhado 
DEBUG  [2024-02-14T20:56:24-03:00] Sending request to Incus                      etag= method=POST url="https://10.11.16.11:8443/1.0/instances?project=compartilhado"
DEBUG  [2024-02-14T20:56:24-03:00] 
	{
		"architecture": "x86_64",
		"config": {
			"image.architecture": "amd64",
			"image.description": "Ubuntu jammy amd64 (20231128_07:42)",
			"image.os": "Ubuntu",
			"image.release": "jammy",
			"image.serial": "20231128_07:42",
			"image.type": "squashfs",
			"image.variant": "cloud",
			"volatile.base_image": "81b028b2bb98a8c11d9c03d49ea67b432b7fbb5549997beb7af0f4d139125df7"
		},
		"devices": {
			"root": {
				"path": "/",
				"pool": "local",
				"type": "disk"
			}
		},
		"ephemeral": false,
		"profiles": [
			"default"
		],
		"stateful": false,
		"description": "",
		"name": "pedrojorge-PAPCMC2",
		"source": {
			"type": "copy",
			"certificate": "",
			"base-image": "81b028b2bb98a8c11d9c03d49ea67b432b7fbb5549997beb7af0f4d139125df7",
			"source": "pedrojorge-PAPCMC",
			"live": true,
			"project": "dedicado",
			"allow_inconsistent": false
		},
		"instance_type": "",
		"type": "container",
		"start": false
	} 
Error: Failed to get address of instance's member: Instance not found

Does it help if you pass --target and specify the server the current instance is on through it?

Either way, it’s a bug, can you file something at Issues · lxc/incus · GitHub and I’ll get it sorted for 0.6.

On the current configuration it’s not possible since the same machine is forbidden. To a valid machine I get the same error. Ideally, the new machine should be selected according to the scheduler.

pargo@bastion:~$ incus copy pedrojorge-PAPCMC pedrojorge-PAPCMC2 --target-project compartilhado --target dedicado01 
Error: Project isn't allowed to use this cluster member: "dedicado01"
pargo@bastion:~$ incus copy pedrojorge-PAPCMC pedrojorge-PAPCMC2 --target-project compartilhado --target compartilhado03
Error: Failed to get address of instance's member: Instance not found

Tested on projects that can use the same cluster members and adding --target to the same machine worked.

pargo@bastion:~$ incus list
+------+---------+------+------+-----------+-----------+-----------------+
| NAME |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |    LOCATION     |
+------+---------+------+------+-----------+-----------+-----------------+
| a1   | STOPPED |      |      | CONTAINER | 0         | compartilhado03 |
+------+---------+------+------+-----------+-----------+-----------------+
pargo@bastion:~$ incus copy a1 a2 --target-project test02 --target compartilhado03
pargo@bastion:~$ incus list --all-projects
+---------------+--------------------+---------+--------------------+------+-----------+-----------+-----------------+
|    PROJECT    |        NAME        |  STATE  |        IPV4        | IPV6 |   TYPE    | SNAPSHOTS |    LOCATION     |
+---------------+--------------------+---------+--------------------+------+-----------+-----------+-----------------+
...
+---------------+--------------------+---------+--------------------+------+-----------+-----------+-----------------+
| test01        | a1                 | STOPPED |                    |      | CONTAINER | 0         | compartilhado03 |
+---------------+--------------------+---------+--------------------+------+-----------+-----------+-----------------+
| test02        | a2                 | STOPPED |                    |      | CONTAINER | 0         | compartilhado03 |
+---------------+--------------------+---------+--------------------+------+-----------+-----------+-----------------+

Issue was posted here. Incus copy doesn't work to move instances between projects in a cluster · Issue #494 · lxc/incus · GitHub

Tested on incus 0.6 and there are still issues.

Copying to a different project gave me the following.

pargo@bastion:~$ incus copy victor-teste victor-teste2 --target-project aux
Error: Error transferring instance data: Failed migration on target: Error from migration control source: Failed migration on source: migration dump failed
(00.002433) Error (criu/namespaces.c:460): Can't dump nested uts namespace for 10384
(00.002435) Error (criu/namespaces.c:721): Can't make utsns id
(00.003832) Error (criu/util.c:642): exited, status=1
(00.004623) Error (criu/util.c:642): exited, status=1
(00.004677) Error (criu/cr-dump.c:2098): Dumping FAILED.

Adding --target to the same cluster member still works fine.

pargo@bastion:~$ incus list
+--------------+---------+---------------------+------+-----------+-----------+-----------------+
|     NAME     |  STATE  |        IPV4         | IPV6 |   TYPE    | SNAPSHOTS |    LOCATION     |
+--------------+---------+---------------------+------+-----------+-----------+-----------------+
| victor-teste | RUNNING | 10.11.21.121 (eth0) |      | CONTAINER | 0         | compartilhado04 |
+--------------+---------+---------------------+------+-----------+-----------+-----------------+
pargo@bastion:~$ incus copy victor-teste victor-teste2 --target-project aux --target compartilhado04
pargo@bastion:~$

Move also doesn´t work and oddly enough, it doesn’t work with --target.

pargo@bastion:~$ incus move victor-teste victor-teste3 --target-project aux
Error: Migration operation failure: snapshot dump failed
(00.102463) Error (criu/namespaces.c:460): Can't dump nested uts namespace for 10384
(00.102465) Error (criu/namespaces.c:721): Can't make utsns id
(00.103477) Error (criu/util.c:642): exited, status=1
(00.104356) Error (criu/util.c:642): exited, status=1
(00.104402) Error (criu/cr-dump.c:2098): Dumping FAILED.
pargo@bastion:~$ incus move victor-teste victor-teste3 --target-project aux --target compartilhado04
Error: The --target-project flag can't be used with --target

Use --stateless or stop your instance before copying. Otherwise Incus will attempt a container live migration which only rarely works out (it’s something I’d love to see improve though).

O wow! Huge mistake I made on this one. Sorry!

Btw, it’s working well now.

On this subject, to improve the user experience, do you think it would make sense to have an incus configuration to set a migration policy for the cluster which would be used as a standard? I was looking for this now and only found it for instances.

EDIT: it should be noted I stopped the instance and it moved correctly. Using --stateless I got an error asking me to stop the instance.

What do you mean by migration policy there?

You mentioned before to add --stateless. I was looking for a config key which would consider all moves to be stateless, since it’s not an issue for me. It would also help students using the cluster to not have the same issue I did of getting an error and not figuring out they forgot to stop the instance.

I actually thought such a configuration key existed. Since I didn’t find it, I just wondered if it would be a good thing to add it.

So one idea would be to make containers also respect migration.stateful, which would then require that flag be set to true for any kind of live migration to be possible.

It wouldn’t really change the fact that it’d fail without --stateless but it would at least not actually attempt a live migration and instead give you a clear error.

1 Like

Not sure you noticed, since I edited a previous comment for others reading this thread to understand, but using --stateless didn’t work on running containers. I’m not sure of it should work or not, but the error was very clear and not like the previous one which left me confused. It just told me to stop the instance before the copy.

Ah, that’s interesting, I’ll need to check why that is. In general I’d expect a stateless copy to be possible with the container left running, but there may be special considerations on some filesystems.

Oh, unless the error only happens on move? That part would probably make sense, though it’s then odd to have the --stateless option even be valid for incus move given that a move really either needs to be live or needs the instance be stopped beforehand.