Error: Failed to setup device mount: idmapping abilities are required but aren't supported on system

Sorry to reopen this thread, I experience this error with the latest incus 6.3 and kernel 6.8.0-38-generic.
Please note, this is on a stock Ubuntu 24.04 kernel, not 6.9.x!

fire@server:~$ incus version
Client version: 6.3
Server version: 6.3

The container BAK1 is supposed to start with a mounted USB device, but it fails with idmapping issues.
Before the start of BAK1 the USB is mounted on the host successfully.
(BAK1 starts normal when the USB drive is not mouted.)

incus create images:ubuntu/noble BAK1

incus config device add BAK1 md1 disk source=/storage0/BAK1/Opslag/Philips path=/storage0/Opslag/Philips shift=true
fire@server:~$ incus start BAK1
Error: Failed to setup device mount "md1": idmapping abilities are required but aren't supported on system
Try `incus info --show-log BAK1` for more info
fire@server:~$ incus info --show-log BAK1
Name: BAK1
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2024/07/16 19:16 CEST
Last Used: 2024/07/23 21:33 CEST

Log:

The log is empty, but the incus monitor gives some info:

fire@server:~$ incus monitor --pretty
DEBUG  [2024-07-23T22:16:00+02:00] Event listener server handler started         id=986fd701-d02e-4a21-8c78-0d34182e80f0 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-07-23T22:26:51+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=fire
DEBUG  [2024-07-23T22:26:51+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/BAK1 username=fire
DEBUG  [2024-07-23T22:26:51+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/events username=fire
DEBUG  [2024-07-23T22:26:51+02:00] Event listener server handler started         id=f27d5c04-be1f-4d7a-bf3e-639ed513b480 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-07-23T22:26:51+02:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/instances/BAK1/state username=fire
DEBUG  [2024-07-23T22:26:51+02:00] Started operation                             class=task description="Starting instance" operation=8cab8fc4-b218-46b8-b0ca-568165e8058f project=default
DEBUG  [2024-07-23T22:26:51+02:00] New operation                                 class=task description="Starting instance" operation=8cab8fc4-b218-46b8-b0ca-568165e8058f project=default
DEBUG  [2024-07-23T22:26:51+02:00] Start started                                 instance=BAK1 instanceType=container project=default stateful=false
INFO   [2024-07-23T22:26:51+02:00] ID: 8cab8fc4-b218-46b8-b0ca-568165e8058f, Class: task, Description: Starting instance  CreatedAt="2024-07-23 22:26:51.221005714 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/BAK1]]" Status=Pending StatusCode=Pending UpdatedAt="2024-07-23 22:26:51.221005714 +0200 CEST"
INFO   [2024-07-23T22:26:51+02:00] ID: 8cab8fc4-b218-46b8-b0ca-568165e8058f, Class: task, Description: Starting instance  CreatedAt="2024-07-23 22:26:51.221005714 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/BAK1]]" Status=Running StatusCode=Running UpdatedAt="2024-07-23 22:26:51.221005714 +0200 CEST"
INFO   [2024-07-23T22:26:51+02:00] Starting instance                             action=start created="2024-07-16 17:16:26.762513444 +0000 UTC" ephemeral=false instance=BAK1 instanceType=container project=default stateful=false used="2024-07-23 19:33:47.114172455 +0000 UTC"
DEBUG  [2024-07-23T22:26:51+02:00] Instance operation lock created               action=start instance=BAK1 project=default reusable=false
DEBUG  [2024-07-23T22:26:51+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/operations/8cab8fc4-b218-46b8-b0ca-568165e8058f username=fire
DEBUG  [2024-07-23T22:26:51+02:00] MountInstance started                         driver=btrfs instance=BAK1 pool=lxd-storage project=default
DEBUG  [2024-07-23T22:26:51+02:00] MountInstance finished                        driver=btrfs instance=BAK1 pool=lxd-storage project=default
DEBUG  [2024-07-23T22:26:51+02:00] Starting device                               device=eth0 instance=BAK1 instanceType=container project=default type=nic
DEBUG  [2024-07-23T22:26:51+02:00] Starting device                               device=root instance=BAK1 instanceType=container project=default type=disk
DEBUG  [2024-07-23T22:26:51+02:00] Starting device                               device=md0 instance=BAK1 instanceType=container project=default type=disk
DEBUG  [2024-07-23T22:26:51+02:00] Starting device                               device=md1 instance=BAK1 instanceType=container project=default type=disk
DEBUG  [2024-07-23T22:26:57+02:00] Stopping device                               device=md1 instance=BAK1 instanceType=container project=default type=disk
DEBUG  [2024-07-23T22:26:57+02:00] Stopping device                               device=md0 instance=BAK1 instanceType=container project=default type=disk
DEBUG  [2024-07-23T22:26:57+02:00] Stopping device                               device=eth0 instance=BAK1 instanceType=container project=default type=nic
DEBUG  [2024-07-23T22:26:57+02:00] Stopping device                               device=root instance=BAK1 instanceType=container project=default type=disk
DEBUG  [2024-07-23T22:26:57+02:00] UnmountInstance started                       driver=btrfs instance=BAK1 pool=lxd-storage project=default
DEBUG  [2024-07-23T22:26:57+02:00] UnmountInstance finished                      driver=btrfs instance=BAK1 pool=lxd-storage project=default
DEBUG  [2024-07-23T22:26:57+02:00] Instance operation lock finished              action=start err="Failed to setup device mount \"md1\": idmapping abilities are required but aren't supported on system" instance=BAK1 project=default reusable=false
DEBUG  [2024-07-23T22:26:57+02:00] Start finished                                instance=BAK1 instanceType=container project=default stateful=false
DEBUG  [2024-07-23T22:26:57+02:00] Failure for operation                         class=task description="Starting instance" err="Failed to setup device mount \"md1\": idmapping abilities are required but aren't supported on system" operation=8cab8fc4-b218-46b8-b0ca-568165e8058f project=default
INFO   [2024-07-23T22:26:57+02:00] ID: 8cab8fc4-b218-46b8-b0ca-568165e8058f, Class: task, Description: Starting instance  CreatedAt="2024-07-23 22:26:51.221005714 +0200 CEST" Err="Failed to setup device mount \"md1\": idmapping abilities are required but aren't supported on system" Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/BAK1]]" Status=Failure StatusCode=Failure UpdatedAt="2024-07-23 22:26:51.221005714 +0200 CEST"
DEBUG  [2024-07-23T22:26:57+02:00] Event listener server handler stopped         listener=f27d5c04-be1f-4d7a-bf3e-639ed513b480 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-07-23T22:27:31+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=fire
DEBUG  [2024-07-23T22:27:31+02:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/instances/BAK1?recursion=1" username=fire
DEBUG  [2024-07-23T22:27:31+02:00] GetInstanceUsage started                      driver=btrfs instance=BAK1 pool=lxd-storage project=default
DEBUG  [2024-07-23T22:27:31+02:00] GetInstanceUsage finished                     driver=btrfs instance=BAK1 pool=lxd-storage project=default
DEBUG  [2024-07-23T22:27:31+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/instances/BAK1/logs/lxc.log username=fire
INFO   [2024-07-23T22:27:31+02:00] Action: instance-log-retrieved, Source: /1.0/instances/BAK1/backups/lxc.log, Requestor: unix/fire (@) 

Details from BAK1:

architecture: x86_64
config:
  boot.autostart: "false"
  boot.autostart.delay: "5"
  boot.autostart.priority: "40"
  boot.stop.priority: "60"
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (20240715_07:42)
  image.os: Ubuntu
  image.release: noble
  image.requirements.cgroup: v2
  image.serial: "20240715_07:42"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 440c38e084228f3a2c2abe4e237268bbd188d6b5b5197d8809e7b5ced82d2d6d
  volatile.cloud-init.instance-id: 76745ea3-1217-4ea8-ac8d-597f4f7e02c1
  volatile.eth0.hwaddr: 00:16:3e:c0:3f:30
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: 2968cc9b-608b-487c-82cd-422974e8e0c8
  volatile.uuid.generation: 7148b9a5-9be1-4afe-ba31-74772cdc69c1
devices:
  md0:
    path: /storage0/NAS1
    readonly: "true"
    shift: "true"
    source: /storage0/BAK1/NAS1
    type: disk
  md1:
    path: /storage0/Opslag/Philips
    shift: "true"
    source: /storage0/BAK1/Opslag/Philips
    type: disk
  md2:
    path: /storage0/Opslag/Seagate
    shift: "true"
    source: /storage0/BAK1/Opslag/Seagate
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
created_at: 2024-07-16T17:16:26.762513444Z
name: BAK1
status: Stopped
status_code: 102
last_used_at: 2024-07-23T19:33:47.114172455Z
location: none
type: container
project: default

Details from incus are:

fire@server:~$ incus info
config:
  core.https_address: 192.168.0.20:9444
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- images_all_projects
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
- disk_io_bus_virtio_blk
- loki_config_instance
- instance_create_start
- clustering_evacuation_stop_options
- boot_host_shutdown_action
- agent_config_drive
- network_state_ovn_lr
- image_template_permissions
- storage_bucket_backup
- storage_lvm_cluster
- shared_custom_block_volumes
- auth_tls_jwt
- oidc_claim
- device_usb_serial
- numa_cpu_balanced
- image_restriction_nesting
- network_integrations
- instance_memory_swap_bytes
- network_bridge_external_create
- network_zones_all_projects
- storage_zfs_vdev
- container_migration_stateful
- profiles_all_projects
- instances_scriptlet_get_instances
- instances_scriptlet_get_cluster_members
- instances_scriptlet_get_project
- network_acl_stateless
- instance_state_started_at
- networks_all_projects
- network_acls_all_projects
- storage_buckets_all_projects
- resources_load
- instance_access
- project_access
- projects_force_delete
- resources_cpu_flags
- disk_io_bus_cache_filesystem
- instance_oci
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: fire
auth_user_method: unix
environment:
  addresses:
  - 192.168.0.20:9444
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----

    -----END CERTIFICATE-----
  certificate_fingerprint: 1234567890
  driver: lxc | qemu
  driver_version: 6.0.1 | 9.0.1
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.8.0-38-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "24.04"
  project: default
  server: incus
  server_clustered: false
  server_event_mode: full-mesh
  server_name: server
  server_pid: 960
  server_version: "6.3"
  storage: btrfs
  storage_version: 6.6.3
  storage_supported_drivers:
  - name: btrfs
    version: 6.6.3
    remote: false
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0
    remote: false
  - name: lvmcluster
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0
    remote: true