Unique Situation I can't seem to find anywhere involving adding a server to my cluster

Hi I have been attempting to install this OS. Ran into a unique problem

Followed the steps as instructed.

Here are the logs/steps I did

C:\Users\redacted>incus cluster join server1: server2:
What IP address or DNS name should be used to reach this server? [default=]:
What member name should be used to identify this server in the cluster? [default=69c7986d-c8a6-b914-aed2-2cf05dd4cc4e]:server2
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose “source” property for storage pool “local”: local/incus
Choose “zfs.pool_name” property for storage pool “local”: local/incus
Error: Failed to join cluster: Failed to setup cluster trust: Failed to add server cert to cluster: Post “https://https//:8443:8443/1.0/certificates”: lookup https on 127.0.0.53:53: server misbehaving"

I noticed that https:// and :8443 shows up two times rather once like most installations do. Anyway I can fix this.

Server 1 is the bootstrapper which was downloaded with default configs ticked
Server 2 is the server im using to attempt to connect to the boot strapper which has its defaults not ticked.

I did use SEED_DATA METHOD but only for reinstallation and network related things since there was some conflicts I ran into during my process

config:
cluster.https_address: 10.0.0.161:8443
core.https_address: 10.0.0.161:8443
storage.backups_volume: local/backups
storage.images_volume: local/images
storage.logs_volume: local/logs
api_extensions:

  • storage_zfs_remove_snapshots
  • container_host_shutdown_timeout
  • container_stop_priority
  • container_syscall_filtering
  • auth_pki
  • container_last_used_at
  • etag
  • patch
  • usb_devices
  • https_allowed_credentials
  • image_compression_algorithm
  • directory_manipulation
  • container_cpu_time
  • storage_zfs_use_refquota
  • storage_lvm_mount_options
  • network
  • profile_usedby
  • container_push
  • container_exec_recording
  • certificate_update
  • container_exec_signal_handling
  • gpu_devices
  • container_image_properties
  • migration_progress
  • id_map
  • network_firewall_filtering
  • network_routes
  • storage
  • file_delete
  • file_append
  • network_dhcp_expiry
  • storage_lvm_vg_rename
  • storage_lvm_thinpool_rename
  • network_vlan
  • image_create_aliases
  • container_stateless_copy
  • container_only_migration
  • storage_zfs_clone_copy
  • unix_device_rename
  • storage_lvm_use_thinpool
  • storage_rsync_bwlimit
  • network_vxlan_interface
  • storage_btrfs_mount_options
  • entity_description
  • image_force_refresh
  • storage_lvm_lv_resizing
  • id_map_base
  • file_symlinks
  • container_push_target
  • network_vlan_physical
  • storage_images_delete
  • container_edit_metadata
  • container_snapshot_stateful_migration
  • storage_driver_ceph
  • storage_ceph_user_name
  • resource_limits
  • storage_volatile_initial_source
  • storage_ceph_force_osd_reuse
  • storage_block_filesystem_btrfs
  • resources
  • kernel_limits
  • storage_api_volume_rename
  • network_sriov
  • console
  • restrict_dev_incus
  • migration_pre_copy
  • infiniband
  • dev_incus_events
  • proxy
  • network_dhcp_gateway
  • file_get_symlink
  • network_leases
  • unix_device_hotplug
  • storage_api_local_volume_handling
  • operation_description
  • clustering
  • event_lifecycle
  • storage_api_remote_volume_handling
  • nvidia_runtime
  • container_mount_propagation
  • container_backup
  • dev_incus_images
  • container_local_cross_pool_handling
  • proxy_unix
  • proxy_udp
  • clustering_join
  • proxy_tcp_udp_multi_port_handling
  • network_state
  • proxy_unix_dac_properties
  • container_protection_delete
  • unix_priv_drop
  • pprof_http
  • proxy_haproxy_protocol
  • network_hwaddr
  • proxy_nat
  • network_nat_order
  • container_full
  • backup_compression
  • nvidia_runtime_config
  • storage_api_volume_snapshots
  • storage_unmapped
  • projects
  • network_vxlan_ttl
  • container_incremental_copy
  • usb_optional_vendorid
  • snapshot_scheduling
  • snapshot_schedule_aliases
  • container_copy_project
  • clustering_server_address
  • clustering_image_replication
  • container_protection_shift
  • snapshot_expiry
  • container_backup_override_pool
  • snapshot_expiry_creation
  • network_leases_location
  • resources_cpu_socket
  • resources_gpu
  • resources_numa
  • kernel_features
  • id_map_current
  • event_location
  • storage_api_remote_volume_snapshots
  • network_nat_address
  • container_nic_routes
  • cluster_internal_copy
  • seccomp_notify
  • lxc_features
  • container_nic_ipvlan
  • network_vlan_sriov
  • storage_cephfs
  • container_nic_ipfilter
  • resources_v2
  • container_exec_user_group_cwd
  • container_syscall_intercept
  • container_disk_shift
  • storage_shifted
  • resources_infiniband
  • daemon_storage
  • instances
  • image_types
  • resources_disk_sata
  • clustering_roles
  • images_expiry
  • resources_network_firmware
  • backup_compression_algorithm
  • ceph_data_pool_name
  • container_syscall_intercept_mount
  • compression_squashfs
  • container_raw_mount
  • container_nic_routed
  • container_syscall_intercept_mount_fuse
  • container_disk_ceph
  • virtual-machines
  • image_profiles
  • clustering_architecture
  • resources_disk_id
  • storage_lvm_stripes
  • vm_boot_priority
  • unix_hotplug_devices
  • api_filtering
  • instance_nic_network
  • clustering_sizing
  • firewall_driver
  • projects_limits
  • container_syscall_intercept_hugetlbfs
  • limits_hugepages
  • container_nic_routed_gateway
  • projects_restrictions
  • custom_volume_snapshot_expiry
  • volume_snapshot_scheduling
  • trust_ca_certificates
  • snapshot_disk_usage
  • clustering_edit_roles
  • container_nic_routed_host_address
  • container_nic_ipvlan_gateway
  • resources_usb_pci
  • resources_cpu_threads_numa
  • resources_cpu_core_die
  • api_os
  • container_nic_routed_host_table
  • container_nic_ipvlan_host_table
  • container_nic_ipvlan_mode
  • resources_system
  • images_push_relay
  • network_dns_search
  • container_nic_routed_limits
  • instance_nic_bridged_vlan
  • network_state_bond_bridge
  • usedby_consistency
  • custom_block_volumes
  • clustering_failure_domains
  • resources_gpu_mdev
  • console_vga_type
  • projects_limits_disk
  • network_type_macvlan
  • network_type_sriov
  • container_syscall_intercept_bpf_devices
  • network_type_ovn
  • projects_networks
  • projects_networks_restricted_uplinks
  • custom_volume_backup
  • backup_override_name
  • storage_rsync_compression
  • network_type_physical
  • network_ovn_external_subnets
  • network_ovn_nat
  • network_ovn_external_routes_remove
  • tpm_device_type
  • storage_zfs_clone_copy_rebase
  • gpu_mdev
  • resources_pci_iommu
  • resources_network_usb
  • resources_disk_address
  • network_physical_ovn_ingress_mode
  • network_ovn_dhcp
  • network_physical_routes_anycast
  • projects_limits_instances
  • network_state_vlan
  • instance_nic_bridged_port_isolation
  • instance_bulk_state_change
  • network_gvrp
  • instance_pool_move
  • gpu_sriov
  • pci_device_type
  • storage_volume_state
  • network_acl
  • migration_stateful
  • disk_state_quota
  • storage_ceph_features
  • projects_compression
  • projects_images_remote_cache_expiry
  • certificate_project
  • network_ovn_acl
  • projects_images_auto_update
  • projects_restricted_cluster_target
  • images_default_architecture
  • network_ovn_acl_defaults
  • gpu_mig
  • project_usage
  • network_bridge_acl
  • warnings
  • projects_restricted_backups_and_snapshots
  • clustering_join_token
  • clustering_description
  • server_trusted_proxy
  • clustering_update_cert
  • storage_api_project
  • server_instance_driver_operational
  • server_supported_storage_drivers
  • event_lifecycle_requestor_address
  • resources_gpu_usb
  • clustering_evacuation
  • network_ovn_nat_address
  • network_bgp
  • network_forward
  • custom_volume_refresh
  • network_counters_errors_dropped
  • metrics
  • image_source_project
  • clustering_config
  • network_peer
  • linux_sysctl
  • network_dns
  • ovn_nic_acceleration
  • certificate_self_renewal
  • instance_project_move
  • storage_volume_project_move
  • cloud_init
  • network_dns_nat
  • database_leader
  • instance_all_projects
  • clustering_groups
  • ceph_rbd_du
  • instance_get_full
  • qemu_metrics
  • gpu_mig_uuid
  • event_project
  • clustering_evacuation_live
  • instance_allow_inconsistent_copy
  • network_state_ovn
  • storage_volume_api_filtering
  • image_restrictions
  • storage_zfs_export
  • network_dns_records
  • storage_zfs_reserve_space
  • network_acl_log
  • storage_zfs_blocksize
  • metrics_cpu_seconds
  • instance_snapshot_never
  • certificate_token
  • instance_nic_routed_neighbor_probe
  • event_hub
  • agent_nic_config
  • projects_restricted_intercept
  • metrics_authentication
  • images_target_project
  • images_all_projects
  • cluster_migration_inconsistent_copy
  • cluster_ovn_chassis
  • container_syscall_intercept_sched_setscheduler
  • storage_lvm_thinpool_metadata_size
  • storage_volume_state_total
  • instance_file_head
  • instances_nic_host_name
  • image_copy_profile
  • container_syscall_intercept_sysinfo
  • clustering_evacuation_mode
  • resources_pci_vpd
  • qemu_raw_conf
  • storage_cephfs_fscache
  • network_load_balancer
  • vsock_api
  • instance_ready_state
  • network_bgp_holdtime
  • storage_volumes_all_projects
  • metrics_memory_oom_total
  • storage_buckets
  • storage_buckets_create_credentials
  • metrics_cpu_effective_total
  • projects_networks_restricted_access
  • storage_buckets_local
  • loki
  • acme
  • internal_metrics
  • cluster_join_token_expiry
  • remote_token_expiry
  • init_preseed
  • storage_volumes_created_at
  • cpu_hotplug
  • projects_networks_zones
  • network_txqueuelen
  • cluster_member_state
  • instances_placement_scriptlet
  • storage_pool_source_wipe
  • zfs_block_mode
  • instance_generation_id
  • disk_io_cache
  • amd_sev
  • storage_pool_loop_resize
  • migration_vm_live
  • ovn_nic_nesting
  • oidc
  • network_ovn_l3only
  • ovn_nic_acceleration_vdpa
  • cluster_healing
  • instances_state_total
  • auth_user
  • security_csm
  • instances_rebuild
  • numa_cpu_placement
  • custom_volume_iso
  • network_allocations
  • zfs_delegate
  • storage_api_remote_volume_snapshot_copy
  • operations_get_query_all_projects
  • metadata_configuration
  • syslog_socket
  • event_lifecycle_name_and_project
  • instances_nic_limits_priority
  • disk_initial_volume_configuration
  • operation_wait
  • image_restriction_privileged
  • cluster_internal_custom_volume_copy
  • disk_io_bus
  • storage_cephfs_create_missing
  • instance_move_config
  • ovn_ssl_config
  • certificate_description
  • disk_io_bus_virtio_blk
  • loki_config_instance
  • instance_create_start
  • clustering_evacuation_stop_options
  • boot_host_shutdown_action
  • agent_config_drive
  • network_state_ovn_lr
  • image_template_permissions
  • storage_bucket_backup
  • storage_lvm_cluster
  • shared_custom_block_volumes
  • auth_tls_jwt
  • oidc_claim
  • device_usb_serial
  • numa_cpu_balanced
  • image_restriction_nesting
  • network_integrations
  • instance_memory_swap_bytes
  • network_bridge_external_create
  • network_zones_all_projects
  • storage_zfs_vdev
  • container_migration_stateful
  • profiles_all_projects
  • instances_scriptlet_get_instances
  • instances_scriptlet_get_cluster_members
  • instances_scriptlet_get_project
  • network_acl_stateless
  • instance_state_started_at
  • networks_all_projects
  • network_acls_all_projects
  • storage_buckets_all_projects
  • resources_load
  • instance_access
  • project_access
  • projects_force_delete
  • resources_cpu_flags
  • disk_io_bus_cache_filesystem
  • instance_oci
  • clustering_groups_config
  • instances_lxcfs_per_instance
  • clustering_groups_vm_cpu_definition
  • disk_volume_subpath
  • projects_limits_disk_pool
  • network_ovn_isolated
  • qemu_raw_qmp
  • network_load_balancer_health_check
  • oidc_scopes
  • network_integrations_peer_name
  • qemu_scriptlet
  • instance_auto_restart
  • storage_lvm_metadatasize
  • ovn_nic_promiscuous
  • ovn_nic_ip_address_none
  • instances_state_os_info
  • network_load_balancer_state
  • instance_nic_macvlan_mode
  • storage_lvm_cluster_create
  • network_ovn_external_interfaces
  • instances_scriptlet_get_instances_count
  • cluster_rebalance
  • custom_volume_refresh_exclude_older_snapshots
  • storage_initial_owner
  • storage_live_migration
  • instance_console_screenshot
  • image_import_alias
  • authorization_scriptlet
  • console_force
  • network_ovn_state_addresses
  • network_bridge_acl_devices
  • instance_debug_memory
  • init_preseed_storage_volumes
  • init_preseed_profile_project
  • instance_nic_routed_host_address
  • instance_smbios11
  • api_filtering_extended
  • acme_dns01
  • security_iommu
  • network_ipv4_dhcp_routes
  • network_state_ovn_ls
  • network_dns_nameservers
  • acme_http01_port
  • network_ovn_ipv4_dhcp_expiry
  • instance_state_cpu_time
  • network_io_bus
  • disk_io_bus_usb
  • storage_driver_linstor
  • instance_oci_entrypoint
  • network_address_set
  • server_logging
  • network_forward_snat
  • memory_hotplug
  • instance_nic_routed_host_tables
  • instance_publish_split
  • init_preseed_certificates
  • custom_volume_sftp
  • network_ovn_external_nic_address
  • network_physical_gateway_hwaddr
  • backup_s3_upload
  • snapshot_manual_expiry
  • resources_cpu_address_sizes
  • disk_attached
  • limits_memory_hotplug
  • disk_wwn
  • server_logging_webhook
  • storage_driver_truenas
  • container_disk_tmpfs
  • instance_limits_oom
  • backup_override_config
  • network_ovn_tunnels
  • init_preseed_cluster_groups
  • usb_attached
  • backup_iso
  • instance_systemd_credentials
  • cluster_group_usedby
  • bpf_token_delegation
  • file_storage_volume
  • network_hwaddr_pattern
  • storage_volume_full
  • storage_bucket_full
  • device_pci_firmware
  • resources_serial
  • ovn_nic_limits
  • storage_lvmcluster_qcow2
  • oidc_allowed_subnets
  • file_delete_force
  • nic_sriov_select_ext
  • network_zones_dns_contact
  • nic_attached_connected
  • nic_sriov_security_trusted
  • direct_backup
  • instance_snapshot_disk_only_restore
  • unix_hotplug_pci
  • cluster_evacuating_restoring
  • projects_restricted_image_servers
  • storage_lvmcluster_size
  • authorization_scriptlet_cert
  • lvmcluster_remove_snapshots
  • daemon_storage_logs
  • instances_debug_repair
  • network_io_bus_ovn
  • dependent
  • metrics_project_resources
  • storage_volume_nbd
  • projects_restricted_storage_pool_access
  • server_shutdown_action
  • instances_placement_scriptlet_rebalance
    api_status: stable
    api_version: “1.0”
    auth: trusted
    public: false
    auth_methods:
  • tls
    auth_user_name:
    auth_user_method: tls
    environment:
    addresses:
    • 10.0.0.161:8443
      architectures:
    • x86_64
    • i686
      certificate: |
      -----BEGIN CERTIFICATE-----
      -----END CERTIFICATE-----
      certificate_fingerprint: 1ae7b70a161855aa03636fe033845431dfd45ea508fc93ca67c6da547100f4eb
      driver: lxc | qemu
      driver_version: 7.0.0 | 11.0.0
      firewall: nftables
      kernel: Linux
      kernel_architecture: x86_64
      kernel_features: {}
      kernel_version: 7.0.3-zabbly+
      lxc_features: {}
      os_name: IncusOS
      os_version: “202605061755”
      project: default
      server: incus
      server_clustered: true
      server_event_mode: full-mesh
      server_name: server1
      server_pid: 2635
      server_version: 7.0.0
      storage: zfs
      storage_version: 2.4.1-1
      storage_supported_drivers:
    • name: dir
      version: “1”
      remote: false
    • name: lvm
      version: 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) / 4.50.0
      remote: false
    • name: zfs
      version: 2.4.1-1
      remote: false
    • name: btrfs
      version: “6.14”
      remote: false
    • name: lvmcluster
      version: 2.03.31(2) (2025-02-27) / 1.02.205 (2025-02-27) / 4.50.0
      remote: true
1 Like

not sure if you put this, but did you leave the ip address blank ? do you have multiple networks you are connecting with ? are you connecting on the right network (i.e. vlans,etc.)?

What IP address or DNS name should be used to reach this server? [default=]:

I connected with the default ip i removed the ip while making the post

Even then. I attempted to connect by inserting the ip manually

Everything is ran locally too so I know its not a network issue. Syntax related for sure

Error: Failed to join cluster: Failed to setup cluster trust: Failed to add server cert to cluster: Post “https://https//<IP>:8443:8443/1.0/certificates”: lookup https on 127.0.0.53:53: server misbehaving"

This where I think its causing problems

If its something that can only be fixed with a reinstall. I guess I will reinstall

My gut feeling is that there may be something weird going on with the Windows client.
Any chance you can try the same operation from a Linux system (possibly just WSL on Windows)?

1 Like

attempted with debian wsl and a reinstall of the OS too both on the cluster bootstrap server(my thinkpad) and node server. Same problem Persists

Note: I do not have a TPM CHIP on my thinkpad t460(my-cluster) so TPM is disabled. then again i don’t think that would cause problems. However on my node there is tpm

incus cluster join my-cluster: server2:
What IP address or DNS name should be used to reach this server? [default=10.0.0.141]:
What member name should be used to identify this server in the cluster? [default=69c7986d-c8a6-b914-aed2-2cf05dd4cc4e]: server2
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose “source” property for storage pool “local”: local/incus
Choose “zfs.pool_name” property for storage pool “local”: local/incus
Error: Failed to join cluster: Failed to setup cluster trust: Failed to add server cert to cluster: Post “https://https//10.0.0.161:8443:8443/1.0/certificates”: lookup https on 127.0.0.53:53: server misbehaving

Cluster seems to be working fine however just can’t join it with server2

If there is possible fix for this which I assume might not be the case due to me using a degraded version. I can survive without clustering. will be tedious but can’t really complain

Following. I am also seeing this issue.
❯ incus cluster join Cluster: IOS4:
What IP address or DNS name should be used to reach this server? [default=10.1.1.166]: 10.1.0.30 #Note: Trying to cluster over Wireguard
What member name should be used to identify this server in the cluster? [default=4c4c4544-0038-5310-8048-c2c04f395a32]: IncusOS-Backup
All existing data is lost when joining a cluster, continue? (yes/no) [default=no] yes
Choose “parent” property for network “eno1”: eno1
Choose “source” property for storage pool “local”: local/incus
Choose “zfs.pool_name” property for storage pool “local”: local/incus
Error: Failed to join cluster: Failed to setup cluster trust: Failed to add server cert to cluster: Post “``https://https//10.1.1.12:8443:8443/1.0/certificates``”: lookup https on 127.0.0.53:53: server misbehaving

2 Likes

Relying on swtpm to run IncusOS shouldn’t affect the ability to create a cluster. If it does, that’s a bug we’ll need to fix.

Problem seems to be we’re setting config.Cluster.ClusterAddress to a full URL including https:// here:

But I suspect its meant to only be a HOST:PORT

Later in the code path its run through internalUtil.CanonicalNetworkAddress which is expecting only HOST:PORT and isn’t handling the https:// prefix correctly.

I see it being modified like this:

  • https://cluster1.ffc.lab:8443 from connectInfo.URL
  • https://cluster1.ffc.lab:8443:8443 after CanonicalNetworkAddress
  • https://https//cluster1.ffc.lab:8443:8443 by the time it throws the error

That makes a lot more sense. I am for now just running it without clustering. Honestly just going to stick without it knowing how long that would take to correct. The Dev is one busy guy

I submitted a PR for the https://https//<IP>:8443:8443.

But I think there is still an issue where the incus CLI is not detecting a default IP address on your system, so you would need to specify that when prompted (note default is not showing anything):

  • What IP address or DNS name should be used to reach this server? [default=]:

For example, running on a server here I get:

  • What IP address or DNS name should be used to reach this server? [default=192.168.1.160]:

Is the incus info you provided from server1?
If you run it against server2 whats listed under addresses:

1 Like

i typed the ip’s all the time. i reinstalled the os like 10+ times and still ran into the same problem. All the certs were set by me and I am confident its correct. I’ve never ran into this issue before and I’ve never seen anybody else personally. It’s like a unique situation cause I don’t add https or anything. It just started happening at random. I thought maybe my installation corrupted or my bios might’ve corrupted but every other os (System Rescue) worked as intended. Which I had to use system rescue 10 different times to wipe the drives clean via wipefs and zapping the drives.

I 100% thought it’d be something related to the windows version like Stefan mentioned before which is why I went ahead and installed WSL and did the same process again. Still ran into the same problem.

Not really necessary for my setup. Its more so of a preference thing rather a thing I need.