Lxc remote add fails with "error: not authorized"

Silentphantom62 · December 27, 2020, 5:36pm

LXC remote add fails with “error: not authorized”

Debug logs below, any help will be appreciated.

    useracc@my-lxd-host-1:~$ lxc remote add my-lxd-host-3my-lxd-host-3 <ip_address_host_3> --password=<password> --debug --accept-certificate
DBUG[07-26|11:11:23] Connecting to a remote LXD over HTTPs
DBUG[07-26|11:11:23] Sending request to LXD                   etag= method=GET url=https://<ip_address_host_3>:8443/1.0
DBUG[07-26|11:11:23] Connecting to a remote LXD over HTTPs
DBUG[07-26|11:11:23] Sending request to LXD                   etag= method=GET url=https://<ip_address_host_3>:8443/1.0
DBUG[07-26|11:11:23] Got response struct from LXD
DBUG[07-26|11:11:23]
	{
		"config": null,
		"api_extensions": [
			"storage_zfs_remove_snapshots",
			"container_host_shutdown_timeout",
			"container_stop_priority",
			"container_syscall_filtering",
			"auth_pki",
			"container_last_used_at",
			"etag",
			"patch",
			"usb_devices",
			"https_allowed_credentials",
			"image_compression_algorithm",
			"directory_manipulation",
			"container_cpu_time",
			"storage_zfs_use_refquota",
			"storage_lvm_mount_options",
			"network",
			"profile_usedby",
			"container_push",
			"container_exec_recording",
			"certificate_update",
			"container_exec_signal_handling",
			"gpu_devices",
			"container_image_properties",
			"migration_progress",
			"id_map",
			"network_firewall_filtering",
			"network_routes",
			"storage",
			"file_delete",
			"file_append",
			"network_dhcp_expiry",
			"storage_lvm_vg_rename",
			"storage_lvm_thinpool_rename",
			"network_vlan",
			"image_create_aliases",
			"container_stateless_copy",
			"container_only_migration",
			"storage_zfs_clone_copy",
			"unix_device_rename",
			"storage_lvm_use_thinpool",
			"storage_rsync_bwlimit",
			"network_vxlan_interface",
			"storage_btrfs_mount_options",
			"entity_description",
			"image_force_refresh",
			"storage_lvm_lv_resizing",
			"id_map_base",
			"file_symlinks",
			"container_push_target",
			"network_vlan_physical",
			"storage_images_delete",
			"container_edit_metadata",
			"container_snapshot_stateful_migration",
			"storage_driver_ceph",
			"storage_ceph_user_name",
			"resource_limits",
			"storage_volatile_initial_source",
			"storage_ceph_force_osd_reuse",
			"storage_block_filesystem_btrfs",
			"resources",
			"kernel_limits",
			"storage_api_volume_rename",
			"macaroon_authentication",
			"network_sriov",
			"console",
			"restrict_devlxd",
			"migration_pre_copy",
			"infiniband",
			"maas_network",
			"devlxd_events",
			"proxy",
			"network_dhcp_gateway",
			"file_get_symlink",
			"network_leases",
			"unix_device_hotplug",
			"storage_api_local_volume_handling",
			"operation_description",
			"clustering",
			"event_lifecycle",
			"storage_api_remote_volume_handling",
			"nvidia_runtime",
			"candid_authentication",
			"candid_config",
			"candid_config_key",
			"usb_optional_vendorid"
		],
		"api_status": "stable",
		"api_version": "1.0",
		"auth": "untrusted",
		"public": false,
		"auth_methods": [
			"tls"
		],
		"environment": {
			"addresses": null,
			"architectures": null,
			"certificate": "",
			"certificate_fingerprint": "",
			"driver": "",
			"driver_version": "",
			"kernel": "",
			"kernel_architecture": "",
			"kernel_version": "",
			"server": "",
			"server_pid": 0,
			"server_version": "",
			"storage": "",
			"storage_version": ""
		}
	}
DBUG[07-26|11:11:23] Sending request to LXD                   etag= method=GET url=https://<ip_address_host_3>:8443/1.0
DBUG[07-26|11:11:24] Got response struct from LXD
DBUG[07-26|11:11:24]
	{
		"config": null,
		"api_extensions": [
			"storage_zfs_remove_snapshots",
			"container_host_shutdown_timeout",
			"container_stop_priority",
			"container_syscall_filtering",
			"auth_pki",
			"container_last_used_at",
			"etag",
			"patch",
			"usb_devices",
			"https_allowed_credentials",
			"image_compression_algorithm",
			"directory_manipulation",
			"container_cpu_time",
			"storage_zfs_use_refquota",
			"storage_lvm_mount_options",
			"network",
			"profile_usedby",
			"container_push",
			"container_exec_recording",
			"certificate_update",
			"container_exec_signal_handling",
			"gpu_devices",
			"container_image_properties",
			"migration_progress",
			"id_map",
			"network_firewall_filtering",
			"network_routes",
			"storage",
			"file_delete",
			"file_append",
			"network_dhcp_expiry",
			"storage_lvm_vg_rename",
			"storage_lvm_thinpool_rename",
			"network_vlan",
			"image_create_aliases",
			"container_stateless_copy",
			"container_only_migration",
			"storage_zfs_clone_copy",
			"unix_device_rename",
			"storage_lvm_use_thinpool",
			"storage_rsync_bwlimit",
			"network_vxlan_interface",
			"storage_btrfs_mount_options",
			"entity_description",
			"image_force_refresh",
			"storage_lvm_lv_resizing",
			"id_map_base",
			"file_symlinks",
			"container_push_target",
			"network_vlan_physical",
			"storage_images_delete",
			"container_edit_metadata",
			"container_snapshot_stateful_migration",
			"storage_driver_ceph",
			"storage_ceph_user_name",
			"resource_limits",
			"storage_volatile_initial_source",
			"storage_ceph_force_osd_reuse",
			"storage_block_filesystem_btrfs",
			"resources",
			"kernel_limits",
			"storage_api_volume_rename",
			"macaroon_authentication",
			"network_sriov",
			"console",
			"restrict_devlxd",
			"migration_pre_copy",
			"infiniband",
			"maas_network",
			"devlxd_events",
			"proxy",
			"network_dhcp_gateway",
			"file_get_symlink",
			"network_leases",
			"unix_device_hotplug",
			"storage_api_local_volume_handling",
			"operation_description",
			"clustering",
			"event_lifecycle",
			"storage_api_remote_volume_handling",
			"nvidia_runtime",
			"candid_authentication",
			"candid_config",
			"candid_config_key",
			"usb_optional_vendorid"
		],
		"api_status": "stable",
		"api_version": "1.0",
		"auth": "untrusted",
		"public": false,
		"auth_methods": [
			"tls"
		],
		"environment": {
			"addresses": null,
			"architectures": null,
			"certificate": "",
			"certificate_fingerprint": "",
			"driver": "",
			"driver_version": "",
			"kernel": "",
			"kernel_architecture": "",
			"kernel_version": "",
			"server": "",
			"server_pid": 0,
			"server_version": "",
			"storage": "",
			"storage_version": ""
		}
	}
DBUG[07-26|11:11:24] Sending request to LXD                   etag= method=POST url=https://<ip_address_host_3>:8443/1.0/certificates
DBUG[07-26|11:11:24]
	{
		"name": "",
		"type": "client",
		"certificate": "",
		"password": "<password>"
	}
DBUG[07-26|11:11:24] Trying to remove /home/useracc/.config/lxc/servercerts/my-lxd-host-3.crt
error: not authorized

stgraber · July 26, 2019, 7:14pm

Wrong password?

Silentphantom62 · July 26, 2019, 7:35pm

I have verified the password and it’s correct.

Any other way to verify the password? @stgraber

stgraber · July 27, 2019, 4:01am

What’s in the server’s log?

If it’s a bad password issue, you should see Rejecting request from untrusted client.

Pucky_wins · October 13, 2020, 3:24pm

Hi, I have the same issue here.

t=2020-10-13T17:19:25+0200 lvl=warn msg=“Rejecting request from untrusted client” ip=10.3.0.58:52334

I have reset the password using
lxc config set core.trust_password=abc123

I still can’t add the node. I have previously removed the node in question from the cluster so I don’t know if it could be blacklisted?

Pucky_wins · October 14, 2020, 8:53am

I got this going with the following command followed by a reload. I’m not sure if the equals sign was making problems or lack of a reload or just me pasting incorrectly.

lxc config set core.trust_password abc123
sudo systemctl reload snap.lxd.daemon.service

strk · December 25, 2020, 3:50pm

Thanks, lxc config set core.trust_password fixed this for me as well, but is there a way to read the trust password ? Becuase lxc config get core.trust_password always return true for me, no matter what I set the trust password to

turtle0x1 · December 25, 2020, 6:28pm

It would appear LXD (the API part) obscures the trust password once its been set so clients cant get it!

From memory you can only set a password so clients can add their own certificate, once the certificate is added you don’t use the trust_password for other API calls. It shouldn’t break existing clients to reset it - although it will break any scripts depending on the password to add the certs!

Im not a LXD dev, but did take a look over the code I may be wrong but im basing my comments on

github.com

lxc/lxd/blob/ee2375faaf91a2031db459b5c97b225711371e55/lxd/config/map.go#L90-L94


// Dump the current configuration held by this Map.
//
// Keys that match their default value will not be included in the dump. Also,
// if a Key has its Hidden attribute set to true, it will be rendered as
// "true", for obfuscating the actual value.

github.com

lxc/lxd/blob/ee2375faaf91a2031db459b5c97b225711371e55/lxd/config/map.go#L104-L108


				if key.Hidden {
					values[name] = true
				} else {
					values[name] = value
				}

This is defined in the “cluster” folder but I think its relevant

github.com

lxc/lxd/blob/ee2375faaf91a2031db459b5c97b225711371e55/lxd/cluster/config.go#L253


	"cluster.images_minimal_replica": {Type: config.Int64, Default: "3", Validator: imageMinimalReplicaValidator},
	"cluster.max_voters":             {Type: config.Int64, Default: "3", Validator: maxVotersValidator},
	"cluster.max_standby":            {Type: config.Int64, Default: "2", Validator: maxStandByValidator},
	"core.https_allowed_headers":     {},
	"core.https_allowed_methods":     {},
	"core.https_allowed_origin":      {},
	"core.https_allowed_credentials": {Type: config.Bool},
	"core.proxy_http":                {},
	"core.proxy_https":               {},
	"core.proxy_ignore_hosts":        {},
	"core.trust_password":            {Hidden: true, Setter: passwordSetter},
	"core.trust_ca_certificates":     {Type: config.Bool},
	"candid.api.key":                 {},
	"candid.api.url":                 {},
	"candid.domains":                 {},
	"candid.expiry":                  {Type: config.Int64, Default: "3600"},
	"images.auto_update_cached":      {Type: config.Bool, Default: "true"},
	"images.auto_update_interval":    {Type: config.Int64, Default: "6"},
	"images.compression_algorithm":   {Default: "gzip", Validator: validateCompression},
	"images.remote_cache_expiry":     {Type: config.Int64, Default: "10"},
	"maas.api.key":                   {},

vrubiolo · January 19, 2021, 2:50pm

Hi there,

Just wanted to send some thanks to @Silentphantom62 for the lxc remote add command w/ password and certificate validation included. I had not found it in the LXC advanced guide and reading this thread answered my question here.

I was trying to automate the lxc remote add command and stumbled upon the interactive prompts, then looking at using shell redirects before hitting a snag w/ the password prompt (it uses ReadPassword from the Go terminal library and this behaves differently in this case). Using --password allowed me to go through. Thanks!

toby63 · January 23, 2021, 2:31pm

@vrubiolo Just a tip: the Advanced Guide states [flags] in many cases, which refers to further flags being available.
You can usually look them up by typing the command without any instructions, for example:
lxc remote add
or with --help:
lxc remote add --help

Which will show you:

Flags:
      --accept-certificate   Accept certificate
     [...]
      --password             Remote admin password

vrubiolo · January 26, 2021, 2:28pm

Thanks @toby63 for the pointer about [flags], it is true one can get the info that way too.

In my case, I was interested in being able to search through the doc so having to run the command is less useful but this is always good to know

toby63 · January 28, 2021, 5:47pm

Well, that is probably a question of taste and priority.
I am the writer of the Advanced Guide (as it is now) and I was thinking about including further options as well, but there are some reasons that speak against it.
In my oppinion:

A guide should mostly contain the most important (and maybe most used) information.
It should also be as short as possible.
Options in commands might change, so I thought it is best, if people look it up directly.

Nonetheless we could:

Add a more visible note to take a look at the help pages of commands.
Add manpages online (so we could directly link to that and people can read it online)
Discuss changes case to case.

vrubiolo · February 1, 2021, 10:22pm

Thanks @toby63 for your reply.

I agree about the fact that command options change and indeed should not be present verbatim in the guides as this is a lot of work to maintain.

Maybe just a simple mention here about the fact that if the --password option was not specified, the user will be asked the store password?

This will make it clearer this is where the password is supposed to be entered+give a hint to users that it could also be passed down on the commandline.

What do you think?

toby63 · February 2, 2021, 9:08pm

It is mentioned already.

Regarding the --password flag:
To be completely honest I would not explicitely mention it (this is just my oppinion though), for the following reasons:

automating a password input is very insecure and should be avoided. If you look at tech news you will very often find security problems that are caused by people (unintentionally) making passwords available (e.g. in logs, files etc. that are publicly accessable)
the other usecase would only be that people prefer to enter it without a prompt. These people can look up the additional flags (see also below). I don’t see a problem in the prompt in general, because every terminal should support that.

Additonal Info:

The LXD team plans to add manpages that will be directly available on the website (see: Issue 471 on Github).
I (or someone else) will probably add links to those in the Advanced Guide etc., too.
I added a note about “command --help” to the Getting Started Guide.