Unable to do Remote API authentication

LXD
1

Trying to follow this documentation and this youtube tutorial to get Remote API Authentication but it doesn’t work.

As per the youtube tutorial I ran

ubuntu@arm1:~$ lxc config set core.https_address :8443                                                                                                                                                                                                        
ubuntu@arm1:~$ lxc config show                                                                                                                                                                                                                                
config:                                                                                                                                                                                                                                                       
  core.https_address: :8443                                                                                                                                                                                                                                   
ubuntu@arm1:~$ lxc config trust add                                                                                                                                                                                                                           
Please provide client name: victoitor-escritorio                                                                                                                                                                                                              
Client victoitor-escritorio certificate add token:                    
<token>

Then, on my home server, I get this error:

victoitor@victoitor-escritorio:~$ lxc remote add arm1 <token>
Error: Failed to add remote

On the remote server I opened port 8443 on iptables.

ubuntu@arm1:~$ sudo iptables -vnL
Chain INPUT (policy ACCEPT 68 packets, 17664 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 330K 1522M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
   18  1004 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8443 /* Victor: allow lxd communication */
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
 2024  186K ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:123
  620 34744 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22

Chain FORWARD (policy ACCEPT 10553 packets, 37M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 75836 packets, 44M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 495K 2765M InstanceServices  all  --  *      *       0.0.0.0/0            169.254.0.0/16      

Chain InstanceServices (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.0.2          owner UID match 0 tcp dpt:3260 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
 488K 2765M ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.2.0/24       owner UID match 0 tcp dpt:3260 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.4.0/24       owner UID match 0 tcp dpt:3260 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.5.0/24       owner UID match 0 tcp dpt:3260 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.0.2          tcp dpt:80 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
 1224  129K ACCEPT     udp  --  *      *       0.0.0.0/0            169.254.169.254      udp dpt:53 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.169.254      tcp dpt:53 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.0.3          owner UID match 0 tcp dpt:80 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.0.4          tcp dpt:80 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
 5620  398K ACCEPT     tcp  --  *      *       0.0.0.0/0            169.254.169.254      tcp dpt:80 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    1   304 ACCEPT     udp  --  *      *       0.0.0.0/0            169.254.169.254      udp dpt:67 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            169.254.169.254      udp dpt:69 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
   28  2128 ACCEPT     udp  --  *      *       0.0.0.0/0            169.254.169.254      udp dpt:123 /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
    0     0 REJECT     tcp  --  *      *       0.0.0.0/0            169.254.0.0/16       tcp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with tcp-reset
    0     0 REJECT     udp  --  *      *       0.0.0.0/0            169.254.0.0/16       udp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with icmp-port-unreachable

Also opened these port 8443 on Oracle Cloud, which I’m running this instance.

What could be the issue here?

2

What I suspect is going on here is that your instance is behind NAT, so the addresses that are included in the token don’t actually line up and aren’t accessible from your client.

You can check that by doing:

echo TOKEN | base64 -d | jq .

You’ll need to install jq. This should show you the content of the token which is made of a list of addresses that the client will attempt to connect to, a secret string and the certificate fingerprint of the server.

In your case, you may need to do:

lxc remote add my-server IP-ADDRESS

Then when prompted for the password, use your token as the password.

3

@ru-fu this is something we ought to document as it will be a somewhat common issue with public cloud environments.

4

Will check when I get home. But I completely agree on the documentation part. After watching the video, I checked the documentation and there was nothing there on this.

Can this IP which is passed by the token be set in LXD? Is it the one which is asked when on lxd init?

I also miss some place which explains how to change everything which is set on lxd init. I tend to run that quite a few times to try different parameters.

5

lxd init doesn’t really do anything special.

All it does can be done directly through lxc config set, lxc profile set, lxc storage or lxc network depending on what part you want to change.

LXD can be configured to listen on a particular IP address, but that address must be directly available on the server. With public clouds, that’s rarely the case. Instead you get a local address and an external public address which is mapped to it. LXD doesn’t have any visibility as to that external address.

We could theoretically introduce a core.https_address.external or something along those lines to hold the external address and then put it in the array of addresses to try.

It gets a bit tricky because when the token is consumed by another instance within your cloud VPC, the private address should be preferred, but when connecting from an external machine, the public address would be better.

That’s why it may make more sense to specify the public address at the time of lxc remote add. If the server isn’t directly reachable through one of its addresses, then you can override it to another address. The instructions I gave above should achieve that, but I think we may be able to do something better by having lxc remote add TOKEN prompt for an alternate address on failure.

1 Like
6

@monstermunchkin could you send a quick PR which adds that?
Basically in the lxc remote add TOKEN case, if all servers listen in the token fail, then ask the user whether they want to specify an alternate server address, then try to add that one using the info from the token.

2 Likes
7

Another idea that may be useful is to add external ip discovery using something like STUN or even just a call to a special endpoint on linuxcontainers.org on start up to get its external ip to add into the remote token. Could be useful for other things too.

8

Maybe make the list of addresses be checked in order and place the external address last.

10

I marked a solution that worked, but even then, I thought the flow was slightly odd.

It came out like this:

victoitor@victoitor-escritorio:~$ lxc remote add arm1 <public ip>
Certificate fingerprint: <fingerprint>
ok (y/n/[fingerprint])? <token>
Error: Please type 'y', 'n' or the fingerprint:
victoitor@victoitor-escritorio:~$ lxc remote add arm1 <public ip>
Certificate fingerprint: <fingerprint>
ok (y/n/[fingerprint])? y
Admin password for arm1: 
Error: not authorized
victoitor@victoitor-escritorio:~$ lxc remote add arm1 <public ip>
Admin password for arm1: 
Client certificate now trusted by server: arm1

So I have no idea why it asks for (y/n/[fingerprint]) the first time. I thought the token was the fingerprint it was asking for but it seemed not to be. I read the explanation again and answered y and then placed the fingerprint when asked Admin password for arm1:. The issue is that the password field is blanked so I thought the paste command didn’t work so I pasted again. Only on the third time I noticed this and pasted only once.

Placing a token on the password field seems like it’s not the correct place to put it.

It worked for me, but in the end it did not seems like the questions were well thought out from a user experience view.

11

The fingerprint question was asking you to verify you were connected to the lxd server you thought you wanted by showing you the remote server’s certificate fingerprint.

12

How can one start a cluster, join a cluster or decide to leave a cluster and delete everything cluster related without being through lxd init?

This is the one that intrigues me the most since I have no idea how to do it without being through lxd init.

13

lxd init is just for initialisation, so you cannot remove members from a cluster using it.

You can enable clustering for the first member using lxc cluster enable and managed member join tokens using lxc cluster add.

I believe you always have to join an existing cluster using lxd init though.

14

And what about shutting down a cluster and all of its resources into segregated LXD nodes maintaining their containers?

Can a Ubuntu fan be created without being through lxd init as well?

15

There is lxc cluster remove for that. Once a member has been removed it cannot be rejoined without being totally wiped and started afresh.