Pylxd 'not authorized' exception when accessing remote

yang · August 19, 2021, 4:28pm

I have a container setup to access the host server’s containers remotely, which is working without issues:

$ lxc remote list
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| host            | https://192.168.6.10:8443                | lxd           | tls         | NO     | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| local (current) | unix://                                  | lxd           | file access | NO     | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+

$ lxc list host:
+-----------------+---------+----------------------+------+-----------------+-----------+----------+
|      NAME       |  STATE  |         IPV4         | IPV6 |      TYPE       | SNAPSHOTS | LOCATION |
+-----------------+---------+----------------------+------+-----------------+-----------+----------+
| lxdui-container | RUNNING | 192.168.6.20 (eth1)  |      | CONTAINER       | 0         | node1    |
|                 |         | 192.168.1.113 (eth0) |      |                 |           |          |
+-----------------+---------+----------------------+------+-----------------+-----------+----------+
| test-container  | RUNNING | 10.20.30.250 (eth0)  |      | CONTAINER       | 0         | node2    |
+-----------------+---------+----------------------+------+-----------------+-----------+----------+
| test-vm         | STOPPED |                      |      | VIRTUAL-MACHINE | 0         | node1    |
+-----------------+---------+----------------------+------+-----------------+-----------+----------+

However it seems that the pylxd call to get all instances has stopped working. I have written a simple Python script to attempt to access the remote server’s instances:

from pylxd import Client

client = Client(endpoint='https://192.168.6.10:8443', verify=False)
client.instances.all()

Running the script gives me the following error which I hadn’t encountered previously:

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    print(client.instances.all())
  File "/usr/local/lib/python3.8/dist-packages/pylxd-2.2.11-py3.8.egg/pylxd/models/instance.py", line 252, in all
    response = client.api[cls._endpoint].get()
  File "/usr/local/lib/python3.8/dist-packages/pylxd-2.2.11-py3.8.egg/pylxd/client.py", line 159, in get
    self._assert_response(response,
  File "/usr/local/lib/python3.8/dist-packages/pylxd-2.2.11-py3.8.egg/pylxd/client.py", line 117, in _assert_response
    raise exceptions.LXDAPIException(response)
pylxd.exceptions.LXDAPIException: not authorized

Would this be due to changes in LXD’s API or pylxd itself? Any tips are appreciated.

turtle0x1 · August 19, 2021, 4:46pm

lxc remote list is not related to your ability to execute remote commands with libraries like pylxd .

remotes is a purely lxc (the http client) concept not LXD (as @stgraber once told me)

At a guess the verify command is related to verifying the remote LXD cert is signed by an appropriate authority.

You will need to provide a certificate to the pylxd library that is authorised on the host https://192.168.6.10:8443

yang · August 19, 2021, 4:53pm

That would make sense, thank you. I previously left the verify option as false since the container was isolated and it simplified the deployment process.

Would this be due to updates in LXD’s API? I am questioning since I had previsously worked with remote servers with pylxd without certificates successfully.

turtle0x1 · August 19, 2021, 4:54pm

You’re welcome.

You might have worked with them by using the local socket without a cert (pylxd might do it behind the scenes), but you haven’t worked with them over HTTP without a cert.

yang · August 19, 2021, 5:59pm

I previously used

client = Client(endpoint='https://192.168.6.10:8443', verify=False)

to skip the certificate verification using pylxd 2.2.11 and it has worked without issues on LXD 4.13 .

I’ll have to retest it using an older version of LXD to confirm. I am unsure about the inner workings of pylxd but since it worked before without the need to setup certificates I left it as is. I’ll report back with updates as soon as I finish testing

turtle0x1 · August 19, 2021, 7:18pm

yang:

I previously used
client = Client(endpoint='https://192.168.6.10:8443', verify=False)
to skip the certificate verification using pylxd 2.2.11 and it has worked without issues on LXD 4.13 .

verify has nothing todo with authenticating you. Could you image if LXD had 1 parameter to skip all authentication? Hacker news / Reddit would have a field day.

github.com

canonical/pylxd/blob/d9f3e8c141aa93227a08f475a95eda9d78f7b335/pylxd/client.py#L329-L332


      
          :param verify: (optional): Either a boolean, in which case it controls
              whether we verify the server's TLS certificate, or a string, in
              which case it must be a path to a CA bundle to use.
              Defaults to ``True``.

It appears looking at pylxd if you pass this configuration;

client = Client(verify=False)
// or
client = Client()

Then pylxd will try to use socket to connect, which may have adverse side effects i’m not aware of yet aware of - abusing the socket is not a solution, default to HTTP then work backwards.

github.com

canonical/pylxd/blob/d9f3e8c141aa93227a08f475a95eda9d78f7b335/pylxd/client.py#L353-L360


      
          else:
              if "LXD_DIR" in os.environ:
                  path = os.path.join(os.environ.get("LXD_DIR"), "unix.socket")
              elif os.path.exists("/var/snap/lxd/common/lxd/unix.socket"):
                  path = "/var/snap/lxd/common/lxd/unix.socket"
              else:
                  path = "/var/lib/lxd/unix.socket"
              endpoint = "http+unix://{}".format(parse.quote(path, safe=""))

yang · August 20, 2021, 9:27am

Skipping verification entirely does indeed not make sense

It seems that pylxd can no longer find the automatically generated client certificate and key .

Manually adding the paths solved the issue:

client = Client(endpoint='https://192.168.6.10:8443', cert=('/root/snap/lxd/common/config/client.crt', '/root/snap/lxd/common/config/client.key'), verify=False)

yang · August 23, 2021, 9:53am

Small update.

Previous versions of LXD seem to create the cert files at :
$HOME/snap/lxd/current/.config/lxc

pylxd (at least on version 2.2.11) automatically detects this path and uses it to fetch the certificate and key files.

Newer versions of LXD, including LTS 4.0.7, seem to create the certs at $HOME/snap/lxd/common/config/ which makes pylxd unable to find the files automatically if the paths are not provided.

stgraber · August 23, 2021, 7:33pm

@ack should be a pretty easy fix for pylxd?

ack · September 22, 2021, 9:46am

Pylxd currently looks for ~/snap/lxd/current/.config/lxc/client.crt and then ~/.config/lxc/client.crt.

I wonder if we shouldn’t just replace the former with /snap/lxd/common/config/client.crt given it seems to have been the default for quite a long time, or add it as a first option.

@stgraber WDYT?

stgraber · September 22, 2021, 5:09pm

Yeah, probably ought to replace the existing snap path as that’s what’s been widely used for some time now.