TLS Handshake error wss: to 1.0/operations

kamzar1 · February 20, 2022, 11:24am

wss://lxd.server:7443/1.0/operations/ b7e4cbcf-…/websocket?secret= 2e4c660…
dies with 1006.

lxd.log:
t=2022-02-20T12:04:00+0100 lvl=info msg=“http: TLS handshake error from 113.91.xx.xx:41741: remote error: tls: unknown certificate”
t=2022-02-20T12:04:01+0100 lvl=info msg=“http: TLS handshake error from 113.91.xx.xx:41745: EOF”

Operation :
[type] => sync
[status] => Success
[status_code] => 200
[operation] =>
[error_code] => 0
[error] =>
[metadata] => Array
(
[id] => b7e4cbcf-…
[class] => websocket
[description] => Showing console
[created_at] => 2022-02-20T11:49:50.042339877+01:00
[updated_at] => 2022-02-20T11:49:50.042339877+01:00
[status] => Running
[status_code] => 103
[resources] => Array
(
[containers] => Array
(
[0] => /1.0/containers/alp1
)

                [instances] => Array
                    (
                        [0] => /1.0/instances/alp1
                    )

            )

        [metadata] => Array
            (
                [fds] => Array
                    (
                        [0] => 2e4c660...
                        [control] => 7a3d499...
                    )

            )

        [may_cancel] => 
        [err] => 
        [location] => none
    )

kamzar1 · February 22, 2022, 8:13am

As certificate I have tried simple (key,cert) and PKI (ca,key,cert) each sha256/sha384 prime256v1/secp384r1.
DNS and IP in cert is packed with all possible domain, *.domain, possible ip …
Yet, as the ca is self signed, client seems to drop the connection wss:… to lxd websocket.
Which logs in lxd as:
lxd.log:
t=2022-02-20T12:04:00+0100 lvl=info msg=“http: TLS handshake error from 113.91.xx.xx:41741: remote error: tls: unknown certificate”

Is there a work around for this?
Apply a real certificate from known root ca for LXD Server?
In client do https to middleman/proxy (nginx) and from there upgrade the connection to wss?

tomp · February 22, 2022, 10:01am

You need to ensure the remote’s certificate is in the LXD trust store using lxc config trust add

kamzar1 · February 22, 2022, 1:44pm

Remote is a web client.

By using certificate, webserver requests an operation.
Operation id and secret are than used to build a websocket link, which is called through a javascript by web client.
TLS handshake error from 113.91.xx.xx ← IP is the web client IP which uses a browser executing within page wss://…
How can a certificate possibly be applied in this context:

wss://lxd.server:7443/1.0/operations/ b7e4cbcf-…/websocket?secret= 2e4c660…

turtle0x1 · February 22, 2022, 1:49pm

It cant, you have to proxy the socket operations with a backend (PHP cant do it, well unless you use swoole)

tomp · February 22, 2022, 1:54pm

@stgraber may have some thoughts on this?

kamzar1 · February 22, 2022, 2:20pm

Thanks for the hint.
Seems right.
I had a guess, but hoped avoiding the extra proxying the connection.
Would an nginx connection upgrade work? Call https://domain/something , than upgrade /something/ to wss:/lxdserver …
or need to call a node port which proxies to lxd server? A node middleware or agent?
It seems there are too many turning wheels in lxd interactive sockets business.

turtle0x1 · February 22, 2022, 2:38pm

nginx probably cant proxy it because the secrets are dynamically generated and not fixed.

If the above is true you need a full proxy written in Node in between, you can add some middlewere to auth your users. LXDMosaic does this but its GPL3 so if your writing a private application you probably need to work it out without looking at that code.

kamzar1 · February 22, 2022, 2:54pm

I will give nginx a try.
I can pass the whole link incl. secret to nginx https://domain/catchthis/operations/ b7e4cbcf-…/websocket?secret= 2e4c660…
Than run it through a rewrite rule to make the /catchtthis/ to proper link , upgrade it to wss:
Should above not work, yes indeed a node middleware is the solution.

kamzar1 · February 23, 2022, 6:40am

Both, nginx and haproxy can upgrade a connection and proxy to websocket.
haproxy seems to have more advanced options.
Request is sent to proxy, it renders http header and by ws/wss upgrades and forwards.

acl host_ws hdr_beg(Host) -i ws.
use_backend bk_ws if host_ws
routing based on websocket protocol header
acl hdr_connection_upgrade hdr(Connection) -i upgrade
acl hdr_upgrade_websocket hdr(Upgrade) -i websocket

I just was looking for a solution where the application server remains simple webserver html/php, not involving another middleman (node.js websocket server as proxy), saving the whole node installation and commission for middleman.

kamzar1 · March 17, 2022, 6:46pm

direct calls to wss://lxd.server:8443/1.0/operations/ b7e4cbcf-…/websocket?secret= 2e4c660…
fail with: 1006 and
lxd.log:

t=2022-02-20T12:04:00+0100 lvl=info msg=“http: TLS handshake error from 113.91.xx.xx:41741: remote error: tls: unknown certificate”
t=2022-02-20T12:04:01+0100 lvl=info msg=“http: TLS handshake error from 113.91.xx.xx:41745: EOF”

as correctly mentioned here:

wss client needs to be either proxied by a wss server instance or by another proxy such as nginx or haproxy.

In order to upgrade the http / https connection to websocket:

nginx takes the following configuration(tested, working). It shows two backends srv4 and srv8, you can create multiple backends and proxy by matching location or rewrite.

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

upstream srv4 {
    server 192.168.1.1:8443; # appserver_ip:ws_port
}
upstream srv8 {
    server 10.0.4.1:8443; # appserver_ip:ws_port
}
server {
      listen 80;
      listen 443 ssl;
      server_name api.domain.com;
      ssl_certificate      /etc/ssl/domain.cer;
      ssl_certificate_key  /etc/ssl/domain.key;
      ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
      ssl_ciphers         HIGH:!aNULL:!MD5;

    location /srv4 {
    rewrite ^/srv4^/ /$1 break;
        proxy_pass https://srv4;
        #proxy_ssl_certificate     /etc/ssl/domain.cer
        #proxy_ssl_certificate_key /etc/ssl/domain.key;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    location /srv8 {
    rewrite ^/srv8^/ /;
        proxy_pass https://srv8;
        #proxy_ssl_certificate     /etc/ssl/domain.cer;
        #proxy_ssl_certificate_key /etc/ssl/domain.key;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    location / {
        proxy_pass https://srv8;
        #proxy_ssl_certificate     /etc/ssl/domain.cer;
        #proxy_ssl_certificate_key /etc/ssl/domain.key;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
  }

now you send a request to https://api.domain.com/1.0/operations/ b7e4cbcf-…/websocket?secret= 2e4c660…
nginx will upgrade and send it to wss://10.0.4.1:8443/1.0/operations/ b7e4cbcf-…/websocket?secret= 2e4c660…

It is bidirectional, lxd reply/data is transferred back to https:// …

HA Proxy can act as middleware too. The advantage is, you can map all your lxd server instances and divert the requests bu filtering url, domain, subdomain,port etc. to desired lxd server instance declared as wss backend:

frontend fe_main
   bind :80
   use_backend websocket_servers if { path_beg /ws }

 frontend fe_main
   bind :80
   default_backend websocket_servers

backend websocket_servers
   option http-server-close
   option logasap
   timeout tunnel 1h
   server s1 192.168.1.1:8443 check