How do you "detach" from the console started from the API?

noah · October 13, 2019, 4:49pm

I’m starting a console operation using: (Host and Containers are running Ubuntu 18.04 with LXD 3.18)

curl -k -X POST --cert /vagrant/lxd.crt --key /vagrant/lxd.key https:/127.0.0.1:8443/1.0/containers/c3/console -d '{"width": 80, "height": 24}' | jq

Then using a command line websocket tool, I’m connecting to the operation websocket for the console using the UUID/secret provided from the above API call:

websocat -kvb wss://127.0.0.1:8443/1.0/operations/7f8c710a-c485-46d8-80ba-9e9bd0eff144/websocket?secret=dc95c8720314ef7abb5d5e998ef57d49bd0b25b493c7032dedd78ed15cb34c7d

This works I’m able to interact and login via the console, however I don’t know how to “detach” from the console.

Closing the websocket connection doesn’t seem to detach the console. When I call the operations API / websocket again or from the command line I get the errors below which I believe means the console is still in use by another process.

Websocket:

[INFO  websocat::ws_client_peer] Connected to ws
Error: Failed running forkconsole: "attaching to the container failed"

Command line:

ubuntu-bionic:~$ lxc console c3
To detach from the console, press: <ctrl>+a q
Error: Failed running forkconsole: "attaching to the container failed"
Error: exit status 1

Other issues:
When you close the websocket connection to the console operation and then reconnect to it again (using the same uuid/secret), interaction with the console no longer seems to work however the console is still attached somehow based on the error messages above. If you start a new console operation and connect to that websocket with the new uuid/secret, then you get the error message from above.

Also may_cancel is set to false for console operations so I’m unable to use the DELETE command to stop that operation. This would be a nice to have for the API to stop any “orphaned” console operations, since I’m unable to find a way to “resume” a previous websocket console operation OR a way to timeout a console operation after X amount of inactivity.

I understand that Ctrl + A then q is used to detach from the console using the lxc console command, but sending those characters via the websocket doesn’t seem to work. Also using Ctrl + c (SIGTERM?) doesn’t seem to work either.

** (I looked through the code at https://github.com/lxc/lxd/blob/873f6e045c291027fe4eba9ab7377cb0a248afde/lxd/container_console.go and I’m not very familiar with GO, but does that mean it’s looking for a SIGTERM to detach the console?)

Thank you in advance for any insight! I really appreciate it.

turtle0x1 · October 13, 2019, 6:11pm

The way I worked around this in my project was sending exit \r on disconnect to the console, which kills the operation

noah · October 13, 2019, 6:21pm

I tried that one too, but in my case it drops me back out to the “login” screen of the console and trying to run the console again (after closing the websocket) via lxc console c3 or the API gives the error Error: Failed running forkconsole: "attaching to the container failed".

I think for exec’ing bash in a console that would work though, but I believe (and I could be wrong) for the console launched via the API, it’s attached to /dev/console the “system” console which is limited to one interactive session.

turtle0x1 · October 13, 2019, 6:25pm

Oh sending exit \r over the websocket (the api) works for me!

What params are you sending to initiate the connection ? Mine look simlar to this

    "command": ["bash"],
    "environment": {
        "HOME": "/root",
        "TERM": "xterm",
        "USER": "root"
    },
    "wait-for-websocket": true,
    "interactive": true,

noah · October 13, 2019, 6:28pm

I’m running:
curl -k -X POST --cert /vagrant/lxd.crt --key /vagrant/lxd.key https:/127.0.0.1:8443/1.0/containers/c3/console -d '{"width": 80, "height": 24}' | jq which is using the /1.0/containers/c3/console end-point. I then take the uuid/secret returned and open the websocket connection.

Is your call using /containers/NAME/console end or /containers/NAME/exec?

turtle0x1 · October 13, 2019, 6:29pm

mines using /containers/NAME/exec

noah · October 13, 2019, 6:31pm

I can try that as another option, but my hope is to use the /console call to get to the console that a user would see as if they were plugged into the server itself. Thanks for the suggestions!

noah · October 15, 2019, 5:35pm

I believe I have figured out how to detach the console with the control websocket, but based on the messages the logger is showing, I’m not for sure if I’m doing it the “correct” way. However the hoped for result is happening now, yay!

I’ve pasted the LXD debug log for the entire session of the operation and control websocket for the /1.0/containers/NAME/console api call below.

Question 1:
When I disconnect from the control socket it kills (detaches) the console which is great, however the messages from the log show: (DBUG[10-15|16:47:17] Got error getting next reader websocket: close 1006 (abnormal closure):) which triggers a SIGTERM eventually, but makes me think I’m not doing it the “correct” way and is just cleaning itself up because of the socket close?

Should a SIGTERM be sent directly over the control socket somehow, like the “signal” command for for the exec api? (I tried this for the console control socket, but didn’t trigger anything).

–

Question 2:
The initial console size passed via /1.0/containers/NAME/console or any resize command sent over the control socket doesn’t seem to be changing the console window size. It always remains at 80x25.

However the debug log does show the command being processed (DBUG[10-15|16:46:27] Set window size to: 80x10) when I send {"command": "window-resize","args": {"width": "80","height": "10"}} over the control socket.

See the screenshot below to see the console size after the command. I’ve tried various width / heights, but console looks the same as the screenshot below.

For reference I tested the resize over the control socket for a bash session that was started with /1.0/containers/NAME/exec and the resize command did work and the console window resized correctly.

–

DEBUG LOG:

DBUG[10-15|16:45:44] Allowing untrusted GET                   url="/1.0/operations/71c1ec84-4069-4e6c-8112-4a223740c0ad/websocket?secret=4666d47c072a00aef65c4db9e46148c75f29d1eea17718a4f41b978289d889bf" ip=127.0.0.1:55800
DBUG[10-15|16:45:44] Connected websocket Operation: 71c1ec84-4069-4e6c-8112-4a223740c0ad 
DBUG[10-15|16:45:44] Handled websocket Operation: 71c1ec84-4069-4e6c-8112-4a223740c0ad 
DBUG[10-15|16:45:44] Starting to mirror websocket 
DBUG[10-15|16:45:59] Allowing untrusted GET                   url="/1.0/operations/71c1ec84-4069-4e6c-8112-4a223740c0ad/websocket?secret=204c365b4d3802e1ff46ce1b7a1d06f44b40fd044680669e9d51bd41ea2214f1" ip=127.0.0.1:55802
DBUG[10-15|16:45:59] Connected websocket Operation: 71c1ec84-4069-4e6c-8112-4a223740c0ad 
DBUG[10-15|16:45:59] Handled websocket Operation: 71c1ec84-4069-4e6c-8112-4a223740c0ad 
DBUG[10-15|16:46:27] Set window size to: 80x10 
DBUG[10-15|16:47:17] Got error getting next reader websocket: close 1006 (abnormal closure): unexpected EOF 
DBUG[10-15|16:47:17] Sent SIGTERM to pid 504 
DBUG[10-15|16:47:17] sending write barrier 
DBUG[10-15|16:47:17] Finished to read websocket 
DBUG[10-15|16:49:39] Got error getting next reader websocket: close 1006 (abnormal closure): unexpected EOF 
DBUG[10-15|16:49:39] Finished to write websocket 
DBUG[10-15|16:49:39] Finished to mirror websocket 
DBUG[10-15|16:49:39] Success for websocket operation: 71c1ec84-4069-4e6c-8112-4a223740c0ad

Screenshots of console after resize command sent with size still at 80x25 it appears:

Sorry to ping you directly @brauner it seemed this may be your area of expertise from the commit log on that area of code (https://github.com/lxc/lxd/commit/4735c05343f67cac86447a1c65249faf3d90379e). I apologize if it’s not as I don’t want to burden you with notifications / help requests. Thank you in advance!