New SFTP API implementation

Hye-Dev · March 8, 2022, 12:23am

Hello! I was looking at the API docs, and noticed that a new SFTP endpoint was added, but I can’t seem to get it to work. It always sends the protocol upgrade response, but I don’t know what to do from there. Can anyone help me out with this?

Thanks - Joseph

stgraber · March 8, 2022, 1:54am

It’s currently just in master, it will be included in LXD 4.24 which is due out later this week.

GET /1.0/instances/NAME/sftp can indeed be upgraded using Upgrade: sftp and Connection: Upgrade. This then turns the connection into a raw bi-directional connection to a sftp server.

In our case, we use Go’s native SFTP client to access it or you can pass that as stdin/stdout to sshfs -o slave (or -o passive in newer releases).

Hye-Dev · March 8, 2022, 2:01am

I’ve been trying it out with the latest/edge channel, trying to figure out a way to maybe use it in JS. So if I just use an sshfs library I should be good to go?

stgraber · March 8, 2022, 3:44am

If there is a library that lets you connect to the sftp server/protocol without establishing an ssh connection, then yes.

sftp package - github.com/pkg/sftp - pkg.go.dev is the Go version of such a client.

turtle0x1 · March 16, 2022, 10:46pm

Looks like if your trying to use this in node.js it will fail silently because node.js doesn’t know how to handle the SFTP protocol upgrade. Of course that’s probably out of scope for them but my plan was to provide the underlying Net.Socket to the SSH2 package but that wont be possible if node.js just kills the socket because it doesn’t understand it.

Hopefully im wrong and someone can correct me on this but in the mean time ive opened an issue here.

tomp · March 17, 2022, 9:32am

Its technically not a websocket, but an upgraded HTTPS connection passing the raw unencrypted SFTP protocol (not inside an SSH connection because its already encrypted by the HTTPS outer layer).

So really node.js just needs to support upgrading an HTTP connection to a raw stream and then providing a connection handle to that stream.

I don’t know very much about node.js, but keep in mind that the raw SFTP stream from LXD’s API is not an SSH stream, it is just an SFTP stream.

tomp · March 17, 2022, 9:33am

This is how we handle both passing stdout/stdin to sshfs and providing a local SSH SFTP listener in the lxc command using the SFTP stream from the LXD API:

github.com

lxc/lxd/blob/master/lxc/file.go#L910-L1240

      
        
            func (c *cmdFileMount) Command() *cobra.Command {
            	cmd := &cobra.Command{}
            	cmd.Use = usage("mount", i18n.G("[<remote>:]<instance>[/<path>] [<target path>]"))
            	cmd.Short = i18n.G("Mount files from instances")
            	cmd.Long = cli.FormatSection(i18n.G("Description"), i18n.G(
            		`Mount files from instances`))
            	cmd.Example = cli.FormatSection("", i18n.G(
            		`lxc file mount foo/root fooroot
               To mount /root from the instance foo onto the local fooroot directory.`))
            
            
	cmd.RunE = c.Run
            	cmd.Flags().StringVar(&c.flagListen, "listen", "", i18n.G("Setup SSH SFTP listener on address:port instead of mounting"))
            	cmd.Flags().BoolVar(&c.flagAuthNone, "no-auth", false, i18n.G("Disable authentication when using SSH SFTP listener"))
            	cmd.Flags().StringVar(&c.flagAuthUser, "auth-user", "", i18n.G("Set authentication user when using SSH SFTP listener"))
            
            
	return cmd
            }
            
            
func (c *cmdFileMount) Run(cmd *cobra.Command, args []string) error {
            	// Quick checks.

This file has been truncated. show original

turtle0x1 · March 17, 2022, 9:34am

The blurb for a Net.Socket is;

" This class is an abstraction of a TCP socket or a streaming IPC endpoint (uses named pipes on Windows, and Unix domain sockets otherwise). It is also an EventEmitter."

Thats whats needed here right…?

tomp · March 17, 2022, 9:40am

Yes sounds reasonable, but passing it to the SSH2 package (if its expecting an SSH stream) may be problematic.

turtle0x1 · March 17, 2022, 9:43am

One step at a time, if the language wont give me a socket cant hack at the next part

Hye-Dev · March 17, 2022, 3:59pm

I’ve already gotten it working in node by spawning a go child process then using a unix socket to communicate node <—> go. Works flawlessly.

tomp · March 17, 2022, 4:10pm

Nice!

We also have a fix that should make node.js happier:

turtle0x1 · March 17, 2022, 4:14pm

Nice!

Not a fan of the socket approach so im waiting on the above!