How to mount the SSH agent socket into LXD container

LXD
1

I would like to mount the ssh-agent socket from the LXD host to the LXD container. It’s basically the equivalent of doing docker run --volume $SSH_AUTH_SOCK:/ssh-agent in Docker.

From this Topic 2579 I have learned that there is a proxy device now.

Based on this, I have tried the following but did not manage to get it work.

  1. Created container

    $ lxc launch ubuntu:18.04 c1

  2. Attached device

    $ lxc config device add c1 ssh-agent proxy \
    connect=unix:/run/user/1000/keyring/ssh \
    listen=unix:/ssh-agent.sock \
    bind=container \
    uid=1000 \
    gid=1000 \
    mode=0777 \
    security.uid=65534 \
    security.gid=1000

  3. Connect to container

    $ lxc exec c1 – sudo --user ubuntu --login

  4. Created environment variable

    c1:~# export SSH_AUTH_SOCK=/ssh-agent.sock

  5. Run ssh-add -L

    c1:~# ssh-add -L
    Error connecting to agent: No such file or directory

  6. Printed proxy.ssh-agent.log

    $ cat /var/snap/lxd/common/lxd/logs/c1/proxy.ssh-agent.log
    Created anonymous pair {5,6} of unix sockets
    Added listener socket file descriptor 5 to epoll instance
    Starting unix ↔ unix proxy
    Error: Failed to connect to target: dial unix /run/user/1000/keyring/ssh: connect: no such file or directory
    Failed to prepare new listener instance: dial unix /run/user/1000/keyring/ssh: connect: no such file or directoryStarting unix ↔ unix proxy

BTW, the users on the host as well as in the container have the UId 1000. The ssh-agent socket file does exist on the LXD host and has following permission.

$ ls -lah /run/user/1000/keyring/ssh
srwxrwxr-x 1 usera usera 0 Sep 6 07:35 /run/user/1000/keyring/ssh

2

Hi!

The socket at /run/user/1000/keyring/ssh is the socket for GNOME Keyring.
SSH’s socket should be in /tmp/ (for ssh-agent). For example,

ubuntu@ssh:~$ export | grep SSH
declare -x SSH_AGENT_PID="392"
declare -x SSH_AUTH_SOCK="/tmp/ssh-mzfg5J9xDfMm/agent.391"
ubuntu@ssh:~$
3

@simos, thanks for your response!

This is running on my laptop and the /run/user/1000/keyring/ssh path is valid and it works fine when I run ssh-add -L on my laptop. Furthermore, I can use this path to mount it to a Docker container and ssh-add -L works as expected from within the Docker container.

This is how my SSH_AUTH_SOCK variable looks like.

$ printenv | grep SSH
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh

Anyway, to rule out any problems with the Gnome keyring, I have tested with a separate ssh-agent but the problem remains :frowning: .

$ eval “$(ssh-agent -s)”
$ printenv | grep SSH
SSH_AGENT_PID=26454
SSH_AUTH_SOCK=/tmp/ssh-sx30Ld1L68wV/agent.26453
$ ssh-add ~/.ssh/id_rsa
# => Verify that it lists your private ssh key
$ ssh-add -L

$ lxc launch ubuntu:18.04 c1
$ lxc exec c1 – sudo --user ubuntu --login
c1 $ mkdir /tmp/ssh-sx30Ld1L68wV
c1 $ export SSH_AUTH_SOCK=/tmp/ssh-sx30Ld1L68wV/agent.26453

> # => Back to the laptop and attach the device
$ lxc config device add c1 ssh-agent proxy \
connect=unix:/tmp/ssh-sx30Ld1L68wV/agent.26453 \
listen=unix:/tmp/ssh-sx30Ld1L68wV/agent.26453 \
bind=container \
uid=1000 \
gid=1000 \
mode=0660 \
security.uid=65534 \
security.gid=1000

> # Get the log
$ sudo cat /var/snap/lxd/common/lxd/logs/c1/proxy.ssh-agent.log
Created anonymous pair {5,6} of unix sockets
Added listener socket file descriptor 5 to epoll instance
# => So far it seems to work as there are no errors

> # => Back to the container c1
c1 $ ssh-add -L
error fetching identities: communication with agent failed

> # Get the log a second time
$ sudo cat /var/snap/lxd/common/lxd/logs/c1/proxy.ssh-agent.log
Created anonymous pair {5,6} of unix sockets
Added listener socket file descriptor 5 to epoll instance
Starting unix <-> unix proxy
Error: Failed to connect to target: dial unix /tmp/ssh-sx30Ld1L68wV/agent.26453: connect: no such file or directory
Failed to prepare new listener instance: dial unix /tmp/ssh-sx30Ld1L68wV/agent.26453: connect: no such file or directory

I have verified that the /tmp/ssh-sx30Ld1L68wV/agent.26453 file exists on both the host and the container and has the correct owner and file permissions. Both users have the UId of 1000 as well as the GId of 1000.

4

Okay, so the important message is this one in the proxy.ssh-agent.log log files.

connect: no such file or directory means that in the context of LXD, there is no ‘/run/user/1000/keyring/ssh’ file.

To verify, I used ssh-agent with the -a parameter (specify yourself the socket location), and tried

eval `ssh-agent -a /home/myusername/agent`

Then, I followed the rest of the steps and it worked.

Now, the issue is, what’s wrong with /tmp/ and /run/?
It is likely that the snap package confinement does not allow access to forkproxy to arbitrary directories.

1 Like
5

@simos Thanks for testing this.

I am using LXD installed by Snap and have not tested it with LXD installed by the Apt package. It’s useful to know that it works when the socket lives in /home/<username>/ directory. Based on your feedback I got it to work too.

Below is the sequence of steps in case it helps someone else.

$ eval ssh-agent -a /home/usera/agent
$ export | grep SSH
SSH_AGENT_PID=26454
SSH_AUTH_SOCK=/home/usera/agent
$ ssh-add ~/.ssh/id_rsa
$ ssh-add -L

$ lxc launch ubuntu:18.04 c1
$ lxc config device add c1 ssh-agent proxy
connect=unix:/home/usera/agent
listen=unix:/home/ubuntu/agent
bind=container
uid=1000
gid=1000
mode=0666
security.uid=1000
security.gid=1000

$ lxc exec c1 – sudo --user ubuntu --login
c1 $ export SSH_AUTH_SOCK=/home/ubuntu/agent
c1 $ ssh-add -L

This is a workaround I can live with, at least when it comes to ssh-agent forwarding. However in other use cases, such as forwarding the /var/run/docker.sock or /run/docker.sock file, this will not be an option. Unless I start Docker on the LXD host and explicitly tell it to write the Docker socket file to somewhere at /home/<username>/.

This looks like a bug to me. What do you think? Would you like me to create an issue on Github?

6

The LXD snap uses strict confinement (see the output of snap info lxd which does not make reference to classic).
However, there is a special interface called lxd-support that was created specially for LXD so that it can do many stuff even with the strict confinement (see the output of snap interfaces lxd).
I am not very familiar with what you get with the lxd-support interface or whether it can be adapted to help here.

Therefore, file a bug report on github (and post the link here).
As a title, use something like Strict confinement in LXD snap does not allow to proxy some sockets.

7

I have created the bug report Strict confinement in LXD snap does not allow to proxy some sockets #5009 on Github.

1 Like
8

There is a fix on this that is scheduled to appear in LXD 3.5,

From what it looks, the issue is with LXD only and does not require changes in the lxd-support interface of the snap packages.

Currently, the LXD snap package in the edge channel is still at commit git-7a22ab3 (see the output of snap info lxd). That commit is from two days ago (https://github.com/stgraber/lxd/commits/import) while the fix for this bug is from one day ago. Therefore, there is an extra day of way if you want to test/verify this fix by switching to the edge channel.

9

That fix is already in the stable snap too.

10

Thanks @stgraber and @simos for the great help and the swift fix.

With this fix in place, I could successfully make the proxy device work with ssh-agent, gpg-agent, docker and lxd.

Below are the socket locations on the host that worked.

ssh-agent:
/run/user/1000/keyring/ssh

gpg-agent:
/run/user/1000/gnupg/S.gpg-agent

lxd:
/var/snap/lxd/common/lxd/unix.socket

docker:
/run/docker.sock

However, if I use the /var/run/docker.sock path for Docker, it does not work and I get the below error in the log.

sudo cat /var/snap/lxd/common/lxd/logs/c1/proxy.docker.log
Error: Failed to connect to target: dial unix /var/lib/snapd/hostfs/var/run/docker.sock: connect: no such file or directory
Failed to prepare new listener instance: dial unix /var/lib/snapd/hostfs/var/run/docker.sock: connect: no such file or directoryError: Failed to connect to target: dial unix /var/lib/snapd/hostfs/var/run/docker.sock: connect: no such file or directory
Failed to prepare new listener instance: dial unix /var/lib/snapd/hostfs/var/run/docker.sock: connect: no such file or directory

Is this expected behaviour?

11

Hmm, so I think that the fact that /var/run is a symlink to /run is effectively working around the code we have for snap path resolution. It may be something we can fix by forcing an earlier symlink resolution, but in general I’d recommend staying away from symlinks.

12

@quater, I’ve followed your steps, but, why is this happening?
lxc exec bionic – sudo --user “ubuntu” --login
Error: invalid argument “ubuntu” for “–user” flag: strconv.ParseUint: parsing “ubuntu”: invalid syntax

Y had to change to “–sudo” instead “- sudo”.
But after that I am getting

ubuntu@bionic:~$ ssh-add -L
error fetching identities: communication with agent failed

I am using this device:

sudo lxc config device add bionic ssh-agent proxy \

connect=unix:/tmp/ssh-LB88TXdxxY/agent.2286 \

listen=unix:/home/ubuntu/agent \

bind=container \

uid=1000 \

gid=1000 \

mode=0666 \

security.uid=1000 \

security.gid=1000

ubuntu@bionic:~$ ls -la
total 28
drwxr-xr-x 3 ubuntu ubuntu 4096 Oct  2 07:37 .
drwxr-xr-x 3 root   root   4096 Sep 30 07:03 ..
-rw------- 1 ubuntu ubuntu  170 Oct  2 07:37 .bash_history
-rw-r--r-- 1 ubuntu ubuntu  220 Apr  4  2018 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3771 Apr  4  2018 .bashrc
-rw-r--r-- 1 ubuntu ubuntu  807 Apr  4  2018 .profile
drwx------ 2 ubuntu ubuntu 4096 Sep 30 07:03 .ssh
**srw-rw-rw- 1 ubuntu ubuntu    0 Oct  2 07:37 agent**
13

Ok, I didn’t do the export and ssh-add first command. With that a new error shows up:

vmasip@localhost:~/git-arm64/lxc$ sudo lxc stop bionic
vmasip@localhost:~/git-arm64/lxc$ sudo lxc start bionic -v
vmasip@localhost:~/git-arm64/lxc$ sudo ls /var/snap/lxd/common/lxd/logs/bionic -lart 
total 48
drwx------ 5 root root  4096 Sep 30 23:05 ..
-rw-r--r-- 1 root root     0 Oct  5 09:23 forkexec.log
-rw-r----- 1 root root  1022 Oct  5 10:46 lxc.log.old
-rw-r----- 1 root root  2237 Oct  5 10:46 lxc.conf
-rw-r--r-- 1 root root     0 Oct  5 10:46 forkstart.log
-rw-r----- 1 root root   720 Oct  5 10:46 lxc.log
drwxr-xr-x 2 root root  4096 Oct  5 10:46 .
-rw-r--r-- 1 root root    16 Oct  5 10:46 proxy.ssh-agent.log
-rw------- 1 root root 22521 Oct  5 10:47 console.log
vmasip@localhost:~/git-arm64/lxc$ sudo cat /var/snap/lxd/common/lxd/logs/bionic/proxy.ssh-agent.log
Status: Started
vmasip@localhost:~/git-arm64/lxc$ sudo lxc exec --verbose  bionic -- sudo --user "ubuntu" --login 
ubuntu@bionic:~$ export SSH_AUTH_SOCK=/home/ubuntu/agent
ubuntu@bionic:~$ ssh-add -L
error fetching identities: communication with agent failed
ubuntu@bionic:~$ exit
logout
vmasip@localhost:~/git-arm64/lxc$ sudo cat /var/snap/lxd/common/lxd/logs/bionic/proxy.ssh-agent.log
Status: Started
Warning: Failed to connect to target: dial unix /var/lib/snapd/hostfs/tmp/ssh-xFfY6m0Xng/agent.17619: connect: permission denied
Warning: Failed to prepare new listener instance: dial unix /var/lib/snapd/hostfs/tmp/ssh-    xFfY6m0Xng/agent.17619: connect: permission denied

Then I remove log and start again for executing with sudo:

  vmasip@localhost:~/git-arm64/lxc$ sudo rm /var/snap/lxd/common/lxd/logs/bionic/proxy.ssh-agent.log
vmasip@localhost:~/git-arm64/lxc$ sudo lxc stop bionic
vmasip@localhost:~/git-arm64/lxc$ sudo lxc start bionic -v
vmasip@localhost:~/git-arm64/lxc$ sudo lxc exec --verbose  bionic -- sudo --user "ubuntu" --login 
ubuntu@bionic:~$ export SSH_AUTH_SOCK=/home/ubuntu/agent
ubuntu@bionic:~$ sudo ssh-add -L
Could not open a connection to your authentication agent.
ubuntu@bionic:~$ exit
logout
vmasip@localhost:~/git-arm64/lxc$ sudo cat /var/snap/lxd/common/lxd/logs/bionic/proxy.ssh-agent.log
Status: Started

Well, looks like strange,but logical because device has de uid:gid it has.

14

I’ve got the same problem EVEN in home folder:

vmasip@localhost:~/git-arm64/lxc$ sudo cat /var/snap/lxd/common/lxd/logs/bionic/proxy.ssh-agent.log
Status: Started
Warning: Failed to connect to target: dial unix /var/lib/snapd/hostfs/home/vmasip/agent: connect: permission denied
Warning: Failed to prepare new listener instance: dial unix /var/lib/snapd/hostfs/home/vmasip/agent: connect: permission denied
Warning: Failed to connect to target: dial unix /var/lib/snapd/hostfs/home/vmasip/agent: connect: permission denied
Warning: Failed to prepare new listener instance: dial unix /var/lib/snapd/hostfs/home/vmasip/agent: connect: permission denied
Warning: Failed to connect to target: dial unix /var/lib/snapd/hostfs/home/vmasip/agent: connect: permission denied
Warning: Failed to prepare new listener instance: dial unix /var/lib/snapd/hostfs/home/vmasip/agent: connect: permission denied
vmasip@localhost:~/git-arm64/lxc$ export | grep SSH
declare -x SSH_AGENT_PID="936"
declare -x SSH_AUTH_SOCK="/home/vmasip/agent"
declare -x SSH_CLIENT="10.0.2.25 32874 22"
declare -x SSH_CONNECTION="10.0.2.25 32874 10.0.3.212 22"
vmasip@localhost:~/git-arm64/lxc$ ps -ef | grep ssh-agent
vmasip     936     1  0 11:48 ?        00:00:00 ssh-agent -a /home/vmasip/agent
vmasip    3403 32296  0 12:14 pts/1    00:00:00 grep --color=auto ssh-agent

Am I doing something wrong?