SSH timeout with port forwards

Hey all, new to LXD so apologies if this is already asked. I tried finding a applicable thread / stackoverflow but I cannot seem to find the answer on my own.

Problem summary: Connecting ssh with the internal network address works. When trying to connect with the listening address (in this case localhost (127.0.0.1)) get timed-out.

I am setting up a tool that creates a bunch of containers for a student lab setting. The generation and deployment of the containers, access keys, and port forward settings works. However, I am missing the final step of actually making the port forwards work.

Time-out often means that a firewall is rejecting a connection. However, I can’t seem to be able to find it. Network config has "ipv4.firewall": "false". NFtables / UFW is not running.

Extra info: Host is on a Hyper-V VM, running Ubuntu 21.10.

Ok. Between the moment of submitting this topic, and this reply. I had a burst of inspiration to not fixate on the 127.0.0.1 aspect of things, but use the external network IP as the listening address…

AND behold! It works! Even from the host’s-host that is running the Hyper-V hypervisor (or that special container that has all access). So for anyone in the future finding this, might I suggest not using 127.0.0.1 as a listening address.

Can someone enlighten me why localhost wouldn’t work for me? I am curious.

I’m not sure of what you’re trying to do or the config you are using as you’ve not provided it.

Please can you show your specific setup as well as the specific command you’re trying that isn’t working.

Please also show the output of lxc config show <instance> --expanded for any relevant instance and the output of ip a and ip r from inside the container and on the host.

Thanks

Thank you for your reply.
It is a python script that uses pylxd.
Code that creates the network:

client.networks.create(network_name, description="Nested Network for Linux Labs", type="bridge", config={
            "ipv4.address": "auto",
            "ipv4.nat": "true",
            "ipv6.address": "none",
            "ipv4.firewall": "false" # I am here for debugging please remove
        })

Code that creates the forward and the port forward:
Of note, I wasn’t able to use pylxd for that and still wish to. However that is another issue outside the scope of my question (maybe) as it pertains a custom post/put call with pylxd.

os.system(f"lxc network forward create {network_name} {listen_address}") #TODO use pylxd api instead of CLI tool

Imagine the next snippet being in a loop that iterates through a participant csv-file, that creates and starts a container-instance. It will create a unique ssh port and web port for each instance.

os.system(f"lxc network forward port add {network_name} {listen_address} tcp {source_port} {inet[0]['address']} {target_port}")

Output from lxc config show <instance> --expanded:

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 21.04 amd64 (release) (20220118)
  image.label: release
  image.os: ubuntu
  image.release: hirsute
  image.serial: "20220118"
  image.type: squashfs
  image.version: "21.04"
  user.user-data: |
    #cloud-config
    users:
    - groups: sudo
      name: test
      shell: /bin/bash
      ssh-authorized-keys:
      - ssh-rsa <<snip>>
      sudo:
      - ALL=(ALL) NOPASSWD:ALL
  volatile.base_image: 1b4e4d326603c8150bcd00857539bc4a5b0f90703f1ec316c6be76749c9651e6
  volatile.eth0.hwaddr: 00:16:3e:e6:7d:b8
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 36b6d000-b3bb-4786-b75b-add06a41d029
devices:
  eth0:
    name: eth0
    network: nestbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
- nestpr0
stateful: false
description: ""

Output of the nestpr0 profile:

config: {}
description: ""
devices:
  eth0:
    name: eth0
    network: nestbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: nestpr0

Port forward config provided by lxc network forward show {network_name} {listen_address}

description: ""
config: {}
ports:
- description: ""
  protocol: tcp
  listen_port: "52200"
  target_port: "22"
  target_address: 10.72.93.115
- description: ""
  protocol: tcp
  listen_port: "58000"
  target_port: "80"
  target_address: 10.72.93.115
listen_address: 172.20.77.89
location: none

Only difference here would be the listening address. That would be 127.0.0.1 while I was testing with localhost.
The following behavior is visible with localhost:

  • When using ssh -p 52200 test@127.0.0.1 it will timeout. 52200 is the configured port.
  • When using any other ports (like 49999) it will refuse the connection.

LXC was initiated with lxd init --auto on Ubuntu 21.10 as I was thinking that lxd was falsely identifying the firewalls of Ubuntu 20.04 as nftables during my initial troubleshooting searches (through lxc info | grep firewall:). I can confirm that 20.04 exhibits the same behavior.

I hope this is enough information.

EDITS: some rewording and clarfication

So are you saying that connecting to 172.20.77.89 works but 127.0.0.1 does not?

I’m still not clear on the actual issue I’m afraid. Can you show command examples of what works and what does work?

Yes. That exactly.

Powershell on the HyperV host:

ssh test@172.20.75.134 -p 52200
The authenticity of host '[172.20.75.134]:52200 ([172.20.75.134]:52200)' can't be established.
ECDSA key fingerprint is SHA256:vr02vfWpMlzs0gvyIpwfnSXKstKv8XZ37cJn6pADQNY.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[172.20.75.134]:52200' (ECDSA) to the list of known hosts.

Output on LXD host:

(.venv) cvirt@cvirt-Virtual-Machine:~/projects/linux-nester$ ssh -i output/nest_1648567950.6796708/keys/Nest-Te-Tester.pem test@172.20.75.134 -p 52200 -v
OpenSSH_8.4p1 Ubuntu-6ubuntu2.1, OpenSSL 1.1.1l  24 Aug 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 172.20.75.134 [172.20.75.134] port 52200.
debug1: Connection established.
debug1: identity file output/nest_1648567950.6796708/keys/Nest-Te-Tester.pem type -1
debug1: identity file output/nest_1648567950.6796708/keys/Nest-Te-Tester.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Ubuntu-6ubuntu2.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4p1 Ubuntu-5ubuntu1.2
debug1: match: OpenSSH_8.4p1 Ubuntu-5ubuntu1.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 172.20.75.134:52200 as 'test'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:aYCp8qhWF/dZ9Z3AwS1/2N6CNxRqZZhY8vVWWuu1ad4
debug1: Host '[172.20.75.134]:52200' is known and matches the ECDSA host key.
debug1: Found key in /home/cvirt/.ssh/known_hosts:4
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: output/nest_1648567950.6796708/keys/Nest-Te-Tester.pem  explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: output/nest_1648567950.6796708/keys/Nest-Te-Tester.pem
debug1: Authentication succeeded (publickey).
Authenticated to 172.20.75.134 ([172.20.75.134]:52200).

Output on LXD host when testing with 127.0.0.1 as listening address.

ssh test@127.0.0.1 -p 52200 -v
OpenSSH_8.4p1 Ubuntu-6ubuntu2.1, OpenSSL 1.1.1l  24 Aug 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 52200.
debug1: connect to address 127.0.0.1 port 52200: Connection timed out
ssh: connect to host 127.0.0.1 port 52200: Connection timed out

Mind that I redeploy the whole stack with a different listener. I am not trying to do both at the same time.

Using network forwards you won’t be able to connec to 127.0.0.1 on the LXD host-side unless you expicitly set up a network forward listening on 127.0.0.1.

I am aware that I cannot use 127.0.0.1 on my Hyper-V host to connect inside my Hyper-V VMs LXC containers. Without some forwarding on the Hyper-V Host side. I didn’t show a powershell attempt of that as I am aware it will be refused as it has nowhere to go.

I purge the LXD installation between every test. Between showing the output for the external IP listener (172.20.75.134) and the internal listener (127.0.0.1) the VM is rolled back to a clean state and the stack is reinstalled entirely.

I am not trying to setup a listener for 172.20.75.134 and magically expect 127.0.0.1 to work as well.

The thing I am trying to figure out is why the stack with the external IP (172.20.75.134) works. Even outside the LXD VM.
Yet, when I redeploy this stack with the listener address set to 127.0.0.1, it will timeout all connection attempts. Knowing I can not access it outside the LXD VM.

I am uncertain if that was clear. I feel that in the response that was given it was assumed I was trying to connect to a 127.0.0.1 address without a listener for it.
I am just trying to understand why it works differently for a listener on 127.0.0.1 and a listener on a different ip.

EDIT: To clarify even more this is the output when the listener is set to the external address IP (172.20.75.134):

ssh -i output/nest_1648567950.6796708/keys/Nest-Te-Tester.pem test@127.0.0.1 -p 52200 -v
OpenSSH_8.4p1 Ubuntu-6ubuntu2.1, OpenSSL 1.1.1l  24 Aug 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 52200.
debug1: connect to address 127.0.0.1 port 52200: Connection refused
ssh: connect to host 127.0.0.1 port 52200: Connection refused

I expect it to refuse the connection, and it does.