Cant SSH out of Container with docker on the host

mpaccione · August 19, 2021, 11:04pm

Hello, Software Engineer here with no networking experience.

I’ve made a LXD container for work on my local workstation. I am looking to use it as a container environment for my day job. Unfortunately when I try to SSH to the AWS instance I don’t have a connection.

I think I’m supposed to do some proxy or port forwarding but because I have no networking experience I need guidance. I can’t do this entirely on my own here from my lack of base knowledge.

Here’s my stack overflow question with more details (Has a bounty!)

Thank you for help!

tomp · August 20, 2021, 8:34am

Please can you confirm that you can access the AWS instance from the host that the LXD instance is running on?

If so, then the please also provide output of the following commands:

sudo lxc config show <instance> --expanded
ip a and ip r on the LXD host and inside the instance.

mpaccione · August 22, 2021, 5:26pm

@tomp Thanks for the assist Thomas!

Please note on the host I am currently SSH’ed in to all the services (3 DB’s)

Here are the output for HOST

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 60:a4:4c:59:07:53 brd ff:ff:ff:ff:ff:ff
3: wlx106f3fec78d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:6f:3f:ec:78:d7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.9/24 brd 192.168.0.255 scope global dynamic noprefixroute wlx106f3fec78d7
       valid_lft 3549sec preferred_lft 3549sec
    inet6 2601:645:8780:3850:2bac:ddff:cf5f:70d4/64 scope global temporary dynamic 
       valid_lft 3598sec preferred_lft 3598sec
    inet6 2601:645:8780:3850:7689:eba9:5877:1c61/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3598sec preferred_lft 3598sec
    inet6 fe80::d1b6:11d2:819a:6192/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: br-1c4579abb489: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:62:0d:43:1c brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-1c4579abb489
       valid_lft forever preferred_lft forever
    inet6 fe80::42:62ff:fe0d:431c/64 scope link 
       valid_lft forever preferred_lft forever
5: br-c0b40a8a1536: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:f4:09:81:e8 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.1/16 brd 172.19.255.255 scope global br-c0b40a8a1536
       valid_lft forever preferred_lft forever
    inet6 fe80::42:f4ff:fe09:81e8/64 scope link 
       valid_lft forever preferred_lft forever
6: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:20:a3:26:e0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:20ff:fea3:26e0/64 scope link 
       valid_lft forever preferred_lft forever
8: veth0406c6c@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c0b40a8a1536 state UP group default 
    link/ether 96:7a:19:4a:db:3d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::947a:19ff:fe4a:db3d/64 scope link 
       valid_lft forever preferred_lft forever
10: veth193b996@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-1c4579abb489 state UP group default 
    link/ether ce:c2:9c:59:d3:d5 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::ccc2:9cff:fe59:d3d5/64 scope link 
       valid_lft forever preferred_lft forever
12: veth439fe4b@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-1c4579abb489 state UP group default 
    link/ether 76:53:7b:e7:ef:24 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::7453:7bff:fee7:ef24/64 scope link 
       valid_lft forever preferred_lft forever
14: veth036c4ed@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c0b40a8a1536 state UP group default 
    link/ether b2:b0:98:0a:b2:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::b0b0:98ff:fe0a:b2e0/64 scope link 
       valid_lft forever preferred_lft forever
15: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:02:75:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.156.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fd42:731d:41b0:7b2d::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe02:75c8/64 scope link 
       valid_lft forever preferred_lft forever
17: veth689b37b6@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether 56:65:99:2f:74:5a brd ff:ff:ff:ff:ff:ff link-netnsid 4

default via 192.168.0.1 dev wlx106f3fec78d7 proto dhcp metric 600 
10.0.156.0/24 dev lxdbr0 proto kernel scope link src 10.0.156.1 
169.254.0.0/16 dev br-1c4579abb489 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-1c4579abb489 proto kernel scope link src 172.18.0.1 
172.19.0.0/16 dev br-c0b40a8a1536 proto kernel scope link src 172.19.0.1 
192.168.0.0/24 dev wlx106f3fec78d7 proto kernel scope link src 192.168.0.9 metric 600

Output for CONTAINER

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:4a:27:9f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.156.93/24 brd 10.0.156.255 scope global dynamic eth0
       valid_lft 3377sec preferred_lft 3377sec
    inet6 fd42:731d:41b0:7b2d:216:3eff:fe4a:279f/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 3306sec preferred_lft 3306sec
    inet6 fe80::216:3eff:fe4a:279f/64 scope link 
       valid_lft forever preferred_lft forever

default via 10.0.156.1 dev eth0 proto dhcp src 10.0.156.93 metric 100 
10.0.156.0/24 dev eth0 proto kernel scope link src 10.0.156.93 
10.0.156.1 dev eth0 proto dhcp scope link src 10.0.156.93 metric 100

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20210812)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20210812"
  image.type: squashfs
  image.version: "20.04"
  volatile.base_image: fab57376cf04b817d43804d079321241ce98d3b5c2296f1a41541de6c100ab09
  volatile.eth0.host_name: veth689b37b6
  volatile.eth0.hwaddr: 00:16:3e:4a:27:9f
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: d828689b-c939-4c54-b00b-939b5bde230e
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

tomp · August 23, 2021, 8:21am

Ah I see you have docker installed on the host, this is most likely the cause of the issue as it modifies the host’s firewall in such a way that prevents other bridges (such as LXD’s lxdbr0) from being able to reach the external network.

See Lxd and Docker Firewall Redux - How to deal with FORWARD policy set to drop - #3 by tomp

mpaccione · August 23, 2021, 9:53pm

Okay great! Thanks for the link. I took a quick look right now and am curious…

If I systemctl stop docker then would this work hypothetically or do I have to change the iptables regardless?

tomp · August 24, 2021, 8:48am

I think you would need to ensure that docker hasn’t started on a fresh boot (so it doesn’t add any iptables rules) and then check if LXD is OK.

If so then you know you’re on the right track.

placid_chat · August 24, 2021, 10:42am

You should be able to add your own firewall rules that take precedent before the docker ones.

mpaccione · August 24, 2021, 9:38pm

Definitely docker… uninstalled that temporarily and I can SSH in perfectly… Now to just figure out how to run the instance and view the clientside in a browser on the host

tomp · August 25, 2021, 8:18am

Can you expand on what you mean by that?

mpaccione · August 25, 2021, 6:54pm

Oh it ended up being so easy tom! ssh -X devbox and having my devbox setup in the ssh_config. Ensuring that xauth and what not is installed on the instance and that X server forwarding is set to Yes.