TCP connection to host ip from container refused

LXD
1

Hi

I am trying to get to work a single sign on of mattermost via gitlab in a container.
On the level of the host I forward port 443 into the container which works seamlessly.
I can access both gitlab and mattermost from outside.
However, single sign on fails just as a ‘telnet host-ip 443’ from within the container fails.
I could of course set 127.0.0.1 for both gitlab and mattermost in /etc/hosts but then I get a certificate mismatch with error “x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0”

What is the recommended way to make this work?

Thanks

2

How are you doing that port forwarding?

3

I have tried both with

iptables -t nat -A PREROUTING -i eno8 -p tcp --dport 443 -j DNAT --to-destination 10.65.xx.xx:443

and

lxc config device add gitlab h443gl proxy listen=tcp:0.0.0.0:443 connect=tcp:127.0.0.1:443

Both lead to the same result.

When I add proxy_protocol=true on the second command I get an error even when accessing the web server which is clear (I would need to configure that).

The thing is, the host needs to forward 443 to 443 in the container, and for the single sign-on the container wants to access the host by addressing the FQDN.
So if I let it address the FQDN and therefore do a loop over the host, there would need to be a loop of 443 forwards which seems odd.
On the other hand, if I would add a SAN to the certificate so that I could redirect in /etc/hosts in the container, which address should I define in the SAN? AFAIK, adding localhost as a SAN to a certificate is not recommended.

4

May be interesting to try the proxy device with nat=true set. You’ll need your container to use a fixed address though ipv4.address.

I believe that when we setup our NAT rules, we try to setup a hairpin too which then handles such cases. Your manual NAT rule will only trigger when the traffic is coming in from eno8 which it isn’t when originating from the container.

1 Like
5

I have set ipv4.address but trying with nat=true I get

david@nnwh:~$ lxc config device add gitlab h443gl proxy listen=tcp:0.0.0.0:443 connect=tcp:127.0.0.1:443 nat=true
Error: Invalid devices: Device validation failed for “h443gl”: Cannot listen on wildcard address “0.0.0.0” when in nat mode

Perhaps a solution could be not to use NAT but a bridge instead?
The disadvantage would be that the container would see the external IP address, which I would like to avoid if possible.

6

Your proxy config is missing the ip address.

Should look like this.

proxy-name:
    connect: tcp:0.0.0.0:443
    listen: tcp:123.123.123.123:443
    nat: "true"
    type: proxy
8

In this case port 443 on the host is only accessible from one single IP address which clearly is not what I need…

9

Error: Invalid devices: Device validation failed for “h443gl”: Cannot listen on wildcard address “0.0.0.0” when in nat mode

As @stgraber said to use a NAT proxy with a fixed IP address, and the proxy config looked incorrect to me.

10

No thinkable combination works at all, even when I use my actual external IP address I am connecting from:

david@nnwh:~$ lxc config device add gitlab h443gl proxy listen=tcp:80.219.xxx.xxx:443 connect=tcp:0.0.0.0:443 nat=true
Device h443gl added to gitlab

Connection doesn’t work.

david@nnwh:~$ lxc config device add gitlab h443gl proxy listen=tcp:80.219.xxx.xxx:443 connect=tcp:127.0.0.1:443 nat=true
Error: Failed to start device “h443gl”: Proxy connect IP cannot be used with any of the instance NICs static IPs

david@nnwh:~$ lxc config device add gitlab h443gl proxy listen=tcp:80.219.xxx.xxx:443 connect=tcp:10.65.xx.xx:443 nat=true
Device h443gl added to gitlab

Connection doesn’t work.

11

You have to use the IP address of the LXD server, ie. the public IP address.

This is my working configuration for a reverse proxy, using the public IP address of the server.

  proxy-8080:
    connect: tcp:0.0.0.0:80
    listen: tcp:123.123.123.111:80
    nat: "true"
    type: proxy
  proxy-443443:
    connect: tcp:0.0.0.0:443
    listen: tcp:123.123.123.111:443
    nat: "true"
    type: proxy

If it is not working, maybe you changed something with IP tables? I have seen other users with similar problem where proxy did not work, they also made changes to IP tables rules. I am not able to help you with that (iptables) sorry.

12

Ok, this also works to access the server in the container (even though my original working setting seems more logical to me, but it doesn’t work with nat=true).
However, my main problem is still there. Interestingly, with this setup I am getting the same error like when I set the FQDN of my Gitlab to localhost in /etc/hosts inside the container:

{“level”:“error”,“ts”:1624622987.7921827,“caller”:“mlog/log.go:247”,“msg”:“Token request failed.”,“path”:“/signup/gitlab/complete”,“request_id”:“fq88dzmzqbndfbshpjqy75a47h”,“ip_addr”:“127.0.0.1”,“user_id”:“”,“method”:“GET”,“err_where”:“AuthorizeOAuthUser”,“http_code”:500,“err_details”:“Post "https://gitlab.example.com/oauth/token\”: x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0"}

So maybe one solution could be to add the now fixed IP of the container as a SAN to the certificate…
But isn’t there an easier way?

13

You need to make sure the service inside the instance is listening on the proxys connect IP rather than tgr loopback address if using Nat mode.