Local Rails Development Process Hanging in Container

LXD
1

We have LXD 3.0.1 installed on an Ubuntu 18.04LTS virtual machine. For the sake of this explanation, I have created two Debian 9 containers for doing development in Ruby on Rails. I have set up the first to use macvlan so that it acquires an IP address from the network DHCP server (we’ll call it ‘c1’). I have set up the second to use the default bridged mode so that it acquires an IP address directly from the LXD host (we’ll call it ‘c2’).

To c2, I added the following proxy devices to port forward from the LXD host to the LXC container:

lxc config device add c2 port22 proxy listen=tcp:0.0.0.0:2000 connect=tcp:localhost:22
lxc config device add c2 port80 proxy listen=tcp:0.0.0.0:2001 connect=tcp:localhost:80
lxc config device add c2 port3750 proxy listen=tcp:0.0.0.0:2002 connect=tcp:localhost:3750

where 22 is for SSH’ing into the container, 80 is for the Nginx webserver installed on the container, and 3750 is the Rails server port I use when doing development of code on the container.

If I’m SSH’d onto c1 (the one with the network granted DHCP address) and I spin up my Rails server …

rails s -p 3750 -b 0.0.0.0

and open up Firefox and direct the URL to http://ip-address-to-container:3750, the pages load fine.

However, on c2, when I spin up the same server on local port 3750 and then try to access them via the LXD host and the port I mapped (http://ip-address-to-lxd-host:2002), the page loads, but the container never appears to send a close connection signal. Thus, in Firefox, the spinning balls that indicate that the browser is still waiting continue to spin. Even if I terminate the local Rails server (ctrl-c), the local process stops in the container, but Firefox still indicates it is waiting.

To further complicate matters, if on c2 I serve the very same project using Nginx (on port 80), and connect via http://ip-address-to-lxd-host:2001, the page loads fine and terminates the connection upon loading the page, as expected.

My questions are:

  1. Have I configured c2 incorrectly? Is there something I need to add to the port forwarding config device command?
  2. Is there something about port 80 versus 3750 (perhaps privileged port vs. unprivileged?) that makes one terminate connections upon completion vs. not?

Thanks for any help; sorry for the long explanation.

Edit: Formatting issues.

2

No takers, eh? :slight_smile:

So, I’m posting this as a workaround in case anyone else stumbles onto the same issue. As an experiment, I removed the proxy device port3750 from my original post

$lxc config device remove c2 port3750

and replaced it with an iptables rule on my LXD host. The IP address of c2 (the one running in bridged mode with an IP address assigned by the LXD host) is 10.77.164.183. Run (as root) on the LXD host machine:

#iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 2002 -j DNAT --to-destination 10.77.164.183:3750

where eth0 is the networking interface for your LXD host machine, 2002 is the port you want to forward to your container, and 3750 is the port you want to listen on inside your container.

You can confirm the new rule has been added by running

#iptables -L -n -t nat

Once I did that and spun up my local Rails server on c2, I was able to access the site in Firefox with no further issues.

3

Hi!

When I read the initial question, it was not clear what to make of it.

The stock answer would have been that in Rails (or similar) some code runs on the back-end (in the container) and some on the front-end (in the browser). Therefore, you could have touched an issue where a configuration (hostname or port) makes sense inside the container, but does not make sense if you run this on the front-end. Or, vice versa.

What was the root cause of the issue that you were facing?

4

A little backstory might be helpful …

I’ve been the one at my company championing the use of LXD containers as a way of isolating various software development projects from each other. Although all of the projects are Ruby on Rails, each requires different version of applications and libraries that in the past have conflicted with each other when the projects lived on the same server. Each container will normally have several developers working on their own copies of the source code (all managed through git repositories). In order to test the code, each developer spins up his/her own temporary Rails server on a pre-chosen port (mine happens to be 3750). This way, you can have multiple Rails servers running on the same container without any port conflicts.

Initially we spun up containers using ‘macvlan’ as our profile so that the containers would be routable directly to our network with network-assigned DHCP IP addresses and this is how we’ve been operating for the last four-five months. But after several of the containers changed IP addresses (which, it is DHCP after all, that does happen), we decided to test a container with bridged LXD host-assigned IP addresses and port-forward on the host by using the “lxc config device add …” command. This seemed to work fine for SSH and HTTP, but I was struggling to get it to work for my development port and Rails. After hours of searching on The Google and not making any headway, I thought I’d post here in case anyone had any suggestions.

5

When you do lxc config device mycontainer add proxy ..., you are adding a LXD proxy device (application layer), which is somewhat different from port forwarding (network layer).

The difference is in that the application in the container only sees the proxy device as the source IP address of all connections, instead of the actual source IP (as in the case of port forwarding).
That is a common issue when using any proxy (kubernetes, AWS, etc).

I gave it a try and I got a Connection timeout when accessing from some other computer, while it worked when I was connecting from the host (with LXD). Most likely this is the source of your timeouts.

The reason why we got the LXD proxy device, is to avoid having to setup port forwarding.
LXD (the snap package) though implements the PROXY protocol, which means the application in the container could figure out the real source IP as long as it understands the PROXY protocol.

6

Thank you for taking the time to look into it and writing your response. In my (tenuous-at-best) understanding of networking, port forwarding and proxy devices are the same, even though I realize that they aren’t.

7

If you were to enable the PROXY protocol in the LXD proxy device, you would just need to replace

lxc config device add c2 port3750 proxy listen=tcp:0.0.0.0:2002 connect=tcp:localhost:3750

with

lxc config device add c2 port3750 proxy listen=tcp:0.0.0.0:2002 connect=tcp:localhost:3750 proxy_protocol=true

And then, configure Ruby/Rails on the receiving end to understand the PROXY protocol (if possible). A quick search showed https://github.com/omarkj/proxyprotocol which means that there is some work on this. Obviously, find a properly supported package for the proxy protocol, if you intent to use the LXD proxy device.
Otherwise, you can just keep using port-forwarding with iptables.