Making UFW (firewalls) work inside containers? Or share log directory from container to main host?

[TomvB (Tom)]

It’s important to keep your host clean. This is the base for all your containers and VMs.

I “attempted” (before you replied) to run nginx on the main host and with a proxy forward to the web containers, but I kept getting site errors. The nginx logs were showing some strange symbols.

That seems more NGINX / config related. That has nothing to do with LXD.

[Skyrider]

As for your rules… I have the following in the rules.v4 file:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [11:736]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -s myownip -j DROP
COMMIT

I refreshed the file

netfilter-persistent reload

And I didn’t appear to get blocked on my own website.

[TomvB (Tom)]

Try iptables -S do you see something there?

[Skyrider]

-P INPUT DROP
-P FORWARD DROP
-P OUTPUT ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -s myip/32 -j DROP

In case it matters. I’m using proxy protocol as device config, not NAT.

[TomvB (Tom)]

@Skyrider

proxy protocol

How to use the LXD Proxy Device to map ports between the host and the containers

For example, when someone connects to your host on port 80 (http), then this connection can be proxied to a container using a proxy device . In that way, you can isolate your Web server into a LXD container. By using a TCP proxy device, you do not need to use iptables instead.

But I understand your problem now. Enable NAT again for your container.
Go to your UFW before.rules config on your HOST:
nano /etc/ufw/before.rules

Check your host network adapter name and add before *filter:
*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -i YOURADAPTER -p tcp --dport 443 -j DNAT --to CONTAINER-IP:443
-A PREROUTING -i YOURADAPTER -p tcp --dport 80 -j DNAT --to CONTAINER-IP:80
COMMIT

Leave the container config. It is still important to use the iptables for fail2ban etc.

[Skyrider]

Done, though now all sites are showing ERR_CONNECTION_RESET

In case you wonder:

*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to 10.248.110.211:443
-A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to 10.248.110.211:80
COMMIT

"80":
  connect: tcp:0.0.0.0:80
  listen: tcp:hostip:80
  nat: "true"
  type: proxy
"443":
  connect: tcp:0.0.0.0:443
  listen: tcp:hostip:443
  nat: "true"
  type: proxy
eth0:
  ipv4.address: 10.248.110.211
  name: eth0
  network: lxdbr0
  type: nic

[TomvB (Tom)]

Be sure that you stop nginx on your host.

Port 80 and 443 are both open in your container.
Be sure port 80 and 443 are also open on your host.

Connect after this to your host ip and it should work.

[Skyrider]

nginx is stopped on my host, else I couldn’t add the NAT proxy device (it would say 80/443 was in use). port 80/443 already exists on my hosts ufw rules. As for the container:

-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

Don’t they open up the ports?

[TomvB (Tom)]

Yes, i’m talking about the host right now. The container is ok.

[Skyrider]

As I mentioned above, port 80/443 is already open in my ufw firewall rules. (on the host)

[TomvB (Tom)]

Ok good, please unban yourself etc. check all configs, restart ufw on the host. Reboot the container

[Skyrider]

Way ahead of you, before I replied above already unbanned myself, restarted ufw and the container. No luck. Still same error.

Checked the error logs of nginx on the container:

" while reading PROXY protocol, client: userip, server: 0.0.0.0:80
2020/07/19 13:20:41 [error] 227#227: 1274 broken header: "^V^C^A^A^P^A^@^A^L^C^C¶Í^@ ^F­Ô^<9f>f<98>²ÿÅò<9c>|~<99>+È<o_<9d>8"ä®xR^@^@pÀ/À+À0À,^@<9e>À’^@gÀ(^@k^@£^@<9f>̨̩̪À¯À­À£À<9f>À]ÀaÀWÀS^@¢À®À¬À¢À<9e>À\ÀÀVÀR" while reading PROXY protocol, client: userip, server: 0.0.0.0:443 2020/07/19 13:20:44 [error] 227#227: *1275 broken header: "^V^C^A^B^@^A^@^Aü^C^Cf7¤, «^@Q8%øîéÙc!§VõýÃ_Ùb}^A<û·wt |®&Ù­^Kh<88><8a><9c>!E<9b><81>ð<80>âoþ~n’Ö^¥UÚ^Z´ÝB^@&ºº^S^A^S^B^S^CÀ+À/̨̩À,À0À À^SÀ
À^T^@<9c>" while reading PROXY protocol, client: userip, server: 0.0.0.0:443

Don’t think the current nginx proxy forward seems to be working anymore?

server {
listen 443 ssl http2;
server_name website;
SSL STUFF

ssl_stapling on;
ssl_stapling_verify on;

ssl on;
ssl_session_cache  builtin:1000  shared:SSL:10m;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;

location / {
  proxy_set_header        Host $host;
  proxy_set_header        X-Real-IP $remote_addr;
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header        X-Forwarded-Proto $scheme;

  proxy_pass          http://container.lxd:80;
  proxy_read_timeout  90;

  proxy_redirect      http://container.lxd:80 https://website;
  real_ip_header proxy_protocol;

}

}

[TomvB (Tom)]

Just keep it simple. Try the default nginx config first, or try a clean container without firewall or configs.

[Skyrider]

@TomvB

Try the default nginx config first, or try a clean container without firewall or configs.

Not going to do any good if the nginx proxy container has no files at all. All domains have their own container. Hence I can’t believe that this cannot be simplified…

maybe LXD needs to add a “pass-firewall=true” rule? So you know… to avoid this kind of silly thing.

[TomvB (Tom)]

Nope, all you have to do is adding the following rules on your host:

*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to 10.248.110.211:443
-A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to 10.248.110.211:80
COMMIT

And open port 80 / 443 on your container if you use NAT. Nothing more, thats it. You can also check if your networking inside the container is ok.

[Skyrider]

Which might be true… but those rules broke all my websites. I was forced to erase the rule from my hosts iptables in order to revert back to proxy protocol to make things work again. (no longer on NAT) Either there’s something terribly wrong… Or the nginx files on the container are just not compatible as it no longer has to follow the proxy protocol rules… .Seeing, there is no proxy anymore.

Have you ever tried these rules on a single nginx proxy container which is connected to web containers? If you have, care to share your nginx config? Because I seriously cannot get this to work.

[TomvB (Tom)]

No config at the moment.

But… What I did before:

• Ubuntu 20.04 LXD Host with Port 80 and 443 forward to my NGINX Proxy
• NGINX Proxy container (NAT IP from lxdbr0) with virtual host proxy to the containers below:
  • NGINX Webserver 1 (NAT IP from lxdbr0)
  • NGINX Webserver 2 (NAT IP from lxdbr0)

Example Proxy:

server {
  listen 443 ssl http2;
  server_name domain;

  client_max_body_size 128M;
  SSL STUFF

  location / {
    proxy_pass http://<NGINX WEBSERVER>:80;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

}

But, if you are not sure about your nginx config. Use a clean container, install nginx and test it with webserver content. Proxy or not, it’s still serving webcontent from your virtualhost.

[Skyrider]

Thanks. I shall try this on my laptop instead. Easier than dealing with all the downtime of all my domains.

[Skyrider]

Btw…

• NGINX Webserver 1 (NAT IP from lxdbr0)
• NGINX Webserver 2 (NAT IP from lxdbr0)

Are you saying the web containers also should contain NAT? Or its config includes the IP from the nginx proxy container? Because 80/443 can only be used on 1 device.

[TomvB (Tom)]

Yes, of course. The host is their gateway. They must all be in the same subnet.