In case you are following, I have decided to nuke my server and start over (LXD proved to be the least of my problems following an ill-advised upgrade to Ubuntu 19.04) As such, I am using this as an opportunity to start from scratch. I like the idea of using the lxdbr0 and forwarding ports from the host to an lxd container. Ive created a container and installed PiHole. Pihole requires ports 53 and 80. I have set-up port forwarding as best I know how (ex. sudo lxc config device add PiHole DNSPort53 proxy listen=tcp:0.0.0.0:53 connect=tcp:localhost:53) and have been able to access port 80 within the container (the PiHole Admin GUI) but have been unable to successfully utilize port 53 within the container.
Does anyone know if this is even possible or is 53 “so baked in” to the host system that an request from outside the host to host:53 will be answered by host irrespective of LXD port forwarding rules?
port 53 has no sacred status, I use 2 Ubuntu 18.04 containers on 2 internet servers to manage an internet domain with bind.
I don’t use LXD proxy though, I have never taken the time to track how it’s done and if I don’t know how network magic is handled, I prefer to manage the network myself.
Ufw file 'before.rules'
# dns
-A PREROUTING -i $interface -p udp --dport 53 -j DNAT --to-destination $ipaddr
-A PREROUTING -i $interface -p tcp --dport 53 -j DNAT --to-destination $ipaddr
Ufw rules
sudo ufw route allow in on $interface to $ipaddr port 53 comment "from internet to DNS server"
sudo ufw route allow in on $interface to $ipaddr6 port 53 comment "from internet to DNS server"
sudo ufw route allow in on lxdbr0 out on $interface from $ipaddr to any port 53 comment "from DNS server to internet DNS servers"
sudo ufw route allow in on lxdbr0 out on $interface from $ipaddr6 to any port 53 comment "from DNS server to internet DNS servers"
The containers themselves are getting their internal address from the default LXD dnsmasq.
The 2 internet hosts are vanilla Ubuntu 16.04 with snap lxd and default systemd resolver.
all of this is working without any problem and the domain is green on all DNS checkers I tried it on.
Thank you for the reply and suggestion. I’ll be honest, I’m not sure whats going on here and therefore I do know how to implement.
If you would be so kind, could you explain:
How ufw knows to forward any knock on host:53 to PiHoleLXD:53? When I use lxdbr0, IP addresses are randomly assigned and not static
when I attempt enter the UFW rules you supply verbatim, I get the following error: ERROR: Bad destination address
I expected an error because to me the rule doesn’t tell UFW where to send the 53 request or where to listen for it (but I admit I don’t understand iptables as well as I should)
Thank you!
Oh, if it makes a difference, host is 19.04 server and containers are 18.04’s
the ‘forward’ from connections attempts on port 53 of the host is done through the prerouting rules in /etc/ufw/before.rules. The 4 following rules are to ensure that these exchanges are effectively authorized (these rules may not be necessary for you according to your default routing rules)
The rules I quoted are not to be taken as is, because your host interface and container IP address may be different of the ones I have on my system.
For the ufw rules you can set them in a script and add at the begining set interface=you_interface and so on. For the before.rules I’m afraid that you have to directly set the ip address and the interface name
And your container IP may be random with lxdbr0 bridge, but once it’s set it does not change again.
Unrelated question because I don’t use a pihole but why do you use sudo with the device proxy command … asking because on ubuntu I don’t and device proxy works ok.
First and foremost thank you for helping. I’ve done as you have suggested but have run into the following when trying to implement the new UFW “before” rules:
ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
ERROR: problem running ufw-init
iptables-restore: line 74 failed
Problem running '/etc/ufw/before.rules'
Err, I think I left out a few details.
It’s necessary to add a specific section for NAT in the before.rules file.
Now, I don’t understand why your don’t have one since LXD setup add one such section for masquerading bridged adresses
But if you really have to add one, it should be something like that:
# NAT table rules
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-F
# Port Forwardings
# dns
-A PREROUTING -i enp2s0 --p udp --dport 53 -j DNAT --to-destination 10.49.71.89:53
-A PREROUTING -i enp2s0 -p tcp --dport 53 -j DNAT --to-destination 10.49.71.89:53
# general
-A POSTROUTING -s 10.49.71.0/24 -o enp2s0 -j MASQUERADE
# do not remove the 'COMMIT' line or these nat table rules won't
# be processed
COMMIT
and the rest of the file for normal forwarding rules.
But it’s strange that default LXD setup has not set something like that already at the beginning of the before.rules file…
As noted below, do not use sudo with any of the lxc commands.
For me, if I see sudo lxc, it is an indication that things will go wrong.
To avoid sudo, you add your non-root user account into the lxc Unix group.
A stock LXD container is already using port 53 (domain) for the local DNS server.
If you want to create a proxy device, you need to figure out how to disable the built-in systemd-resolved.
An LXD proxy device uses a process for each proxy device. If you run ps, you will see a forkproxy process for each connection. Compared to iptables, with a proxy device you get the easiness and integration. When you delete the container, the proxy device is gone as well.
With iptables though, it is all happening in the kernel. You should notice performance difference if you pass lots and lots of traffic.
gpatel-fr if you would be so kind, I would like to revisit this topic because despite a lot of help, I just can’t get lxd’s proxy device to work so I would like to give iptables a try. I attempt to in the past but I believe a fundamental misunderstanding prevented me from successfully implementing your advice.
The router is going to hand-out IP addresses to all computers on the LAN including the Host [thought the IP is static for the Host] the Router will also tell all computers on the LAN to resolve DNS queries through the Host. Because the DNS resolver - PiHole - is in a LXD container and obtains network connectivity via lxdbr0, I need to tell the Host to forward all queries sent to it on Ports 53 & 4711 to the LXD PiHole container.
To Be Done on the Host:
Therefore, I need to set up iptable forwarding rules - ipv4 & ipv6 - on the host to send those requests to the LXD container, correct? If so, how do I do that?
Also, based on what Simos stated, I believe I need to disable systemd-resolved and NetworkManager
To Be Done on the LXD container:
??? Any iptables?
Thank you, I am very grateful for any help anyone can offer.
I missed your reply at first, was busy elsewhere.
I use ufw to manage my firewall, however for nat, unfortunately IMO, ufw dev refused to implement it in a friendly fashion and one have to write raw iptables rules so it don’t change much. So here is what I have in my /etc/ufw/before.rules on the host:
-A PREROUTING -i eth0 -p udp --dport 53 -j DNAT --to-destination 10.11.0.178
-A PREROUTING -i eth0 -p tcp --dport 53 -j DNAT --to-destination 10.11.0.178
eth0 is the name of the physical ethernet device on my host
10.11.0.178 is the IP address of my container hosting my bind instance (bind is the historical ancestor of all DNS servers)
If ufw (and not raw iptables) is used, this should be preceded by
before the rest of before.rules ufw file. But when lxd is used, this section is automatically added to
before.rules so that masquerading containers work, so it is never needed in my case.
What does it do ? it is making the network kernel redirect everything going in the eth0 interface on the 53 port with udp and tcp protocols going to the same port on the container. Importantly, it does nothing for anything running on the host, including other containers. For all these processes, the host internet card is never something they are sending to, it is only used to send to the outside (other computers on your internal network or the internet)
I have also rules for allowing in dns, since natting is not allowing; but it may or may not be necessary depending on default rules. As I don’t like depending on default rules I have set them explictly
sudo ufw route allow in on $interface to $ipaddr port 53 comment “Internet to dns”
sudo ufw route allow in on $interface to $ipaddr6 port 53 comment “Internet to dns”
sudo ufw route allow in on lxdbr0 out on $interface from $ipaddr to any port 53 comment “dns out to internet dns servers”
sudo ufw route allow in on lxdbr0 out on $interface from $ipaddr6 to any port 53 comment “dns out to internet dns servers”
That is translated by ufw to
(forward)
ACCEPT tcp – dns-ns1 anywhere tcp dpt:domain
ACCEPT udp – dns-ns1 anywhere udp dpt:domain
ACCEPT tcp – anywhere dns-ns1 tcp dpt:domain
ACCEPT udp – anywhere dns-ns1 udp dpt:domain
(input)
ACCEPT tcp – anywhere anywhere tcp dpt:domain
ACCEPT udp – anywhere anywhere udp dpt:domain
ACCEPT tcp – anywhere anywhere tcp dpt:domain
ACCEPT udp – anywhere anywhere udp dpt:domain
I can’t give you more about direct iptables implementation, I have used direct iptables ages ago and I soon switched to ufw and never looked back.
possibly. I am not going to reread all this thread. I usually try very hard to never disable OS stuff because it would make my life seemingly easier. These people managing my distro are smart, many of them are smarter than me and collectively they are vastly smarter than myself, if I disable something I am more likely to make things worse. On most of my computers there is no Network manager anyway, since it’s not installed on servers (no graphical interface)
no, nothing. I never touch iptables on containers (I don’t count fail2ban since it does everything all by itself). I manage internet access only on the host.
Thank you, gpatel-fr! I will attempt everything this evening (in about 12 hours or so).
One last question: notwithstanding the systemd-resolved / NetworkManager aspect of the equation, would the above instruction cause all DNS inquiries made TO the host (i.e. the router will be directing DNS inquiries to the host IP) to be forwarded on to the LXD container that will be running the DNS resolver (PiHole)?
You may be right I did not point this out since I don’t use lxd proxy, you are quite right that UDP is needed and it’s needed first, but tcp is needed too for proper dns handling.
PiHole uses both TCP & UDP but yes, that was something I tried as well. Thank you for pointing it out. Once I get a final solution I’ll clean up this tread or offer to write a how-to (because I assume there are others who would prefer running PiHole in a container other than docker)
I’ve had success with that setup before but - as I understand it - macvlan prohibits the host and container from interacting without some additional changes. Plus, I like the idea of the rest of the LAN having indirect access to the containers vs direct access
DNS (in general) uses both UDP and TCP. if a question (hardly ever) or answer (often enough to be very concerned) exceeds a certain size considered too large for UDP, (flagged in the UDP answer if only the answer is too large) the caching server (and supposedly the resolver library) will make a TCP connection and send the question that way (and get the answer to it back over the same connection). if you want to see it happen on your own domain zone, make a huge TXT record. i have seen at least one caching server have a configuration option to always use TCP. in cases where UDP won’t work but TCP will, this is a useful workaround (but an authoritative server for a public domain name really needs both).
You have been of great help and I am sure I have taken much of your time already, but I am afraid to report that I just can’t get this to work.
Here are my ufw rules:
To Action From
-- ------ ----
80/tcp ALLOW IN Anywhere # WAN to http
443/tcp ALLOW IN Anywhere # WAN to https
80/tcp (v6) ALLOW IN Anywhere (v6) # WAN to http
443/tcp (v6) ALLOW IN Anywhere (v6) # WAN to https
0.0.0.0/24 4711 on enp2s0 ALLOW FWD Anywhere on lxdbr0 # PiHole OUT FTL
0.0.0.0/24 53 on enp2s0 ALLOW FWD Anywhere on lxdbr0 # PiHole OUT DNS
10.96.103.252 4711 on lxdbr0 ALLOW FWD Anywhere on enp2s0 # PiHole IN FTL
10.96.103.252 53 on lxdbr0 ALLOW FWD Anywhere on enp2s0 # PiHole IN DNS
I have not tried disabling systemd-resolved because there are so many reported ways of doing it and I wouldn’t know how to ensure I wasn’t f’ing up LXD or my ability to resolve DNS queries.
My systemd is version 240
I’m running “Kubuntu” 19.04
NetworkManager is running but who knows how it interacts with systemd-resolved
Netplan just passes the buck to NetworkManager
Aside from the LXD containers (one of which utilizes the built in proxy to listen for and receive 80/443 requests sent to the host) and the UFW mods, this system is stock.
