This concerns a Debian 12 LXD install with a Debian 12 x86 container image (both cloud-init and standard). There are no issues with the network in the container until the DNS server Unbound is configured on it in a fairly standard installation.
It is probably to do with a conflict between some hidden dns server somewhere under the hood in systemd/lxd and the new server I want to set up. I am aware that lxd contains dnsmasq functionality and to circumvent this I configured lxd to run on a new unmanaged bridge, avoiding lxcbr0.
The contents of /etc/dnsmasq.d/lxd:
bind-interfaces
except-interface=lxcbr0
replacing this with except-interface=br1 (the name of the bridge in use) breaks dns for the host system and all other containers even if they point to an external dns service.
The relevant lines in ‘journalctl’ are…
unbound[151]: [1704993386] unbound[151:0] error: can't bind socket: Cannot assign requested address for 198.52.44.22 port 53
unbound[151]: [1704993386] unbound[151:0] fatal error: could not open ports
..
Started unbound-resolvconf.service - Unbound asyncronous resolvconf update helper.
resolvconf[160]: Dropped protocol specifier '.unbound' from 'lo.unbound'. Using 'lo' (ifindex=1).
resolvconf[160]: No DNS servers specified, refusing operation.
So neither the network nor the unbound services come up properly. The behaviour is ‘erratic’ in the sense that when I manually restart the network and unbound services in the host system they come up properly. If I restart the container from the host system things come up properly as well, but if the host system is rebooted then they don’t.
Any help much appreciated. Just migrated to incus which didn’t resolve the issue.
unbound starting before networkd has properly set the IP address, failing to listen due to that
unbound conflicting with systemd-resolved
Given you’re trying to bind a particular address, it shouldn’t be the second one as the second is mostly an issue when you trying to bind port 53 on all addresses.
Unbound does support ip-freebind which allows for binding addresses that do not yet exist. ip-transparent sounds pretty similar too. Those two options may be worth a try.
Thank you. I tried that and it helps to get the unbound service going, but then it still fails on dns-container resolvconf[158]: No DNS servers specified, refusing operation. and unbound-resolvconf.service .
Other network dependent services are also failing - see a bigger part of the log is at the bottom of this post. So it looks to me like the underlying problem is that the network fails to initialize at an early stage. The output from ‘service systemd-networkd status’ is :
So the network service is ‘inactive (dead)’. There is nothing wrong with the network config itself - which works reliably in other situations.
Below is output from ‘journalctl’:
Jan 11 22:44:41 dns-container systemd-resolved[134]: Positive Trust Anchors:
Jan 11 22:44:41 dns-container systemd-resolved[134]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jan 11 22:44:41 dns-container systemd-resolved[134]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jan 11 22:44:42 dns-container systemd-resolved[134]: Using system hostname 'dns-container'.
Jan 11 22:44:42 dns-container systemd[1]: Started systemd-resolved.service - Network Name Resolution.
Jan 11 22:44:42 dns-container systemd[1]: Reached target sysinit.target - System Initialization.
Jan 11 22:44:42 dns-container systemd[1]: Started apt-daily.timer - Daily apt download activities.
Jan 11 22:44:42 dns-container systemd[1]: Started apt-daily-upgrade.timer - Daily apt upgrade and clean activities.
Jan 11 22:44:42 dns-container systemd[1]: Started dpkg-db-backup.timer - Daily dpkg database backup timer.
Jan 11 22:44:42 dns-container systemd[1]: Started e2scrub_all.timer - Periodic ext4 Online Metadata Check for All Filesystems.
Jan 11 22:44:42 dns-container systemd[1]: Started exim4-base.timer - Daily exim4-base housekeeping.
Jan 11 22:44:42 dns-container systemd[1]: fstrim.timer - Discard unused blocks once a week was skipped because of an unmet condition check (ConditionVirtualization=!container).
Jan 11 22:44:42 dns-container systemd[1]: Started systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories.
Jan 11 22:44:42 dns-container systemd[1]: Reached target timers.target - Timer Units.
Jan 11 22:44:42 dns-container systemd[1]: Listening on dbus.socket - D-Bus System Message Bus Socket.
Jan 11 22:44:42 dns-container systemd[1]: Reached target sockets.target - Socket Units.
Jan 11 22:44:42 dns-container systemd[1]: systemd-pcrphase-sysinit.service - TPM2 PCR Barrier (Initialization) was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f).
Jan 11 22:44:42 dns-container systemd[1]: Reached target basic.target - Basic System.
Jan 11 22:44:42 dns-container systemd[1]: Started cron.service - Regular background program processing daemon.
Jan 11 22:44:42 dns-container systemd[1]: Starting dbus.service - D-Bus System Message Bus...
Jan 11 22:44:42 dns-container systemd[1]: Starting e2scrub_reap.service - Remove Stale Online ext4 Metadata Check Snapshots...
Jan 11 22:44:42 dns-container systemd[1]: getty-static.service - getty on tty2-tty6 if dbus and logind are not available was skipped because of an unmet condition check (ConditionPathExists=!/usr/bin/dbus-daemon).
Jan 11 22:44:42 dns-container (crub_all)[138]: e2scrub_reap.service: Failed to set up network namespacing: Permission denied
Jan 11 22:44:42 dns-container (crub_all)[138]: e2scrub_reap.service: Failed at step NETWORK spawning /sbin/e2scrub_all: Permission denied
Jan 11 22:44:42 dns-container cron[136]: (CRON) INFO (pidfile fd = 3)
Jan 11 22:44:42 dns-container systemd[1]: Starting snmpd.service - Simple Network Management Protocol (SNMP) Daemon....
Jan 11 22:44:42 dns-container systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
Jan 11 22:44:42 dns-container systemd[1]: Starting systemd-logind.service - User Login Management...
Jan 11 22:44:42 dns-container systemd[1]: systemd-pcrphase.service - TPM2 PCR Barrier (User) was skipped because of an unmet condition check (ConditionPathExists=/sys/firmware/efi/efivars/StubPcrKernelImage-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f).
Jan 11 22:44:42 dns-container systemd[1]: Starting systemd-user-sessions.service - Permit User Sessions...
Jan 11 22:44:42 dns-container systemd[1]: Starting unbound.service - Unbound DNS server...
Jan 11 22:44:42 dns-container systemd[1]: e2scrub_reap.service: Main process exited, code=exited, status=225/NETWORK
Jan 11 22:44:42 dns-container systemd[1]: e2scrub_reap.service: Failed with result 'exit-code'.
Jan 11 22:44:42 dns-container systemd[1]: Failed to start e2scrub_reap.service - Remove Stale Online ext4 Metadata Check Snapshots.
Jan 11 22:44:42 dns-container systemd[1]: Finished systemd-user-sessions.service - Permit User Sessions.
Jan 11 22:44:42 dns-container systemd[1]: Started console-getty.service - Console Getty.
Jan 11 22:44:42 dns-container systemd[1]: getty@tty1.service - Getty on tty1 was skipped because of an unmet condition check (ConditionPathExists=/dev/tty0).
Jan 11 22:44:42 dns-container systemd[1]: Reached target getty.target - Login Prompts.
Jan 11 22:44:42 dns-container cron[136]: (CRON) INFO (Running @reboot jobs)
Jan 11 22:44:43 dns-container systemd-logind[141]: New seat seat0.
Jan 11 22:44:43 dns-container systemd[1]: Started dbus.service - D-Bus System Message Bus.
Jan 11 22:44:43 dns-container systemd[1]: Started systemd-logind.service - User Login Management.
Jan 11 22:44:43 dns-container dbus-daemon[137]: [system] Successfully activated service 'org.freedesktop.systemd1'
Jan 11 22:44:43 dns-container systemd[1]: Started unattended-upgrades.service - Unattended Upgrades Shutdown.
Jan 11 22:44:43 dns-container sshd[152]: Server listening on 0.0.0.0 port 22.
Jan 11 22:44:43 dns-container sshd[152]: Server listening on :: port 22.
Jan 11 22:44:43 dns-container systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
Jan 11 22:44:44 dns-container unbound[151]: [151:0] notice: init module 0: subnetcache
Jan 11 22:44:44 dns-container unbound[151]: [151:0] notice: init module 1: validator
Jan 11 22:44:44 dns-container unbound[151]: [151:0] notice: init module 2: iterator
Jan 11 22:44:44 dns-container unbound[151]: [151:0] info: start of service (unbound 1.17.1).
Jan 11 22:44:44 dns-container systemd[1]: Started unbound.service - Unbound DNS server.
Jan 11 22:44:44 dns-container systemd[1]: Reached target nss-lookup.target - Host and Network Name Lookups.
Jan 11 22:44:44 dns-container systemd[1]: Starting exim4.service - LSB: exim Mail Transport Agent...
Jan 11 22:44:44 dns-container systemd[1]: Started unbound-resolvconf.service - Unbound asyncronous resolvconf update helper.
Jan 11 22:44:44 dns-container snmpd[139]: Error opening specified endpoint "10.10.10.10"
Jan 11 22:44:44 dns-container snmpd[139]: Server Exiting with code 1
Jan 11 22:44:44 dns-container resolvconf[158]: Dropped protocol specifier '.unbound' from 'lo.unbound'. Using 'lo' (ifindex=1).
Jan 11 22:44:44 dns-container resolvconf[158]: No DNS servers specified, refusing operation.
Jan 11 22:44:44 dns-container systemd[1]: unbound-resolvconf.service: Main process exited, code=exited, status=1/FAILURE
Jan 11 22:44:44 dns-container systemd[1]: unbound-resolvconf.service: Failed with result 'exit-code'.
Jan 11 22:44:44 dns-container systemd[1]: snmpd.service: Main process exited, code=exited, status=1/FAILURE
Jan 11 22:44:44 dns-container systemd[1]: snmpd.service: Failed with result 'exit-code'.
Jan 11 22:44:44 dns-container systemd[1]: Failed to start snmpd.service - Simple Network Management Protocol (SNMP) Daemon..
Jan 11 22:44:47 dns-container exim4[153]: Starting MTA: exim4.
Jan 11 22:44:47 dns-container systemd[1]: Started exim4.service - LSB: exim Mail Transport Agent.
Jan 11 22:44:47 dns-container systemd[1]: Reached target multi-user.target - Multi-User System.
Jan 11 22:44:47 dns-container systemd[1]: Reached target graphical.target - Graphical Interface.
Jan 11 22:44:47 dns-container systemd[1]: Starting systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP...
Jan 11 22:44:47 dns-container systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully.
Jan 11 22:44:47 dns-container systemd[1]: Finished systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP.
Jan 11 22:44:47 dns-container systemd[1]: Startup finished in 13.161s.
Going for the second possibility you mention, that unbound conflicts with systemd-resolved, I removed the latter in favour of the resolvconf package.
Now unbound starts properly and the error messages in journalctl shown in the original post are gone, but the network remains down. There are no error messages in journalctl regarding this.
– edit –
When I used an Alpine linux container and set up Unbound in it, it worked. So the problem is systemd specific.
What I suspect is happening is that the debian lxc images have more systemd in them than the usual debian install and some of that systemd stuff breaks some packages. Probably, if I created a debian container image from the Debian source, which still uses /etc/network/interfaces, it would also work.
Probably not. It’s some kind of incompatibility between the Debian unbound packaging and the systemd-networkd+systemd-resolved setup, not something we can really do too much about in LXC or even in our images.
We’re not planning on switching the images to ifupdown as systemd-networkd allows us to have the same network configuration on a wide variety of distributions and it’s a supported option on Debian (alongside ifupdown, Network-Manager and others).
If you want to see this kind of thing work better out of the box, it may be worth trying to report this as a bug against the unbound package in Debian where they’ll most likely pull in the packagers of resolvconf and systemd to see if something can be done to get them all to play nice.
I did a test with a fresh Debian 12 install on a desktop; systemd-networkd, systemd-resolved and Unbound. This works fine with minimal configuration. All services come up as usual and ip addresses are correctly assigned to the interfaces where configured.
So it’s not a bug for the Debian maintainers and must be something Incus/LXC/LXD specific.