Possible bug mapping ro file into OCI container (Incus 6.17)

Incus
1

My homeassistant OCI is running fine with host networking (raw.lxc=“lxc.net.0.type=none”) but when I try to map host file /run/systemd/resolve/stub-resolv.conf into the container as /etc/resolv.conf in read-only ro mode, the container somehow manages to overwrite the mapped file.

I can see the mountpoint inside the container even after the overwrite, and if I mount it under /tmp/resolv.conf instead then it looks as it should, but I need it under /etc/resolv.conf. I’m thinking that allowing this to be overwritten may be a bug in Incus? Isn’t ro file mapping made for exactly for this scenario? I’m hesitant to open a bug report before running it by the community.

HERE IS MY ACTUAL USE-CASE:

The container uses host networking so let’s inspcect split DNS on the host:

$ resolvectl status br1 incusbr0
Link 4 (br1)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.0.1
       DNS Servers: 192.168.0.1 fd01:7439:2568::1
        DNS Domain: private.lan
     Default Route: yes

Link 5 (incusbr0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.210.158.1
       DNS Servers: 10.210.158.1
        DNS Domain: incus
     Default Route: yes

The above DNS servers are consolidated by systemd-resolved split DNS like so:

$ cat /etc/resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search incus private.lan

But OCI container homeassistant ignorantly requests its own DNS address for the host’s interface, and unfortunately OpenWrt complies, thereby gutting split DNS with this nonsense inside the container:

$ incus exec homeassistant -- cat /etc/resolv.conf
nameserver 192.168.0.1
domain private.lan

So I desperately try to restore proper operation by mapping resolv.conf from the host into the container:

$ incus config device add homeassistant resolvconf disk source=/run/systemd/resolve/stub-resolv.conf path=/etc/resolv.conf readonly=true

But in spite of the above being mapped in as ReAd OnLy, the container fiendishly manages to overwrite it with the hapless DNS option it misappropriated via its rogue DHCP request, thereby instantly ruining split DNS name resolution inside the container and rendering my life entirely worthless. This would all be oh-so-simple if I just had the bigger hammer that ro might have been…

P.S. I created the container like this:

$ incus create ghcr:home-assistant/home-assistant:stable homeassistant \
--no-profiles \
--storage default \
-c raw.lxc="lxc.net.0.type=none" \
-c security.privileged=true \
-c boot.autostart=true \
-c boot.autorestart=true \
-c environment.TZ=America/Vancouver

$ incus config device add homeassistant dbus disk source=/run/dbus path=/run/dbus readonly=true

$ incus storage volume create default ha_config
$ incus storage volume set default ha_config security.shifted=true
$ incus storage volume attach default ha_config homeassistant /config
2

Might be mistaken but the instance runs “privileged” which means it has full root access? Systemd might move the original (read-only) file away and writes it’s own version instead.

Might be a an idea to test this with a simple OCI container using the same base OS image. Start it unprivileged (privileged=false) and see what happens.

3

I suspect the read-only flag works just fine, but your mount is simply not used because Incus’ OCI containers run their DHCP client and that DHCP client works using its own set of mounts for /etc/resolv.conf and /etc/hostname, so your read-only mount just ends up being over-mounted.

You could probably look at cat /proc/self/mountinfo inside the container to see what’s going on with resolv.conf. I’m guessing you’ll find two separate entries for it, one being your read-only entry from the disk device and one being the one used to configure networking in OCI containers.

4

ooooh YES. I was wondering why that looked so strange and now I know. Thanks.

5

Requesting suggestions on how best to override Incus’ built-in management of the /etc/resolv.conf file inside an OCI container that uses host-networking via raw.lxc=“lxc.net.0.type=none”. The implication of using the various CLI commands is not clear to me in the host-networking context, since they seem intended at containerized (separate) network namespace management.

6

After many hours of work on this, I’m realizing that any container running in host-networking mode should be inheriting /etc/resolv.conf directly from the host (split DNS stub), so therefore Incus should not be injecting the rogue host file that I am seeing. This is looking like a true bug to me now. Requesting comments.

7

There is no such thing as host networking mode in Incus.

You’re using raw.lxc to go behind Incus’ back and make it happen, so it’s not surprising that Incus won’t go out of its way to parse that raw config and try to guess what it should or shouldn’t do :slight_smile:

Maybe try using more raw.lxc stuff to get yourself out of this, like adding something like lxc.mount.entry=/etc/resolv.conf etc/resolv.conf bind defaults,ro 0 0 or something along those lines to try to pile up yet another resolv.conf on top of the existing two.

1 Like
8

:face_savoring_food: I’ve got a fully functioning Home Assistant with Matter over WiFi and Thread, all working reliably in “host networking” mode under Incus (alongside a bunch of other VM’s and stuff). That’s fantastic and I appreciate having had the opportunity to present this use-case and its quirks. Many thanks for Incus and the tips.

1 Like