Just in case anyone else has the same or a similar problem; Nginx Proxy Manager (in particular) uses an init system called “s6” which seems to be popular with lightweight docker images, but it’s missing some key tools like “ip” and “ping”, which make setting up a second interface a little tricky. This is what I’ve ended up with for now, it’s a bit of a kludge, but it survives instance restarts and and docker image rebuilds, so for now it’s “a” way to work with stock docker images.
This will not work “as-is” for you, you will need to tweak it
First you need an instance and we’re assuming that it has a persistent “/data” attachment. Then you need to assign it a static IP address for each of your interfaces, the first (eth0) needs to be on your host bridge network, the second on your OVN network. (otherwise you won’t be able to do a proxy with nat bound to the host, which means you won’t see real client IP’s in the container, which means you won’t have any IP based access control or useful logging …)
# Assume "private" is attached to your OVN network range on 10.103.0.0/24
# Assume "public" is attached to your host network bridge on 10.100.0.100/24
incus config device add (instance) eth0 nic network=private ipv4.address=10.103.0.100 name=eth0
incus config device add (instance) eth1 nic network=public ipv4.address=10.100.0.100 name=eth1
The following script should;
- blindly detach and delete volume “(instance)-cont-init.d”
- (re)create that same volume
- create a 10-config-eth1.sh script and copy it into this volume
- copy binaries for “ip” and “ping” to the instance’s /data/s6 folder
- copy missing shared libraries to the instance’s /data/s6 folder
Making a blind assumption that the host distro is the same as the container distro, if not, you may need to tweak the ip and ping binaries and associated shared libraries
make_eth1.sh
#!/usr/bin/env bash
#
ADDRESS="10.103.0.100/24"
GATEWAY="10.103.0.1"
ROUTING="10.4.0.0/24"
INSTANCE="npm"
#
incus storage volume detach default ${INSTANCE}-cont-init.d ${INSTANCE} > /dev/null 2>&1
incus storage volume delete default ${INSTANCE}-cont-init.d > /dev/null 2>&1
#
incus storage volume create default ${INSTANCE}-cont-init.d
incus storage volume attach default ${INSTANCE}-cont-init.d ${INSTANCE} /etc/cont-init.d
#
cat - > 10-config-eth1.sh << EOF
#!/usr/bin/env bash
#
# Script for s6-init based systems to initialise
# a second interface with a static IP address
#
IP="/data/s6/bin/ip"
export LD_LIBRARY_PATH=/data/s6/lib
echo "\$(date) Attempting to configure eth1 with static IP $ADDRESS"
\${IP} link set eth1 up
\${IP} addr add "${ADDRESS}" dev eth1
if [ -n "${GATEWAY}" ]; then
echo "\$(date) - Adding route ${ROUTING} via ${GATEWAY} on eth1"
\${IP} route add ${ROUTING} via ${GATEWAY} dev eth1
fi
echo "\$(date) Static routing complete"
EOF
chmod +x 10-config-eth1.sh
incus file push 10-config-eth1.sh ${INSTANCE}/etc/cont-init.d/
incus file create --type=directory ${INSTANCE}/data/s6
incus file create --type=directory ${INSTANCE}/data/s6/bin
incus file create --type=directory ${INSTANCE}/data/s6/lib
incus file push /usr/sbin/ip ${INSTANCE}/data/s6/bin/ip
incus file push /usr/bin/ping ${INSTANCE}/data/s6/bin/ping
incus file push /lib/aarch64-linux-gnu/libbpf.so.1 ${INSTANCE}/data/s6/lib/libbpf.so.1
incus file push /lib/aarch64-linux-gnu/libelf.so.1 ${INSTANCE}/data/s6/lib/libelf.so.1
incus file push /lib/aarch64-linux-gnu/libmnl.so.0 ${INSTANCE}/data/s6/lib/libmnl.so.0
Then on the host;
bash make_eth1.sh
When you restart the container, the s6 init system should see this new script and run it “first” as a part of the container boot sequence. The script should use the “ip” binary to add the specified address to “eth1” along with the specified OVN “routing” address. (so you’ll get a default route on the host bridge and a route to your OVN network on eth1)
This relies on s6 apparently looking in /etc/cont-init.d for startup scripts, which is appears to do even for containers that don’t actually come with an /etc/cont-init.d directory. We’re also assuming that nothing else is using this folder, if it is, again a tweak will be needed. One option would be to use a file path and bind to the actual file rather than the folder.
It seems like a lot of hoop jumping, but from a long-term maintnance perspective using docker images seems like the way to go, and the OCI setup just has a nice consistent Incus feel to it.
What I’d REALLY like to do is push the init script into /data, scrap the conf-init.d volume, and instead be able to map a symlink from /etc/conf-init.d/10-config-eth1.sh to /data/10-config-eth1.sh , that would be a neater / more efficient solution. Not sure how doable virtual symlinks are tho’ … 
)