I am looking to better understand if it is possible use multiple NICs with an OCI instance.
My goal is to to run a OCI reverse proxy in front of multiple instances from distinct networks. The idea here is that unrelated instances can be on separate managed networks, but the reverse proxy can connect to all networks and access instances on them, ideally using Incus’s DNS resolution.
I believe that similar isolation can be achieved with firewall configuration and/or that I can manually configure multiple NICs, Subnets, DNS, using LXC + Cloud-Init instead of OCI or by using macvlan and managing networking almost entirely outside of Incus. However I’d rather not move away from managed bridge networks if possible to avoid added complexity.
This seems to imply that multiple DHCP clients should work across multiple NICs, but in my experience only the “eth0” NIC appears to get an IPv4 address from it’s managed network. Other configured NICs seem to only have a self-assigned IPv6 address.
With the desired setup I described I suppose I’d also need a way to specify a default gateway (or at least expect a reasonable default like “eth0”), but otherwise the hope would be that DNS would be automatically configured to resolve each managed network based on it’s configured dns.domain.
Thank you for sharing. I’m very familiar with docker compose, but unfortunately the compose model doesn’t really meet my needs in this particular case. That said, I’ll take a look at the source and try to figure out how you manage multiple networks like this and see if I can extrapolate how to do so manually for the issue I am facing.
I would still love to understand how incus OCI is expected to behave with multiple managed NICs and/or how to adjust configuration for better default handling.
@davek I’m sorry I didn’t read you’r OP trough and trough, incus-compose isn’t doing what you want, yet as it doesn’t handle internal / as in firewalled bridges yet.
About DNS for managed bridges I found an easy way to configure additional hosts raw.dnsmasq on the network like:
I haven’t had much time to dedicate to investigation here, though I imagine raw.dnsmasq and external firewalld (or similar firewall) could help with the DNS and isolation goals I outlined. It certainly doesn’t maintain the simplicity of default managed bridge networks, but perhaps the simplicity I am looking for isn’t attainable.
If I understand it correctly, it appears that multiple DHCP clients should work across multiple NICs for OCI, but in my experience only the “eth0” NIC appears to get an IPv4 address from it’s managed network. Other configured still NICs seem to only have a self-assigned IPv6 address.