When I began with Incus, I struggled with this topic, and I never liked bridged routing or macvlan. I want a dedicated connection to my router, allowing me to release a DHCP lease or set its own proper IP address within the range. At some point, I got it working and just left it. Recently decided to restructure my disks and general incus setup, making better use of modern filesystem ZFS and its replication features instead of rsync backups. In this process I had two network challenges.
I was able to move my data around and set up a ZFS backup pool, creating temporary storage to move my instances. To import them on a clean installation. After everything was moved, my instances suddenly stopped connecting; weirdly enough, I was unable to find the root cause. Reconfigured my instance profile again, and all was back.
Then, on the clean install, I was unable to recreate my networking setup. Found an old topic of mine with the missing link, which I also used in the previous step. Only works on the command line. Now I’m trying to grasp what I’m doing below, why it works as I want, and why this cannot be set up differently or through UI.
incus@incus:~$ incus network list ±-----±---------±--------±-----±-----±------------±--------±------+ | NAME | TYPE | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY | STATE | ±-----±---------±--------±-----±-----±------------±--------±------+ | eno1 | physical | NO | | | | 0 | | ±-----±---------±--------±-----±-----±------------±--------±------+ | lo | loopback | NO | | | | 0 | | ±-----±---------±--------±-----±-----±------------±--------±------+ incus@incus:~$ incus network attach-profile eno1 default
Looks like this is now behaving like a macvlan. My host is unable to use an instance as DNS (adguardhome). Weirdly, I’m 100% sure I had it set up as such previously with the same trick.
Thanks or the reply. I understand the limitations of the macvlan, it is not what is happening here. Almost feels like something bugged. So, mostly here trying to understand how Incus allows manual attachment of the profile, kinda forcing the network onto the instances over the profile.
What extra info is needed? Happy to provide, want to understand this one, feels wrong.
Fresh debian trixie installation
Fresh incus stable (zabby) installation
Did a default incus admin init, default dir storage, no bridge network setup
Imported my zpool storage that includes all my instances
incus recovery to obtain them
The instances where already configured with network inherited from default profile
Default profile only has my eno1 debian nic.
Your not allowed to attach eno1 to an instance. Using attach-profile, you can do the same that you should not be able to do on the default profile; it gets pushed to all instances, and it works. Feels incorrect and trying to understand what is happening.
You certainly can overwrite the network device on a single instance. It is actually required if you want to provide a fixed IP but also supports changing the nic type to macvlan.
Yes, changing it on the profile is global to all instances but it can be also applied to individual instances. Done it multiple times…
Don’t think this is the real problem here it is more about how it has been configured in the old setup? Think it would have been great to see the old configuration as it would make it simpler to figure out what the differences are. As we don’t have it, there is properly no way to get the right answer as @catfish said.
When I apply it to a single instance, it completely takes over the host NIC, leaving the host unreachable. What is the attach-profile host nic result behaviour here then? Must be some kind of macvlan that is then created, unable to reach my instances from the host. If that is the case, it might be better not to allow this and require a network setup.
(1) If you really want a DEDICATED NIC for your system you will need a separate physical NIC for each of those - I doubt this is really what you are looking for. Having said that there are for sure usacases for this, I am i.e. running my pfSense routers with dedicated INTEL GbE NICs (on a separate PCIe). I pass them to my pfSense VMs simply using
incus config device add pfsense igb nic nictype=physical parent=enp1s0f0
(2) I know you stated you never liked bridges - but probably the easiest way to give the impression of a network interface directly connected to the router, allowing for DHCP, or static IP, etc. is a bridge. If you are using this for Incus and other stuff you might want to just setup a static bridge on OS level, for debian this would mean your /etc/network/interfaces looks something like this:
auto lo
iface lo inet loopback
auto br0
iface br0 inet static
address 192.168.1.250/24
gateway 192.168.1.1
bridge_ports enp1s0
bridge_stp off
iface enp1s0 inet manual
You do not assign your host IP to the physical interface (enp1s0 in this case), but to the bridge. Same if you would do DHCP for the host, DHCP config/client would be on bridge. Now you have a bridge you can use for your containers/VMs to attach a NIC to, something like:
incus config device add notpfsense eth0 nic nictype=bridged parent=br0
(reboot or systemctl restart networking after changing the file - and you might want to install package bridge-utils now since you have a bridge)
(3) There is probably a way to create a managed bridge on top of your network interface from Incus - but I’m using IncusOS and every interface is a bridge by default there so don’t know / never tried.
You literally passthrough eno1 to a instance, it’s like you pull out host’s network cable and plug it into the instance, that why your host can’t reach any instance.
And you only have one NIC(eno1), you either use bridge network or macvlan, or plug in a usb NIC and passthrough to the instance.
If you need help to setup bridge or macvlan, we can help.
And incus network attach and incus config device add instance devicename nic use the same api.
Thanks for the extensive replies, both! Much appreciated. I just noticed that the UI was omitting showing the nic type, so I actually assumed I attached my eno1 to all instances and had them working as such. Looking at the profile yaml, that’s not actually the case; it’s indeed a macvlan called eno1 with eno1 as its parent. You’re correct, charging that to physically will jam the plug into one instance, losing the host. Using the UI, I got the impression it copied my eno1 physical, since the UI does not state the type, but the YAML does.
My question is solved. Going to create a bridge host and try that.
This. It’s exactly what I was looking for. My understanding of the bridge was only the incus-managed version, where I needed to set up routing on my network.
This is perfect and exactly what I’m looking for. Thank you.
