So quick question as im sure it pertains to others who use this infrastructure… i use network sub-functions … which are essentially ultra light sriov Virtual functions. Unlike traditional Virtual Functions , Sub-functions (or sometimes referred to scalable functions) allow the user to live migrate , as long as its properly setup using the switchdev mode of a capable interface. So far i’ve setup my own networks as bridges to my sub-functions (with asap^2 offloading) but i would love to be able to have incus do something similair as it does with its SR-IOV option for networks (as in it takes automatically from a parent interface and sets up as needed).
Would i just use SRIOV as the network for this to work or would incus then create virtual functions instead of using the sub-functions(or scalable functions)?
Surprisingly this works well with using them as bridges… im currently writing up a presentation on doing that and accelerating the network storage by using RDMA. I am still curious though @stgraber , do we treat SF’s as physical connections ? Or is some other mechanism that can auto allocate them in incus ?
We only support the representor logic with vDPA and OVN offloaded networks, though even that isn’t logic that’s particularly used or tested at this point.
At the time the scalable function stuff wasn’t mainline yet (looks like it is now though the NVIDIA docs still only refer to the old implementation) and they also had a bunch of limitations as far as their use outside of VMs (but now I’m seeing some mentions of use in containers).
Anyway, if they work properly on your system, you can hack things together by setting one up and passing the guest side interface (for a container) or VF (for a VM) with the representor being bridged into a correctly configured OVS bridge and things “should” work.
More integrated support with Incus would be interesting and may be something that @bensmrs would have some interest in looking into once he’s sorted out the SR-IOV + net_failover path to live migration, effectively offering vDPA outside of OVN as another alternative, then backing that either with regular VFs or SFs where supported.
Yeah, I think in order of complexity for your setup, we’d be looking at:
sr-iov + net_failover
vdpa + regular sr-iov vf
vdpa + scalable sr-iov vf
They’d all provide the same ability of getting hardware accelerated networking which allows for live migration. The switch to vdpa allows for a single PCI device in the guest, then the switch to scalable VF would allow for more VFs and less overhead on the card but would otherwise be transparent to the guest.
Basically all fitting in the “full hardware offload with live-migration” topic.
