Yeah, memory limits especially can have a pretty funny behavior.
When you get close to hitting them (whether artificially like through a resource limit, or physically on a machine), the kernel starts working very hard to not have you run out of memory. That means it starts asking all processes to release anything they don’t need, it will flush the page cache, …
This then results in extremely slow I/O as there’s no more page cache and all filesystem access must directly access the disk. And if you’re so low on memory that you can’t even allocate enough memory for a new process entry, then you end up with a complete hang whenever trying to spawn a new task.
Increasing the memory limit will normally immediately unblock everything.
Worth noting that Incus always sets a soft limit at 90% of the hard limit. This is used to have the kernel begin reclaiming memory a bit earlier and avoid those kind of effects in most cases. But if you have runaway processes that just eat whatever memory they can, the soft limit only slightly delays the inevitable.
That said, the limits should still do their job, that is, protect the host and the other instances.
Those should all keep running exactly as normal.