Ok so this is a little long in an attempt to get the best advice but my main question is clearly marked at the bottom with “question:”.
Background:
I’ve been building a multi tenant cloud platform over the last two years. Incus provides me with a “cloud per user” and my platform then integrates remote object storage, cert management, and reverse proxy automation on top. Most recently, I’ve spent a lot of time testing different ways to sync containers to a users remote storage bucket with speed and performance being the primary objective. I have this sync happening automatically every time a container for any user is stopped - So I’m finding myself having to engineer things outside the defaults to make this work.
Issue with default:
If I use “incus export” by itself for containers, it seems to use a full CPU core and goes surprisingly slow at only a few MB per second. With multiple users on a single host, this is not ideal.
Observation:
If I export an incus container with the –no-compression flag, and then gzip the created directory, I seem to get the same exact end result on a fraction of the time. The only noticable difference being that instead of streaming directly into a compressed archive, I require local disk space for the non compressed directory before compressing it.
Question:
Since for my application, the convenience of incus export alone streaming directly into an archive is far outweighed by doing it without compression first and then compressing after - is there something I’m missing in regards to why incus doesn’t do it this way natively? Should I be getting faster exports than 4-5mb/sec using incus export default? Because I get 10x that when using no compression during export, and then zipping the resulting directory only takes a few seconds. It’s the difference of minutes vs seconds for what seems to be exact same end result.
Incus effectively gets a streaming tarball for the container, then pipes that over to the compressor (gzip in your case) and out the network to you.
One difference there compared to what you’re doing is that Incus doesn’t have access to the entire data, instead it can only feed chunks of 4MB of data at a time to gzip (to avoid excessive memory usage). That likely makes the compression quite inefficient but has the benefit of not eating all the server memory or disk space by either reading massive chunks into memory or having to put the entire thing down to disk before compressing it.
Ok that’s actually what I assumed the thought process was, so thank you for clarifying! I think In my case since exports are going to be quite frequent, having bursty high memory/disk usage is better than longer running CPU utilization for every export.
Is that 4MB chunk size something I could adjust for further testing though?
Not without rebuilding Incus. It’s our internal data copy function which runs using 4MiB chunks with CopyN everywhere rather than using io.Copy which can lead to large memory usage.