Local image server and deltas

LXD
1

I’ve been using lxc on debian stretch and now buster for a while and have been exploring lxd installed via snap (3.0 channel). I think I have a good grasp of the basics and can create an image server and use it from client machines to download a custom image to start up containers.

Building an updated image and moving the alias over lets me pick up the new image on the client, but it seems to download the whole image each time. This is a unfortunate since in many cases the new image is almost identical to the previous one. The aim is to have nightly rebuilds picking up any updated debian packages so we can test them before putting the image into production.

I have seen some mentions of image deltas online but can’t seem to find any hard details on them. Is this something do-able, or do I just have to live with downloading the whole image each time?

TIA

2

LXD can handle image deltas during download but this is limited to simplestreams based servers.

The built-in image server in LXD itself (using the LXD protocol) doesn’t support generating delta images and it’s not obvious how we’d actually expose such deltas in the API at all.

3

Not saying it’s not something we could do, it’s certainly possible, but it would be quite a bit of work and would still involve manual intervention to tell LXD what deltas to generate, it would also then require the source image be kept around as the delta would be tied to it.

I suspect the changes would look something like:

  • Add a /1.0/images/FINGERPRINT/deltas API which can be used to generate, list and delete deltas from other images.
  • Require both the delta source and target images to be public images and prevent them from being marked private so long as deltas exist (as the binary delta would effectively leak information on the source image).
  • Support publishing LXD images as squashfs with a set of option compatible with xdelta3 and require the source and target images to both use squashfs.
  • Add the actual logic to drive xdelta3 for delta generation.
  • Add new Go API bindings and CLI options to drive the delta generation API.
  • Add download logic to the LXD server so when retrieving a remote public image, it attempts to fetch the list of deltas, compare to images in the local system image store (no cross cluster) and see if a delta can be used rather than full download.

As you can tell, this is quite a bit of work that would need to happen and the feature would have to be restricted to public images only to avoid security concerns with data leakage in deltas.

4

LXD can handle image deltas during download but this is limited to simplestreams based servers.

Which would explain why I’d seen some mention of it and then couldn’t find it in the docs.

…it would also then require the source image be kept around as the delta would be tied to it.

That makes sense.

Ah - I can see what the simplestreams stuff is doing here (incus/client/simplestreams_images.go at main · lxc/incus · GitHub). That makes sense.

// Look for deltas (requires xdelta3)
downloaded := false
_, err := exec.LookPath("xdelta3")
if err == nil && req.DeltaSourceRetriever != nil {
	for filename, file := range files {
		if !strings.HasPrefix(filename, "root.delta-") {
			continue
		}

		// Check if we have the source file for the delta
		srcFingerprint := strings.Split(filename, "root.delta-")[1]
		srcPath := req.DeltaSourceRetriever(srcFingerprint, "rootfs")

OK. Not sure I’ve got the time to investigate simplestreams at the moment. I shall just have to accept a slightly heavier download.

Thanks for the quick reply.