Simplestreams image server

[kamzar1 (Dauerurlaub)]

I recently have been updating code for private image server.
There are numerous changes in upstream “https://uk.lxd.images.canonical.com” , with which I try to catch up.

1. There are root.squashfs and root.tar.xz files.
   Which is by default fetched when creating a container? Is there an optional choice?
2. There are multiple versions of item sets maintained, Is this is an insurance policy?
   I guess, by launching a container, the latest version is fetched.
   In order to save space and migration, I maintain only one version. In this case, can I skip the createion of ‘root.date…vcdiff’: ftype:squashfs.vcdiff diff delta?
3. lxc image list has a field “type” : container/virtual-machine. Is this based on whether “disk.qcow2” is given in i.versions.tems []l?
4. Is there a way to publish a unified tarball instead of split (meta+root)?
   how would that unified item look like? Compared to this split context:
   “items”: {
   “lxd.tar.xz”: {
   “ftype”: “lxd.tar.xz”,
   “sha256”: “fc”,
   “size”: 1356,
   “path”: “images/almalinux/8/amd64/cloud/20220209_23:09/lxd.tar.xz”,
   “combined_sha256”: “ed”,
   “combined_rootxz_sha256”: “ed”,
   “combined_squashfs_sha256”: “b3”,
   “combined_disk-kvm-img_sha256”: “bc”
   },
   “root.tar.xz”: {
   “ftype”: “root.tar.xz”,
   “sha256”: “d3128e914e2a4c8f4539f3fdb5a8c4b802a132c9c9fb8c53fa128476419d3d58”,
   “size”: 128108152,
   “path”: “images/almalinux/8/amd64/cloud/20220209_23:09/rootfs.tar.xz”
   },
   “root.squashfs”: {
   “ftype”: “squashfs”,
   “sha256”: “eaf930fa7c057fdd737272f23f6346cec61b3b6eb1d7198db44152728523c938”,
   “size”: 153804800,
   “path”: “images/almalinux/8/amd64/cloud/20220209_23:09/rootfs.squashfs”
   },

Generally maintaining an image server consumes a lot of resources and time, I dont run all images, but certain os are chosen with extra options to create hundreds ready applications.
The fresh fetching os and creation of those applications is taking hours/day with distrobuilder.
Now sha256 once single once combined is quite challenge. I used to assign attr to those tarballs or get the timestamp of the files, in order to prevent a nonsense re-calculating of sha256s when only selected images where newly created.
Also selectively reduce the images.json inplace by selecting “amd64” or "default’ or by the lastest set of items.

[stgraber (Stéphane Graber)]

@kamzar1

There are root.squashfs and root.tar.xz files.
Which is by default fetched when creating a container? Is there an optional choice?

LXD supports squashfs, xz and gz and will attempt to use them in that order.
It detects their presence by looking at the LXD hash properties.

@kamzar1

There are multiple versions of item sets maintained, Is this is an insurance policy?
I guess, by launching a container, the latest version is fetched.
In order to save space and migration, I maintain only one version. In this case, can I skip the createion of ‘root.date…vcdiff’: ftype:squashfs.vcdiff diff delta?

Right, we keep the past 3 images which can be useful when an issue happens in a newer image and lets us compute delta (vcdiff) to reduce the overall bandwidth use as clients download the refreshed images. If you don’t care about that, then having a single image is fine.

@kamzar1

lxc image list has a field “type” : container/virtual-machine. Is this based on whether “disk.qcow2” is given in i.versions.tems []l?

Yes, if we have a qcow2 image and matching hash key, then we list the image as available for VM usage.

@kamzar1

Is there a way to publish a unified tarball instead of split (meta+root)?
how would that unified item look like? Compared to this split context:

I don’t believe so. It’d certainly be possible but that would need deciding on a new ftype for this and having LXD detect such a case and only download the one file instead of looking for the split layout.

@kamzar1

Now sha256 once single once combined is quite challenge. I used to assign attr to those tarballs or get the timestamp of the files, in order to prevent a nonsense re-calculating of sha256s when only selected images where newly created.

On the production image server, we have hidden files to keep that data and avoid having to re-calculate it when just re-generating the index.

root@shell01:~# ls -lah /data/images.linuxcontainers.org/images/ubuntu/focal/amd64/default/20220213_07:42/
total 462M
drwxrwxr-x 2 lxc lxc   21 Feb 13 10:14 .
drwxrwxr-x 5 lxc lxc    3 Feb 13 10:24 ..
-rw-rw-r-- 1 lxc lxc   65 Feb 13 10:14 .lxd_fingerprint
-rw-rw-r-- 1 lxc lxc   65 Feb 13 10:14 .lxd_fingerprint_qcow2
-rw-rw-r-- 1 lxc lxc   65 Feb 13 10:13 .lxd_fingerprint_squashfs
-rw-rw-r-- 1 lxc lxc  633 Feb 13 10:14 SHA256SUMS
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 SHA256SUMS.asc
-rw-rw-r-- 1 lxc lxc 111K Feb 13 10:13 build.log
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 build.log.asc
-rw-rw-r-- 1 lxc lxc 3.2M Feb 13 10:14 delta-20220212_07:42.vcdiff
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 delta-20220212_07:42.vcdiff.asc
-rw-rw-r-- 1 lxc lxc 252M Feb 13 10:13 disk.qcow2
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 disk.qcow2.asc
-rw-rw-r-- 1 lxc lxc  26K Feb 13 10:13 image.yaml
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 image.yaml.asc
-rw-rw-r-- 1 lxc lxc  680 Feb 13 10:13 lxd.tar.xz
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 lxd.tar.xz.asc
-rw-rw-r-- 1 lxc lxc  952 Feb 13 10:13 meta.tar.xz
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 meta.tar.xz.asc
-rw-rw-r-- 1 lxc lxc 110M Feb 13 10:13 rootfs.squashfs
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 rootfs.squashfs.asc
-rw-rw-r-- 1 lxc lxc  98M Feb 13 10:13 rootfs.tar.xz
-rw-rw-r-- 1 lxc lxc  833 Feb 13 10:14 rootfs.tar.xz.asc

[kamzar1 (Dauerurlaub)]

Thanks for elaboration.
The index.json is requested when lxd pulls the image server, but it doesnt make real use of it, as long as format convention is kept, it accepts it without caring about products listed there.

When I create new images, without diff delta and disabled images auto update, there still is the function of caching simplestreams remotes. By launching a container it throws an error message of hash mismatch.
Any workaround for that?