Important notice for LXD users (image server)

News
1

Introduction

The image server at https://images.linuxcontainers.org has been in operation since early 2014, first offering images to LXC users through the download template and then once LXD came around, it was expanded to also provide LXD container and eventually virtual-machine images.

The infrastructure used to build and distribute those images has always been purposefully kept community owned and operated. That’s so that every distribution represented on the image server is on equal footing and can be sure it will always be represented at its best.

Today, we serve images to over half a million monthly users across a variety of platforms, ranging from LXC, LXD, Incus to OpenNebula and more.

Current situation

Maintaining this service comes at a cost, both financial (build machinery, data center, bandwith) and human (keeping up with the different distribution releases, handling build and test failures, …). Up until recently, this cost was shared between @monstermunchkin at Canonical who was maintaining distrobuilder and helping keep the list of releases we’re building for up to date and @stgraber who was responsible for all the infrastructure.

@monstermunchkin moved on from Canonical at the end of November and Canonical decided not to continue his work on this shared project.

Additionally, Canonical has recently been taking a number of other community hostile actions, essentially preventing our community from benefiting from any of the work that they are doing.
This change also prevents us from debugging any LXD-specific issues reported against our images.

As a result, the Linux Containers project has made the decision to phase out access to our image server for LXD users. We realize that this may cause significant impact on the LXD user base and so have opted for a progressive phase-out matching those users’ ability to migrate to Incus.

NOTE: This does not affect any existing instances you may have running, nor does it affect anyone exclusively using official Ubuntu images (ubuntu: and ubuntu-daily: remotes).

Timeline

Date Change
2024/01/01 LXD 5.20+ users running on Arch, Debian, Fedora, Gentoo, NixOS or Ubuntu will be restricted to Ubuntu, Debian, CentOS and Alpine images
2024/01/15 LXD 5.20+ users running on Arch, Debian, Fedora, Gentoo, NixOS or Ubuntu will completely lose access
2024/01/22 Non-LTS LXD users running on Arch, Debian, Fedora, Gentoo, NixOS or Ubuntu will be restricted to Ubuntu, Debian, CentOS and Alpine images.
2024/02/01 Non-LTS LXD users running on Arch, Debian, Fedora, Gentoo, NixOS or Ubuntu will completely lose access
2024/02/15 LXD 5.20+ users on any distribution will be restricted to Ubuntu, Debian, CentOS and Alpine images
2024/03/01 Non-LTS LXD users on any distribution will be restricted to Ubuntu, Debian, CentOS and Alpine images
2024/03/15 LXD 5.20+ users running on any distribution will completely lose access
2024/04/01 Non-LTS LXD users running on any distribution will completely lose access
2024/04/15 All LXD LTS users will be restricted to Ubuntu, Debian, CentOS and Alpine images
2024/05/01 All LXD users will lose access

The list of host distributions affected is subject to change, it’s based on the list of distributions with readily available, good quality Incus packages that users can easily migrate to. Considering current packaging efforts, this list will very likely be expanded over the coming weeks.

The list of distributions for which images are temporarily still made available is based on usage statistics, they are the top 4 most used and account for 90% of uses, this allows a one week brownout period prior to access being revoked for those users.

The 6.0 LTS release of LXC, LXCFS and Incus is scheduled to happen towards the end of March, making it so that LXD LTS users will have an LTS upgrade path by the time image server is revoked for them too a couple of months later.

How to switch to Incus

You’ll find information on how to install Incus for your distribution here: How to install Incus - Incus documentation

As well as how to easily migrate all your LXD data over to Incus here: Migrating from LXD - Incus documentation

On most systems, the migration can be done with just a few seconds of downtime.

NOTE: You may need to wipe /var/cache/incus/ and restart Incus with systemctl restart incus to clear any bad image cache from your previous LXD installation.

Conclusion

Between Canonical’s decision to no longer fund any work on the image build tooling or infrastructure, as well as the recent community hostile licensing change which prevents Incus from benefiting from any of the work on Canonical LXD, it doesn’t really make a lot of sense to keep spending significant time and money in operating this infrastructure for LXD users.

We all would have far preferred that Canonical stay committed to providing good quality non-Ubuntu images for their users as we would have preferred that they don’t take community hostile actions by re-licensing and requiring the signature of a CLA for new contributions. But they have done neither of those things and so it’s time for us to focus on the future.

18 Likes
5

Will incus still maintain access to all prebuilt distributions?

1 Like
6

Yes, Incus and LXC both have access to all the images on the image server.

3 Likes
7

That great, it will effect LXD only not Incus :+1:

1 Like
8

Thanks so much @monstermunchkin for his work maintaining the distros :clap:

Who is going to be doing this distro maintainer work now?

1 Like
9

I will continue maintaining distrobuilder and the images.

11 Likes
10

I can donate a server with 30+ Terabytes of storage including bandwidth if you need it as long as someone maintains it.

6 Likes
11

I perfectly understand, approve and back this move, however I’ve a question about the current state of things and specifically Debian 12 users.

Debian includes LXD LTS 5.0.2 on their repositories and that version will be still be around after 2024/05 and trying to use the image server. Debian won’t likely change stable to include Incus until 2025, what’s the suggested path here?

Thank you.

1 Like
12

We’ve been working pretty closely to Debian on this. I expect we’ll keep allowing Debian users of LXD 5.0.2 to interact with the image server either until trixie is released with Incus available OR a backport of Incus is made available in bookworm-backports, whichever happens first.

It’s worth noting though that we won’t be doing any image testing on LXD, so things may degrade and as we have to stay away from the LXD code now, we may not be able/willing to debug such regressions.

For those who don’t mind the external repository, I do build packages for both Debian 11 and Debian 12 that can be used to move to Incus without having to wait for the next Debian release or for a proper backport. GitHub - zabbly/incus: Incus package repository

3 Likes
14

That’s great to know, as usual you’ve thought everything through.

Be it backport or your repository how does that play with Debian’s LXC versions? And what about more “exotic” architectures such as arm64?

Thank you.

1 Like
15

I build my packages (and personally run them) for both 64-bit Intel/AMD and 64-bit Arm.
My packages are “fat” packages in that they bundle all the needed bits, including liblxc, QEMU, … This allows getting a consistent experience across distributions and releases and avoids any conflict with what may already be on the system.

4 Likes
16

That makes sense. Thank you.

1 Like
17

@stgraber , I would like to strongly suggest the bandwidth capping for LXD users while they still have access.

It is not fair that you pay alone the cost of five more additional months of their usage (a significant usage, by the way), and this seems to me as a good strategy to avoid excessive usage and costs till the full block.

You can also apply a progressive bandwidth capping percentual, if you believe that simply capping would be too aggressive.

1 Like
18

This misses the point that we want to actually fully phase out those images.
We can’t test or debug them anymore, so continuing to provide them just doesn’t make much sense.

Our infrastructure also runs across a dozen edge servers in various countries.
Applying any kind of bandwidth capping, whether limiting the speed or total daily bandwidth would require changes to all the edge servers, which is a lot more work than just changing the content of the index provided to a given user by the primary servers.

Degrading the quality of the service by limiting the speed would just make users blame their internet or blame our infrastructure, it wouldn’t cause them to go look at why their favorite images are no longer available. Limiting download speed also means clients remaining connected to servers for longer, actually increasing system load on our side which is also not desired.

2 Likes
19

You said, you expect Incus to have a LTS release in 4-5 months.
So we would have roughly less than 1 month to migrate from LXD LTS to Incus LTS?

That’s tight.

1 Like
20

We’re aiming for mid to late March for Incus 6.0 LTS, so yeah, looking at about a month after that before access becomes restricted for the remaining LXD LTS users.

Though keep in mind that this only affects the ability to create new, non-Ubuntu instances.
Our stats so far show that LTS users pull very very few new images, presumably being primarily in the camp of creating their entire infrastructure once and then operating it, rather than constantly creating new instances.

As mentioned in other posts, we’ll keep an eye on the user stats and see how things go and may delay some of the steps above. We have to keep in mind that during those 5 months, if something breaks badly for LXD users, there’s not going to be much we’ll be able to do about it either, so we really do want to get ourselves out of that position sooner rather than later.

1 Like
21

However, LXD 5.x LTS is supported until 2027 right, you mentioned something that could badly break?
How so? LTS is not suppose to introduce any breaking changes, why would the images you provide break? I get the point if you are on the non LTS branch or upgrade to LXD 6.x LTS or upgrade the Hostsystem, it could break, but on LTS? If you leave it as is.

2-4 weeks is quite short, however I cache the images locally anyway, because usually the servers are far remote and the downloads speeds are not great.
I don’t mind using the images for a few weeks longer before I have to switch.

Any simple guide to run your own image server for LXD currently?
If people donate resources are you willing to extend the deadlines?

1 Like