Introduction
An updated version of this tutorial is available at https://discourse.ubuntu.com/t/managing-the-lxd-snap-package/37214
The LXD snap produced by the LXD team is the preferred way to consume LXD. This allows the LXD team to distribute all of LXD’s dependencies in one package and have LXD run in a consistent environment whilst allowing it to be installed on many different Linux distributions. This greatly simplifies support as other than the kernel version, everything else is going to be identical on all systems.
The snapd packaging system uses the concept of “tracks” and “channels” to allow the user to choose a specific series of the software to install based on their appetite for stability vs new features.
By default snapd will automatically update LXD to the most recent version in the user’s chosen channel up to four times a day.
However as LXD is often used to run critical services some users may prefer to exercise a greater level of control over when LXD is updated. There are several approaches that can be taken to achieve this goal and this guide will go through each approach available.
LXD Tracks and Channels
When installing the LXD snap, you can specify the channel as follows:
sudo snap install lxd --channel=latest/stable
The channel specified is made up of two components; the track and the risk level.
So in this case, we are specifying the latest track with a risk level of stable, meaning that your LXD version will contain all of the latest features, but will be updated when the LXD team decide a feature is ready and no issues have been revealed by users running same revision on the more riskier branches (edge and candidate).
In addition to the latest track, the LXD team also maintains 3 LTS (long term support) tracks that receive only bug fixes and do not receive any new features. These are much ‘slower’ tracks that do not change frequently.
The LTS tracks are currently: 5.0, 4.0, 3.0 and 2.0.
Just like the latest track, you can also specify your risk preference on these tracks by specifying stable, candidate or edge. However these risk levels are still relative to the overall channel’s risk (i.e the 5.0/edge channel is less risky than the latest/edge channel as it will only include bug fixes and not new features).
E.g.
sudo snap install lxd --channel=5.0/stable
Managing Updates
Snap Hold
Since snapd 2.58 snap supports holding updates for a particular snap package for a certain period of time, or forever. This allows the system administrator to manage their own update schedule by periodically un-holding the package, refreshing and then holding again.
E.g.
snap --version
snap 2.59.1
snapd 2.59.1
sudo snap refresh --hold=24h lxd
General refreshes of "lxd" held until 2023-04-24T10:06:22Z
sudo snap refresh --hold lxd
General refreshes of "lxd" held indefinitely
To remove a hold and immediately refresh use:
sudo snap refresh --unhold lxd
sudo snap refresh lxd
Please see https://snapcraft.io/blog/hold-your-horses-i-mean-snaps-new-feature-lets-you-stop-snap-updates-for-as-long-as-you-need and https://snapcraft.io/docs/keeping-snaps-up-to-date#heading--control for more information.
Pinned Feature Channels
If you need a feature from one of the feature releases available in the “latest” track, but do not want to take on the risk profile of even the latest/stable channel or cannot use snap refresh --hold (see above) then pinned feature channels offer another approach to manage update schedules.
To allow more fine grained control over the version of LXD you are running, the LXD team also provides “pinned” channels for the current and previous releases that do not get updated.
E.g.
sudo snap install lxd --channel=5.n/stable
This will install the specific version and no further automatic updates will occur, but only if the version picked is not the latest version. If, however, you pick the latest release version it will receive the same cherry-picked updates as the latest/stable channel until the next release is made, at which time it will receive no further updates.
When you want to upgrade to the next feature version you can run:
sudo snap refresh lxd --channel=5.n/stable
You can see all of the available channels by running:
snap info lxd
System Refresh Windows
Snapd allows some degree of control over when updates are applied so that the time frame when updates are applied can be restricted to times that are suitable for dealing with any unexpected issues that may arise from the update.
These settings apply to all snaps installed on the system, not just LXD.
You can set specific time ranges when refreshes can occur using the refresh.timer setting.
The following example asks the system to only refresh snaps between 4.00am and 7.00am, and 7.00pm and 10:10pm:
sudo snap set system refresh.timer=4:00-7:00,19:00-22:10
You can also delay the refresh from happening until a certain date by setting the refresh.hold setting.
E.g.
sudo snap set system refresh.hold="$(date --date=tomorrow +%Y-%m-%dT%H:%M:%S%:z)"
However after 90 days a refresh will occur irrespective of the value of refresh.hold.
Please see https://snapcraft.io/docs/keeping-snaps-up-to-date for more information on tuning refresh windows.
Refresh behavior in clusters
LXD clusters must all run the same LXD version. It is also recommended that they run with the exact same amount of bug fixes (so same snap revision) as the API could get very inconsistent if that’s not the case.
All LXD servers in a cluster should therefore be configured to be on the same track/channel and if cohorts are in use, must also be part of the same cohort (see more on cohorts below).
As soon as one of the servers refreshes and detects it’s now running on a newer version than the rest, all cluster traffic will be held until things are consistent again. All the other servers will detect this and trigger a self-refresh so that the entire cluster is refreshed as soon as possible.
To manually trigger a refresh on a cluster member and ensure that all cluster members are refreshing using the same phasing cohort, and thus get the same version, use:
sudo snap refresh lxd --cohort="+"
For more information on upgrading a cluster see https://linuxcontainers.org/lxd/docs/master/howto/cluster_manage/#upgrade-cluster-members
Cohorts Pinning
Snapd supports the concept of “cohorts”. This is a snapshot of a package’s current revisions at a point in time. You can then install that package on one or more systems using the cohort’s snapshot combined with the usual channel/risk pair to ensure all systems use the same revision. Additionally the cohort snapshot is retained for 90 days after which a new snapshot of the package’s revision list is made and then retained for a further 90 days. In this way you can limit the amount of updates that snapd will install to once per 90 days.
This combined with the refresh windows discussed above allow you to both limit the frequency of automatic updates as well as specify the time range when they can occur.
NOTE: This isn’t recommended by the LXD team unless someone is paying very close attention to what’s released in the original channel and refreshes the cohort whenever a security or critical bugfix has been pushed. Otherwise, such long term version pinning can result in critical security issues being present on the systems.
snap create-cohort lxd
Which will output:
cohorts:
lxd:
cohort-key: <generated key>
Then use the generated key displayed to install lxd using the specific cohort snapshot:
snap install --cohort=<generated key> lxd
Snap Store Proxy
If you manage multiple LXD nodes in a large deployment and cohorts are not sufficient because you need absolute control over when updates are applied, then the Snap Store Proxy may provide the solution you need.
The Snap Store Proxy is a separate application that sits between the snap client command on your nodes where you want to install LXD and the Internet. With this running you can then specify that the Snap Store Proxy only makes a specific revision of LXD available to install for each architecture.
There is a detailed guide on how to set up the Snap Store Proxy here https://docs.ubuntu.com/snap-store-proxy/.
Once your snap clients are configured to use your the proxy, you can then instruct it to only make a specific LXD revision available using:
sudo snap-proxy override lxd <channel>=<revision>
E.g.
sudo snap-proxy override lxd stable=15457
You can see a list of current revisions by running:
snap info lxd
The number in brackets beside the date is the revision number.
There is more information on snap proxy overrides here https://docs.ubuntu.com/snap-store-proxy/en/overrides.
Snap store proxy has a free tier with a limit on the number of nodes that can be connected to it. See https://docs.ubuntu.com/snap-store-proxy/en/register for more information on device limits and if you require more then please contact Canonical using https://ubuntu.com/support/contact-us.
Recommendations
Development system:
- Use the
latest/stablechannel. - Set the refresh window to be outside of work hours.
Production server:
- Use the
latest/stablechannel if you need the latest features and can specify a frequent refresh window. - Use a
snap refresh --hold lxdif you want to avoid the automatic release upgrade and have time to do manage the refresh cycle manually to ensure you get updates. - Use an
$LTS/stablechannel if you don’t need any of the features that were added since the last LTS release but still want bug fixes and security updates. - Set a refresh window to match your system maintenance window.
Staging server:
- Same as production server, but use the matching “candidate” channel so you can identify any breaking change ahead of it hitting production.
Large deployments and/or air-gapped environments:
- Set up a Snap Store Proxy.
- Run test systems on the upstream stable channels.
- Pin your proxy to the upstream revisions you’ve tested.
- Frequently re-evaluate and update your pinning.
