Managing the LXD snap

Introduction

An updated version of this tutorial is available at https://discourse.ubuntu.com/t/managing-the-lxd-snap-package/37214

The LXD snap produced by the LXD team is the preferred way to consume LXD. This allows the LXD team to distribute all of LXD’s dependencies in one package and have LXD run in a consistent environment whilst allowing it to be installed on many different Linux distributions. This greatly simplifies support as other than the kernel version, everything else is going to be identical on all systems.

The snapd packaging system uses the concept of “tracks” and “channels” to allow the user to choose a specific series of the software to install based on their appetite for stability vs new features.

By default snapd will automatically update LXD to the most recent version in the user’s chosen channel up to four times a day.

However as LXD is often used to run critical services some users may prefer to exercise a greater level of control over when LXD is updated. There are several approaches that can be taken to achieve this goal and this guide will go through each approach available.

LXD Tracks and Channels

When installing the LXD snap, you can specify the channel as follows:

sudo snap install lxd --channel=latest/stable

The channel specified is made up of two components; the track and the risk level.

So in this case, we are specifying the latest track with a risk level of stable, meaning that your LXD version will contain all of the latest features, but will be updated when the LXD team decide a feature is ready and no issues have been revealed by users running same revision on the more riskier branches (edge and candidate).

In addition to the latest track, the LXD team also maintains 3 LTS (long term support) tracks that receive only bug fixes and do not receive any new features. These are much ‘slower’ tracks that do not change frequently.

The LTS tracks are currently: 5.0, 4.0, 3.0 and 2.0.

Just like the latest track, you can also specify your risk preference on these tracks by specifying stable, candidate or edge. However these risk levels are still relative to the overall channel’s risk (i.e the 5.0/edge channel is less risky than the latest/edge channel as it will only include bug fixes and not new features).

E.g.

sudo snap install lxd --channel=5.0/stable

Managing Updates

Snap Hold

Since snapd 2.58 snap supports holding updates for a particular snap package for a certain period of time, or forever. This allows the system administrator to manage their own update schedule by periodically un-holding the package, refreshing and then holding again.

E.g.

snap --version
snap    2.59.1
snapd   2.59.1

sudo snap refresh --hold=24h lxd
General refreshes of "lxd" held until 2023-04-24T10:06:22Z

sudo snap refresh --hold lxd
General refreshes of "lxd" held indefinitely

To remove a hold and immediately refresh use:

sudo snap refresh --unhold lxd
sudo snap refresh lxd

Please see https://snapcraft.io/blog/hold-your-horses-i-mean-snaps-new-feature-lets-you-stop-snap-updates-for-as-long-as-you-need and https://snapcraft.io/docs/keeping-snaps-up-to-date#heading--control for more information.

Pinned Feature Channels

If you need a feature from one of the feature releases available in the “latest” track, but do not want to take on the risk profile of even the latest/stable channel or cannot use snap refresh --hold (see above) then pinned feature channels offer another approach to manage update schedules.

To allow more fine grained control over the version of LXD you are running, the LXD team also provides “pinned” channels for the current and previous releases that do not get updated.

E.g.

sudo snap install lxd --channel=5.n/stable

This will install the specific version and no further automatic updates will occur, but only if the version picked is not the latest version. If, however, you pick the latest release version it will receive the same cherry-picked updates as the latest/stable channel until the next release is made, at which time it will receive no further updates.

When you want to upgrade to the next feature version you can run:

sudo snap refresh lxd --channel=5.n/stable

You can see all of the available channels by running:

snap info lxd

System Refresh Windows

Snapd allows some degree of control over when updates are applied so that the time frame when updates are applied can be restricted to times that are suitable for dealing with any unexpected issues that may arise from the update.

These settings apply to all snaps installed on the system, not just LXD.

You can set specific time ranges when refreshes can occur using the refresh.timer setting.

The following example asks the system to only refresh snaps between 4.00am and 7.00am, and 7.00pm and 10:10pm:

sudo snap set system refresh.timer=4:00-7:00,19:00-22:10

You can also delay the refresh from happening until a certain date by setting the refresh.hold setting.

E.g.

sudo snap set system refresh.hold="$(date --date=tomorrow +%Y-%m-%dT%H:%M:%S%:z)"

However after 90 days a refresh will occur irrespective of the value of refresh.hold.

Please see https://snapcraft.io/docs/keeping-snaps-up-to-date for more information on tuning refresh windows.

Refresh behavior in clusters

LXD clusters must all run the same LXD version. It is also recommended that they run with the exact same amount of bug fixes (so same snap revision) as the API could get very inconsistent if that’s not the case.

All LXD servers in a cluster should therefore be configured to be on the same track/channel and if cohorts are in use, must also be part of the same cohort (see more on cohorts below).

As soon as one of the servers refreshes and detects it’s now running on a newer version than the rest, all cluster traffic will be held until things are consistent again. All the other servers will detect this and trigger a self-refresh so that the entire cluster is refreshed as soon as possible.

To manually trigger a refresh on a cluster member and ensure that all cluster members are refreshing using the same phasing cohort, and thus get the same version, use:

sudo snap refresh lxd --cohort="+"

For more information on upgrading a cluster see https://linuxcontainers.org/lxd/docs/master/howto/cluster_manage/#upgrade-cluster-members

Cohorts Pinning

Snapd supports the concept of “cohorts”. This is a snapshot of a package’s current revisions at a point in time. You can then install that package on one or more systems using the cohort’s snapshot combined with the usual channel/risk pair to ensure all systems use the same revision. Additionally the cohort snapshot is retained for 90 days after which a new snapshot of the package’s revision list is made and then retained for a further 90 days. In this way you can limit the amount of updates that snapd will install to once per 90 days.

This combined with the refresh windows discussed above allow you to both limit the frequency of automatic updates as well as specify the time range when they can occur.

NOTE: This isn’t recommended by the LXD team unless someone is paying very close attention to what’s released in the original channel and refreshes the cohort whenever a security or critical bugfix has been pushed. Otherwise, such long term version pinning can result in critical security issues being present on the systems.

snap create-cohort lxd

Which will output:

cohorts:
  lxd:
    cohort-key: <generated key>

Then use the generated key displayed to install lxd using the specific cohort snapshot:

snap install --cohort=<generated key> lxd

Snap Store Proxy

If you manage multiple LXD nodes in a large deployment and cohorts are not sufficient because you need absolute control over when updates are applied, then the Snap Store Proxy may provide the solution you need.

The Snap Store Proxy is a separate application that sits between the snap client command on your nodes where you want to install LXD and the Internet. With this running you can then specify that the Snap Store Proxy only makes a specific revision of LXD available to install for each architecture.

There is a detailed guide on how to set up the Snap Store Proxy here https://docs.ubuntu.com/snap-store-proxy/.

Once your snap clients are configured to use your the proxy, you can then instruct it to only make a specific LXD revision available using:

sudo snap-proxy override lxd <channel>=<revision>

E.g.

sudo snap-proxy override lxd stable=15457

You can see a list of current revisions by running:

snap info lxd

The number in brackets beside the date is the revision number.

There is more information on snap proxy overrides here https://docs.ubuntu.com/snap-store-proxy/en/overrides.

Snap store proxy has a free tier with a limit on the number of nodes that can be connected to it. See https://docs.ubuntu.com/snap-store-proxy/en/register for more information on device limits and if you require more then please contact Canonical using https://ubuntu.com/support/contact-us.

Recommendations

Development system:

  • Use the latest/stable channel.
  • Set the refresh window to be outside of work hours.

Production server:

  • Use the latest/stable channel if you need the latest features and can specify a frequent refresh window.
  • Use a snap refresh --hold lxd if you want to avoid the automatic release upgrade and have time to do manage the refresh cycle manually to ensure you get updates.
  • Use an $LTS/stable channel if you don’t need any of the features that were added since the last LTS release but still want bug fixes and security updates.
  • Set a refresh window to match your system maintenance window.

Staging server:

  • Same as production server, but use the matching “candidate” channel so you can identify any breaking change ahead of it hitting production.

Large deployments and/or air-gapped environments:

  • Set up a Snap Store Proxy.
  • Run test systems on the upstream stable channels.
  • Pin your proxy to the upstream revisions you’ve tested.
  • Frequently re-evaluate and update your pinning.
8 Likes

Typo here, should be /snap-store-proxy i guess Discourse might have nagged you about posting the same link more than once?

Thanks, fixed, not sure what happened there.

1 Like

As you had below, one doesn’t need to sudo to run snap info <snap>

1 Like

If you first sudo snap login, then you do not need sudo in any of the subsequent snap commands.

3 Likes

It would be nice if this article is updated with information on updating clusters.
For example: I’m wondering what the best way to update all nodes in a production env would be?
I could run snap refresh lxd --channel=5.8/stable on each node. This will result in nodes in the cluster getting the blocked state until all nodes are updated.
But containers also get the errored state, which is kinda scary. I didn’t expect that to happen. I thought containers would just keep running.

Then again, maybe there’s a better way to do this?

Edit: I know there is information about this in this article. To clarify: I would like to know if there are best practices.
In my case for running LXD on a small cluster.

If LXD doesn’t start on a particular cluster member because its waiting for the other members to come up to the same version then the instances list on the other members will show those instances in ERROR state because the member is unreachable, or the cluster database is not operational.

However the instances themselves should still be running as they are not stopped as part of the snap refresh.

For cluster refreshes I recommend using:

sudo snap refresh lxd --cohort="+"

This ensures that all cluster members are in the same snap cohort, and thus they will get the same LXD version at the same time and will not be affected by snap’s phasing process.

We have a section on upgrading clusters in our docs:

@ru-fu @stgraber do you think its worth tweaking those steps to include the --cohort="+" to ensure all members are getting the same version and not affected by phasing?

I know that LXD does try and place the snap into the + cohort, but I’ve seen occurrences where this hasn’t happened for various reasons. So recommending that in the docs would seem to make sense?

I’ve updated the Refresh behavior in clusters section.

I think there we run into the problem again that we don’t really want to document snap-specific behavior in the docs.
We have a plan to document the snap-specific features on the website, so maybe we could add a link there once we have this in place.

1 Like

I meant really only changing the existing sudo snap refresh lxd that is in our docs to sudo snap refresh lxd --cohort="+"

So its not adding any additional snap documentation that isn’t already there.
But does avoid having users having their clusters have issues with snap’s phasing feature.

Yeah, that’d make sense I think.

Oh okay. Sure, that makes sense. :slight_smile:

How is it possible that it is tracking latest/stable and the latest/stable is 5.12-c63881f but it shows installed latest/stable as 5.13-cea5ee2?

It’s because it’s in the middle of a phased update.
snap info lxd only shows the new version once phasing hits 100%, until then the previous version (5.12) is shown.

1 Like

I opened an issue about this in the snap project:

https://bugs.launchpad.net/snapstore-server/+bug/1990954

Hopefully it can be changed to offer a better end user experience.

1 Like