How to create LXD binaries from the source code and side load them in an existing snap installation

An updated version of this tutorial is available at https://discourse.ubuntu.com/t/building-custom-lxd-binaries-for-side-loading-into-an-existing-snap-installation/37327

This can be useful if you want to test a feature in a specific environment you already have set up, like a cluster or one with multiple projects, profiles, …, etc. You can place your newly compiled binaries in that environment as .debug files, restart the LXD daemon for the system to pick up these special binaries, test and then simply erase the .debug files and restart the daemon to revert back to normal.

Note that if you test a code change that modifies your database, you will have to revert your database manually when you go back to the previous state or wait for that feature to make it to the official store before you can remove your .debug files.

1. Use a builder container with the same version we are using to create official releases. This will ensure you have the same dependencies as those used in the published snaps. The table below lists the base containers we use in all channels.

Snap Channel Base Container
latest/edge 22.04
latest/candidate 22.04
latest/stable 22.04
5.0/stable 20.04

Here my container is called “builder”:

lxc launch images:ubuntu/22.04 builder
lxc shell builder

2. Inside the container, from instructions provided here

apt update
apt -y upgrade

Then install the required packages except golang which we will install as a snap later.

apt -y install acl attr autoconf automake dnsmasq-base git libacl1-dev libcap-dev liblxc1 liblxc-dev libsqlite3-dev libtool libudev-dev liblz4-dev libuv1-dev make pkg-config rsync squashfs-tools tar tcl xz-utils ebtables
apt -y install lvm2 thin-provisioning-tools
apt -y install btrfs-progs
apt -y install curl gettext jq sqlite3 socat bind9-dnsutils
apt -y install shellcheck

3. Install a recent Go environment

apt -y install snapd
snap install go --classic

4. Get the source code

git clone https://github.com/lxc/lxd

5. Build

cd lxd

Build raft and dqlite

make deps

As per make’s instructions

export CGO_CFLAGS="-I/root/go/deps/raft/include/ -I/root/go/deps/dqlite/include/"
export CGO_LDFLAGS="-L/root/go/deps/raft/.libs -L/root/go/deps/dqlite/.libs/"
export LD_LIBRARY_PATH="/root/go/deps/raft/.libs/:/root/go/deps/dqlite/.libs/"
export CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)"

Build lxd, lxc, … etc

make
exit

6. Install

Get binaries from your builder and add .debug extension

lxc file pull builder/root/go/bin/lxd lxd.debug
lxc file pull builder/root/go/bin/lxc lxc.debug

Push binaries in your current working environment

sudo mv lx[cd].debug /var/snap/lxd/common

Restart LXD daemon

sudo systemctl reload snap.lxd.daemon

Check using top that lxd.debug is running

7. Revert

sudo rm /var/snap/lxd/common/lx[cd].debug
sudo systemctl reload snap.lxd.daemon

Check using top that the regular lxd is running

6 Likes

Thanks for this.

Just a curiosity: I noticed that when I used images:ubuntu/20.04 as opposed to ubuntu:20.04, the cloud-config embedded in my LXD profile was not applied. Just curious if the major difference between these images is the lack of cloud-init, or if there was something else that might be significant.

images: correspond to the community image server where you’ll find a bunch of distros with Ubuntu of course but many more. For some distros you’ll also have the /cloud variant that has cloud-init. We also have a few with graphical desktop /desktop variant.

ubuntu: correspond to Ubuntu Server/Cloud images built (and QA’ed) by Canonical. Those all have cloud-init.

1 Like

Thanks for the insight.
Is there a way using those generated binaries lxd, lxc and simply make them executable on host without snap stuff?

Yeah, that’s fine but then you end up being responsible for writing the systemd units to start them, you also have to make sure to have a compatible version of qemu, edk2, squashfs-tools, …

Certainly possible and a few of us do that pretty regularly for development, but you’ll typically need a pretty recent OS version to get all the features working as normal.

You are right.
Getting all those tools/components in compatible versions, working together will be a nightmare at every update.

Was only a consideration, due persistent snap struggle with namespace, shiftings, zfs bugs, breaking by some updates need to force downgrade or temporary jump to edge …

I am aware, snap gives LXD thousand advantages which I by myself don’t want miss.
But it comes at a price, that makes sometimes desperate! :cry:
Best would be a snap as well as deb/apt package as free choice.

From the 5.12 release announcement, I think the base for the snap has been moved to core22, if I remember correctly. Do I correctly assume that means we should use a 22.04 image to build a compatible binary now?

That’s correct.

@egelinas want to update the post?

@egelinas maybe add a table with a map of snap channel to current base image which we can update as they change over time.

will do

2 Likes