Force snap package to use /var/lib/lxd

Unfortunately, 2.21 is not available for 18.04 Bionic so i have to use the snap package. However, it uses /var/snap/lxd/common/lxd instead of /var/lib/lxd. Is it possible to use /var/lib/lxd for the snap package?

I am trying to share containers between ubuntu and gentoo/openrc. Thus i need consistent PATHS between the two. I was thinking of using a ZFS dataset ‘rpool/liblxd’ mounted to ‘/var/lib/lxd’. This dataset could then be used for both Gentoo and Ubuntu. This is not possible to do with the custom snap path.

Hmm, so it’s likely going to be a bit harder than that.

For LXD itself, yes, a bind-mount would work fine. The problem is that LXD also expects all the container datasets to have a valid mountpoint under its main directory, this wouldn’t be true in your case, so you’d likely need a small init script which updates the mountpoint property of all your datasets at boot.

It may actually be easier to do it the other way around, running the non-snapped LXD using /var/snap/lxd/common/lxd as its directory. In theory all you’d need to do to make that happen is update the init script in your distro to set LXD_DIR=/var/snap/lxd/common/lxd.

Nice idea. I modified my gentoo init script and setup ZFS mountpoint to /var/snap/lxd/common/lxd and it worked in gentoo. Unfortunately ‘lxc list’ in Ubuntu is giving me an empty list,

$ lxc list

# ls -la /var/snap/lxd/common/lxd
total 37
drwx--x--x 14 root root    18 Mar 31 02:36 .
drwxr-xr-x  5 root root     9 Mar 31 02:36 ..
drwx------  2 root root     3 Mar 31 02:36 cache
drwx--x--x  2 root root     2 Mar 31 02:36 containers
drwx--x--x  2 root root     2 Mar 31 02:36 devices
drwxr-xr-x  2 root root     2 Mar 31 02:36 devlxd
drwx------  2 root root     2 Mar 31 02:36 disks
drwx------  2 root root     2 Mar 31 02:36 images
drwxr-xr-x  3 root root     4 Mar 31 02:36 logs
-rw-r--r--  1 root root 58368 Mar 31 02:36 lxd.db
drwx--x--x  2 root root     2 Mar 31 02:36 networks
drwx------  2 root root     2 Mar 31 02:36 security
-rw-r--r--  1 root root  1927 Mar 31 02:36 server.crt
-rw-------  1 root root  3243 Mar 31 02:36 server.key
drwx--x--x  2 root root     2 Mar 31 02:36 shmounts
drwx------  2 root root     2 Mar 31 02:36 snapshots
drwx--x--x  3 root root     3 Mar 31 02:37 storage-pools
srw-rw----  1 root lxd      0 Mar 31 02:36 unix.socket

I can see the containers,

# ls -la /var/snap/lxd/common/lxd/storage-pools/default/containers
total 18
drwxr-xr-x 4 root    root    4 Mar 31 02:37 .
drwxr-xr-x 3 root    root    3 Mar 31 02:37 ..
drwxr-xr-x 4 1000000 1000000 6 Mar 31 01:33 openhab
drwxr-xr-x 4 1000000 1000000 6 Mar 31 01:33 zmvm

Just for posterity I had to make following changes in Gentoo,

# cat etc/init.d/lxd 
# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2



depend() {
    need net
    use lxcfs

start() {
    ebegin "Starting lxd service"

    start-stop-daemon --start \
                      --pidfile ${PIDFILE} \
                      --exec ${DAEMON} \
                      -e  LXD_DIR=/var/snap/lxd/common/lxd \
                      --background \
                      --make-pidfile \
                      -- \

    eend $?

stop() {
    if [ "$RC_GOINGDOWN" = "YES" ] || [ "$RC_REBOOT" = "YES" ]; then
        ebegin "Stopping lxd service (but not containers)"
        start-stop-daemon --stop --quiet -R TERM/45 -p ${PIDFILE}
        eend $?

stopall() {
    ebegin "Stopping lxd service and containers"
    if "${DAEMON}" shutdown; then
        /etc/init.d/lxd zap
        rm -f ${PIDFILE}
    eend $?

# cat etc/env.d/52lxd 

Solved. It seems that my ZFS dataset was not actually mounted.

Now, i am able share LXC containers in both Gentoo and Ubuntu.

1 Like

After an upgrade to LXD V3, lxc list shows up as an empty list.

# ps aux|grep lxd
root     14109  0.3  0.1 1370808 20820 ?       Ssl  19:44   0:01 /usr/sbin/lxd --group lxd --debug --verbose --logfile /var/log/lxd/lxd.log

I started it manually and it shows that storage pool not detected.

# LXD_DIR=/var/snap/lxd/common/lxd /usr/sbin/lxd --group lxd --debug --verbose --logfile /var/log/lxd/lxd.log
INFO[05-05|20:02:50] LXD 3.0.0 is starting in normal mode     path=/var/snap/lxd/common/lxd
INFO[05-05|20:02:50] Kernel uid/gid map: 
INFO[05-05|20:02:50]  - u 0 0 4294967295 
INFO[05-05|20:02:50]  - g 0 0 4294967295 
INFO[05-05|20:02:50] Configured LXD uid/gid map: 
INFO[05-05|20:02:50]  - u 0 1000000 65536 
INFO[05-05|20:02:50]  - g 0 1000000 65536 
WARN[05-05|20:02:50] AppArmor support has been disabled because of lack of kernel support 
WARN[05-05|20:02:50] Couldn't find the CGroup network class controller, network limits will be ignored. 
INFO[05-05|20:02:50] Initializing database gateway 
INFO[05-05|20:02:50] Start database node                      address= id=1
INFO[05-05|20:02:50] Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}] 
INFO[05-05|20:02:50] Raft: Node at 0 [Leader] entering Leader state 
INFO[05-05|20:02:50] LXD isn't socket activated 
INFO[05-05|20:02:50] Starting /dev/lxd handler: 
INFO[05-05|20:02:50]  - binding devlxd socket                 socket=/var/snap/lxd/common/lxd/devlxd/sock
INFO[05-05|20:02:50] REST API daemon: 
INFO[05-05|20:02:50]  - binding Unix socket                   socket=/var/snap/lxd/common/lxd/unix.socket
DBUG[05-05|20:02:50] No existing storage pools detected. 
DBUG[05-05|20:02:50] Initialized inotify with file descriptor 14 
INFO[05-05|20:02:50] Pruning expired images 
INFO[05-05|20:02:50] Done pruning expired images 
INFO[05-05|20:02:50] Expiring log files 
INFO[05-05|20:02:50] Updating instance types 
INFO[05-05|20:02:50] Updating images 
INFO[05-05|20:02:50] Done updating images 
INFO[05-05|20:02:50] Done expiring log files 
INFO[05-05|20:02:54] Done updating instance types

I have a storage pool (ZFS rpool/lxd) which has my containers,

# ls /var/snap/lxd/common/lxd/storage-pools/default/
containers  custom  images  snapshots

The profile does not show the storage pool,

# lxc profile show default
config: {}
description: Default LXD profile
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
name: default
used_by: []

If i try to add the existing dataset (with previous container), i get an error,

# lxc storage create default zfs source=rpool/lxd
Error: Provided ZFS pool (or dataset) isn't empty

How can i add my dataset to LXD?

This helped

I suspect the problem here is that we’ve changed the database path in recent commits which have been included in the snap.

So one of your installs is now looking at /var/snap/lxd/common/lxd/database/local.db and /var/snap/lxd/common/lxd/database/global and the other is looking at /var/snap/lxd/common/lxd/lxd.db and /var/snap/lxd/common/lxd/raft.

If that’s the issue, then there’s not much you can do really until the non-snap side of your system catches up and looks at the right place.

That’s assuming you’ve still got that setup in place and that the output above comes from the gentoo side.

I get the following error when running snap package under Ubuntu.

Failed to start the daemon: Both legacy and new local database files exists

On the other hand Gentoo lxd containers are working fine.

On Gentoo i get

# lxc --version
# lxd --version

Is gentoo using /var/snap/lxd/common/lxd/database/local.db or /var/snap/lxd/common/lxd/lxd.db ?

Gentoo is likely using lxd.db, so that’d mean that the snap did the migration to /database/, then was probably working fine, switching back to Gentoo re-created lxd.db and now things are pretty confused.

This is a bit odd. Both of them are running the same version, so why they are looking for files in different locations?

Because the snap package has received a number of post-release bugfixes (cherry-picks) which I suspect haven’t been applied to the Gentoo package.

When someone finds a bug in LXD and they’re using the snap, we push the fix on top of the snap package.
So while the version number doesn’t change, we do push new revisions of it with bugfixes.

The same is true for some distributions (the Ubuntu 3.0.0 has 20-ish fixes on top) but there’s never any guarantee that they’re all perfectly in sync.

Well I don’t mean to sound pedant (so please don’t take it as such, we :heart: the work on LXD ), but I can’t refrain from wondering what good are versions number if different software versions have the same version number??

Why not incrementing the patch part of the version number (the Z in version X.Y.Z) after each bugfix?

For non-backward compatible changes, the Y or even X version should change according to semantic versioning.

Distributions cherry-picking fixes on top of the upstream release isn’t a new thing, this doesn’t come with a change to the version number because upstream didn’t actually release a new version. Instead the packaging revision is what’s bumped, as is the case here.