Dual-booting Host, same LXD. Possible?

Dual-booting a host “temporarily” (it will take months until I can get rid of the old system for the new one). Both systems have access to the ZFS dataset full of LXD containers. Is it possible to configure LXD on both systems in such a way that no matter whether I boot into the old system or the new system, the containers and the virtual network are all the same?

The bare metal is a new machine with M.2 SSD. The SATA SSD was pulled from the old metal that died on me too early. I can dual-boot into M.2 or SATA. SATA is Xubuntu 18.04.3 (started life as 14.04). M.2 is Ubuntu Server 18.04.3 (fresh). I need access to the LXD containers from both systems. So far I have found a lot of documentation on how to migrate LXD containers from one host to another, not on how to access the same containers from two hosts dual booting on the same bare metal, which is probably a weird / transient scenario. If I did not accumulate so much cruft on my SATA SSD over the years, I could probably do the migration during one weekend…

Hi!

LXD keeps the configuration in /var/lib/lxd (for the deb package) and /var/snap/lxd/ (for the snap package).
I suppose you could configure both hosts to use the same path for the LXD configuration, from some common disk. Then, the ZFS dataset remains the same.
You could create a separate partition on one of the disks for /var/lib/lxd or /var/snap/lxd, and have it mounted on both hosts.
If you manage to keep both LXD installations at the same version, then it should be OK.

Thanks for trying to help. If only it was that easy. LXD is indeed in /var/snap/lxd on the old SATA SSD. On the new system, I have an fstab entry to mount it on /old. I did

mv /var/snap/lxd /var/snaplxd_disabled
ln -s /old/var/snap/lxd /var/snap/
ls -l /var/snap

lrwxrwxrwx 1 root root 17 Aug 12 13:54 lxd -> /old/var/snap/lxd

and yet after rebooting to the new system, the containers are not there. In the old system, they are still sailing nicely along…

/var/snap/ is managed by snapd. You need some extra care with mount points.
But most importantly, you need some reproducible setup that can be used to confirm that the whole process is doable.

The following is a proof-of-concept.

Let’s go.

On the host do the following,

mkdir ~/VARSNAPLXDCOMMON
chmod 777 ~/VARSNAPLXDCOMMON
lxc launch ubuntu:18.04 mylxd1 -c security.nesting=true
lxc launch ubuntu:18.04 mylxd2 -c security.nesting=true
lxc config device add mylxd1 mylxddirectory disk source=/home/user/VARSNAPLXDCOMMON/ path=/var/snap/lxd/common/

On mylxd1 perform the following.

lxc ubuntu mylxd1
mylxd1> sudo apt remove lxd
mylxd1> sudo apt autoremove
mylxd1> sudo snap install lxd
mylxd1> sudo lxd init
...
mylxd1> lxc launch images:alpine/3.6 mycontainer
mylxd1> logout

On mylxd2 perform the following.

lxc ubuntu mylxd2
mylxd1> sudo apt remove lxd
mylxd1> sudo apt autoremove
mylxd1> sudo snap install lxd
mylxd1> sudo lxd init
...
mylxd1> logout

Now, stop mylxd1 so that we perform the attachment to mylxd2.

lxc stop mylxd1
lxc config device add mylxd2 mylxddirectory disk source=/home/user/VARSNAPLXDCOMMON/ path=/var/snap/lxd/common/
lxc start mylxd2

Then, get into mylxd2 to reap the fruits of the shared LXD installation.

$ lxc ubuntu mylxd2
+ exec /bin/login -p -f ubuntu
Last login: Mon Aug 12 21:46:12 UTC 2019 on UNKNOWN
ubuntu@mylxd2:~$ lxc list
To start your first container, try: lxc launch ubuntu:18.04

+-------------+---------+------+------+------------+-----------+
|    NAME     |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+-------------+---------+------+------+------------+-----------+
| mycontainer | STOPPED |      |      | PERSISTENT | 0         |
+-------------+---------+------+------+------------+-----------+
ubuntu@mylxd2:~$ lxc list
+-------------+---------+----------------------+-----------------------------------------------+------------+-----------+
|    NAME     |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
+-------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| mycontainer | RUNNING | 10.219.149.162 (eth0) | fd42:8200:a6e8:c3e4:216:3eff:fe0e:562b (eth0) | PERSISTENT | 0         |
+-------------+---------+----------------------+-----------------------------------------------+------------+-----------+
ubuntu@mylxd2:~$ logout
$ 

Looks cool. Now let’s stop mylxd2, start mylxd1 and do a lxc list in there.

lxc stop mylxd2
lxc start mylxd1
lxc ubuntu mylxd1
ubuntu@mylxd1:~$ lxc list
+-------------+---------+-----------------------+-----------------------------------------------+------------+-----------+
|    NAME     |  STATE  |         IPV4          |                     IPV6                      |    TYPE    | SNAPSHOTS |
+-------------+---------+-----------------------+-----------------------------------------------+------------+-----------+
| mycontainer | RUNNING | 10.219.149.162 (eth0) | fd42:8200:a6e8:c3e4:216:3eff:fe0e:562b (eth0) | PERSISTENT | 0         |
+-------------+---------+-----------------------+-----------------------------------------------+------------+-----------+
ubuntu@mylxd1:~$ 

The container shows up back in mylxd1.

Note that this is proof-of-concept. Most likely LXD was not tested to work like this.
There should be some tightening of the permissions of the shared directory.
Good luck!