I recently installed the snap version of LXD after running the packaged version included with Ubuntu 17.10 (just before this I also upgraded to 18.04). I ran lxd.migrate and got the following error:
=> Connecting to source server
=> Connecting to destination server
=> Running sanity checks
=== Source server
LXD version: 3.0.0
LXD PID: 1824
Resources:
Containers: 6
Images: 0
Networks: 0
Storage pools: 2
=== Destination server
LXD version: 3.0.0
LXD PID: 4517
Resources:
Containers: 0
Images: 0
Networks: 0
Storage pools: 0
The migration process will shut down all your containers then move your data to the destination LXD.
Once the data is moved, the destination LXD will start and apply any needed updates.
And finally your containers will be brought back to their previous state, completing the migration.
Are you ready to proceed (yes/no) [default=no]? yes
=> Shutting down the source LXD
=> Stopping the source LXD units
=> Stopping the destination LXD unit
=> Unmounting source LXD paths
=> Unmounting destination LXD paths
=> Wiping destination LXD clean
=> Moving the data
=> Moving the database
=> Backing up the database
=> Opening the database
=> Updating the storage backends
error: Failed to update the storage pools: no such table: storage_pools
The old installation is still there, but it’s empty. The new installation lists my containers but won’t start any, with a Error: no such file or directory error. My storage pools are btrfs looped images, and I see they’ve been moved to /var/snap/lxd/common/lxd/disks. I thought I might just be able to set the source of the pools to the new path, but I get Error: The [source] properties cannot be changed for "btrfs" storage pools
I also don’t know what other migration steps were left after configuring the storage pools.
I’ve still got what I presume to be the original database at /var/lib/lxd/lxd.db, but at /var/snap/lxd/common/lxd there is only lxd.db.bak. There is no lxd.db. There is a local.db datbase in the /database directory, but it only has a few tables.
At this point I think I need to make sure the database is in the correct path and possibly manually update the database entries that contain the source paths for the storage pools. I’m going to hold off on trying to make changes for now as I’m very unfamiliar.
As you’ve noticed, the database doesn’t work the same way with LXD 3.0, that’s not a problem when doing LXD 2.x to LXD 3.x upgrades, but moving from LXD 3.0 deb to LXD 3.0 snap is a bit of a problem…
Right now the safest would be to revert back to the deb, remove the snap and wait until we sort this out.
To do so, this should work:
mv /var/snap/lxd/common/lxd /var/lib/lxd
systemctl start lxd
/usr/bin/lxc list
If you see all your containers and everything looks right, then you may remove the LXD snap with snap remove lxd. DO NOT remove the LXD snap until you’ve confirmed that your data is accessible at the deb’s location as removing the snap will wipe everything under /var/snap/lxd
So I ran ‘mv /var/snap/lxd/common/lxd /var/lib/lxd’ and I’m noticing that it looks like that made an lxd subdirectory at /var/lib/lxd. That’s probably not intended, and maybe I should’ve run ‘mv /var/snap/lxd/common/lxd /var/lib’ instead? Right now, I can start the deb LXD but it’s still looking empty.
So, I did what you suggested, then started LXD again with ‘systemctl start lxd’ and everything still looked empty. Remember that after the failed migration, I did not have an lxd.db in the snap location. The contents of /var/lib/lxd/lxd.db now looked empty, so I went ahead and moved that to lxd.db.bak2, then copied lxd.db.bak to lxd.db, then started LXD again. Now my container list looks correct, but when I try to launch a container I get Error: saving config file for the container failed
Apologies, but I won’t be able to touch this again until after work in about 8 hours. Hopefully you’re still available around then! Unfortunately, my remote access was served via a container and I haven’t set up an alternative method yet.
Out of curiosity, was it correct to move lxd.db like I did? I’m not aware of exactly how the database setup has changed in LXD 3.0, though I have tried reading through the docs.
Sorry, I forgot about this part, despite both snap and deb having the same version, the snap has a number of cherry-picked improvements from upstream, moving those database files being one of them…
I’m currently in Europe for work meetings so I’ll be sleeping by the time you get back home but hopefully you have enough information above to move things back.
The long version I think is:
Move /var/snap/lxd/common/lxd to become /var/lib/lxd (which you’ve kinda done, just needs fixing per above)
Move /var/lib/lxd/database/global to /var/lib/lxd/raft
Move /var/lib/lxd/database/local.db to /var/lib/lxd/lxd.db
Remove empty /var/lib/lxd/database directory
systemctl start lxd
Check that it works with /usr/bin/lxc list
If it’s all back, remove the snap
Until the snap is removed, you’ll need to type /usr/bin/lxc to access the client tool, otherwise you’ll use the snap version which will attempt to talk to the wrong LXD server.
I just want to clarify…after the failed migration, at /var/snap/lxd/common/lxd I did not have an lxd.db, only a lxd.db.bak. Additionally, /var/snap/lxd/common/lxd/database/local.db didn’t appear to have most of the data from my old database - checking tables with sqlite3 showed only a few tables (I’m assuming this is because of the failed migration step noted originally).
So I do think that the only correct version of my previous database is the lxd.db.bak that was created, which is why I moved that one back to /var/lib/lxd/lxd.db. Checking tables in that file show everything you’d expect. Let me know your thoughts on this.
Do you think, after everything, that the contents of /var/lib/lxd/database/global will be correct at this point for moving back to /var/lib/lxd/raft? If so, I will move it when I get home, remove the empty database directory, then see how it goes. If I get the same error, I’ll also check for the presence of /var/log/lxd.
Is there a way to verify that the data in /global is correct in that case? I looked through that folder and didn’t see any databases besides logs.db. I will keep a backup copy of my current lxd.db that I restored, of course, I’m just worried that if I move local.db to lxd.db and /global doesn’t have the right data I’ll hit another roadblock.
Thanks for the info - in that case I will stop worrying so much about lxd.db and just make sure I’ve got my backup, then perform the steps you suggested. Do you still think /var/log/lxd might be an issue, or should I start with the other steps first?
I was able to run home and try a few things out. Here’s what I did:
moved database/local.db to /var/lib/lxd/lxd.db
I moved files in database/global one-by-one. I will note that besides the /global directory and local.db, the database directory also contained db.bin and logs.db. I initially accidentally moved /database/db.bin to /var/lib/lxd/raft/db.bin, realized my mistake since it wasn’t from the global directory, then moved the correct file from database/global/db.bin instead. Unfortunately I neglected to back it up first, so I no longer have the original db.bin that was located at /var/lib/lxd/database.
After finishing moving the rest of the files in /var/lib/lxd/database/global to /var/lib/lxd/raft, I moved /var/lib/lxd/database to /var/lib/lxd/database.bak.
When I tried to start lxd with ‘systemctl start lxd’ after this, it hung and never started. I rebooted the host, and LXD seems to be running now. I can still list my containers and configs and everything looks correct, but I’m receiving the same Error: saving config file for the container failed error when trying to start containers.
For what it’s worth, I do indeed have /var/log/lxd, as well as /var/lib/lxd/logs. Let me know if want output from either of those. I’ve got a PC started up at home now so I can try to work on this remotely.
At this point, it might be easiest to just purge and uninstall both LXD instances completely, then start from scratch with the snap package. As long as I can extract the data I need from my btrfs images (which I would also need help to do), it wouldn’t take me too long to reconfigure LXD and the containers. Is that something you can help me out with?