Please, any help recovering and restoring my containers would be helpful. Thanks in advance.
~$ which lxc
/usr/bin/lxc
~$ which lxd
/usr/bin/lxd
~$ lxd --version
4.0.2
~$ sudo lxd.migrate
=> Connecting to source server
error: Unable to connect to the source LXD: Get "http://unix.socket/1.0": dial unixy
LXD on ZFS
lxd --debug
DBUG[07-08|10:10:37] Connecting to a local LXD over a Unix socket
DBUG[07-08|10:10:37] Sending request to LXD method=GET url=http:/=
systemctl list-jobs
JOB UNIT TYPE STATE
2 multi-user.target start waiting
132 snap.lxd.activate.service start running
145 systemd-update-utmp-runlevel.service start waiting
1 graphical.target start waiting
tomp
July 8, 2020, 2:54pm
2
stgraber
July 8, 2020, 2:57pm
3
It’s possible but unlikely unless clustering was similarly mis-enabled on this system.
stgraber
July 8, 2020, 2:57pm
4
Your output is truncated above, please make sure to copy paste the entire thing.
Can you show:
snap list
journalctl -u snap.lxd.daemon -n 300
systemctl -a
jeff@cojoserver:~$ snap list
stgraber
July 8, 2020, 3:27pm
8
Once again, you’re truncating every line of the output making it unusable…
Maybe use a different terminal or tool so that the lines wrap but aren’t truncated.
jeff@cojoserver:~$ journalctl -u snap.lxd.daemon -n 300
jeff@cojoserver:~$
jeff@cojoserver:~$ systemctl -a
modprobe@drm.service loaded inactive dead Load Kernel Module drm >
systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories >
LOAD = Reflects whether the unit definition was properly loaded.
538 loaded units listed.
It might be indeed the same problem we saw yesterday as @tomp was saying.
@cojojeff please can you paste the output of:
sudo sqlite3 /var/snap/lxd/common/lxd/database/local.db "SELECT * FROM config"
and:
sudo sqlite3 /var/snap/lxd/common/lxd/database/local.db "SELECT * FROM raft_nodes"
?
You might need to install the sqlite3 package if you don’t have it already.
jeff@cojoserver:~$ sudo sqlite /var/snap/lxd/common/lxd/database/local.db “SELECT * FROM config”
jeff@cojoserver:~$ sudo sqlite /var/snap/lxd/common/lxd/database/local.db “SELECT * FROM raft_nodes”
stgraber
July 8, 2020, 6:33pm
17
@freeekanayaka told you that you need to install the sqlite3 package
cojojeff:
eff@cojoserver:~$ sudo sqlite /var/snap/lxd/common/lxd/database/local.db “SELECT * FROM config”
jeff@cojoserver:~$ sudo sqlite /var/snap/lxd/common/lxd/database/local.db “SELECT * FROM raft_nodes”
The command to use is sqlite3 not sqlite, see my original reply.
Thanks for your patience…
jeff@cojoserver:~$ sudo sqlite3 /var/snap/lxd/common/lxd/database/local.db “SELECT * FROM config”
jeff@cojoserver:~$ sudo sqlite3 /var/snap/lxd/common/lxd/database/local.db “SELECT * FROM raft_nodes”
stgraber
July 9, 2020, 3:01pm
20
Ok, so it is the same as @laney
@freeekanayaka I think we need some kind of LXD patch to sort this out, seems like it’s hitting a bunch of people. It makes no sense to me that we can have a properly configured cluster.https_address but 0 as address in raft_nodes.
Yeah, looking into this. It must be a regression caused by some of the recent changes that landed. It’s slightly different than @laney ’s since there is actually a valid cluster.https_address value. My working theory is that the address 0 comes from an upgrade from the time we where using hashicorp/raft.
Not sure if this helps but it was after ID 13 when I lost everything.
jeff@cojoserver:~$ snap changes
jeff@cojoserver:~$
@cojojeff please can you make sure your LXD daemon is not running (e.g. sudo systemctl stop snap.lxd.daemon and then ps aux|grep lxd to make sure no process is left) and then try this fix:
sudo cp /var/snap/lxd/common/lxd/database/local.db /var/snap/lxd/common/lxd/database/local.db.bak.1
sudo sqlite3 /var/snap/lxd/common/lxd/database/local.db "UPDATE raft_nodes SET address='192.168.1.20:8443'"
?
and finally restart the daemon with sudo systemctl start snap.lxd.daemon.
1 Like
@stgraber I have the feeling that this commit is what is causing the regression. Is there a way to know if that was indeed included in the snap change @cojojeff pointed at? If it’s not easy, that’s ok, it’s not critical, but would help adding context.