After upgrade LXD 2.0.11 > 3.0.3 on Ubuntu server 16.04.5 my containers won’t start anymore.
The zfs pool is present but LXD is not configured to use it anymore. It looks like the database is purged. Perhaps the upgrade is designed to do this but I realy like to restore the containers!
server:~$ sudo lxd import DC1
Error: open /var/lib/lxd/storage-pools/lxd/containers/DC1/backup.yaml: no such file or directory
Running LXD init won’t let me add an existing zfs pool:
server:~$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: n
Do you want to configure a new storage pool? (yes/no) [default=yes]: n
Would you like to connect to a MAAS server? (yes/no) [default=no]: n
Would you like to create a new local network bridge? (yes/no) [default=yes]: n
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: y
Name of the existing bridge or host interface: br0
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:
server:~$ lxc config show
config: {}
server:~$ lxc profile show default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
name: default
used_by: []
Any help to restore my containers is greatly appreciated
Hmm, that’s weird, it’s an upgrade path we test daily so I’m unsure what’s caused problems in your case.
/var/log/lxd/lxd.log may have some more details (also look for the rotated ones, like lxd.log.1)
Can you show sudo find /var/lib/lxd/ -mount?
The upgrade does make a copy of the old database, so even if things went very badly, it should be possible to clean things up and manually re-trigger the update.
lxd import above didn’t work because LXD 2.0.x containers don’t contain the backup data that’s needed, this was introduced in a later 2.x release. That’s not to say they can’t be imported, but it’d be a much more manual process. Getting the database to behave should be considerably easier.
Well, the upgrade did not give errors. But I did not stop the containers before the upgrade. This led me in the situation that the server (LXD-host) could not reboot. It did hang at closing the LXC containers. I forced a hard reset at that moment.
server:~$ sudo apt install -t xenial-backports lxd lxd-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
ebtables liblxc-common liblxc1 libuv1 lxcfs xdelta3
Suggested packages:
criu lxd-tools
The following packages will be REMOVED:
lxc-common
The following NEW packages will be installed:
ebtables liblxc-common libuv1 xdelta3
The following packages will be upgraded:
liblxc1 lxcfs lxd lxd-client
4 upgraded, 4 newly installed, 1 to remove and 0 not upgraded.
Need to get 9,186 kB of archives.
After this operation, 6,981 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 liblxc1 amd64 3.0.3-0ubuntu1~16.04.1 [264 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 liblxc-common amd64 3.0.3-0ubuntu1~16.04.1 [110 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ebtables amd64 2.0.10.4-3.4ubuntu2.16.04.2 [79.9 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 libuv1 amd64 1.8.0-1 [57.4 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 lxcfs amd64 3.0.3-0ubuntu1~16.04.1 [38.6 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 lxd amd64 3.0.3-0ubuntu1~16.04.1 [5,542 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 lxd-client amd64 3.0.3-0ubuntu1~16.04.1 [3,027 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 xdelta3 amd64 3.0.8-dfsg-1ubuntu2 [67.1 kB]
Fetched 9,186 kB in 1s (5,906 kB/s)
(Reading database ... 93684 files and directories currently installed.)
Preparing to unpack .../liblxc1_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Unpacking liblxc1 (3.0.3-0ubuntu1~16.04.1) over (2.0.8-0ubuntu1~16.04.2) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
(Reading database ... 93684 files and directories currently installed.)
Removing lxc-common (2.0.8-0ubuntu1~16.04.2) ...
Selecting previously unselected package liblxc-common.
(Reading database ... 93663 files and directories currently installed.)
Preparing to unpack .../liblxc-common_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Unpacking liblxc-common (3.0.3-0ubuntu1~16.04.1) ...
Selecting previously unselected package ebtables.
Preparing to unpack .../ebtables_2.0.10.4-3.4ubuntu2.16.04.2_amd64.deb ...
Unpacking ebtables (2.0.10.4-3.4ubuntu2.16.04.2) ...
Selecting previously unselected package libuv1:amd64.
Preparing to unpack .../libuv1_1.8.0-1_amd64.deb ...
Unpacking libuv1:amd64 (1.8.0-1) ...
Preparing to unpack .../lxcfs_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Unpacking lxcfs (3.0.3-0ubuntu1~16.04.1) over (2.0.8-0ubuntu1~16.04.2) ...
Preparing to unpack .../lxd_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Warning: Stopping lxd.service, but it can still be activated by:
lxd.socket
Unpacking lxd (3.0.3-0ubuntu1~16.04.1) over (2.0.11-0ubuntu1~16.04.4) ...
Warning: Stopping lxd.service, but it can still be activated by:
lxd.socket
Preparing to unpack .../lxd-client_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Unpacking lxd-client (3.0.3-0ubuntu1~16.04.1) over (2.0.11-0ubuntu1~16.04.4) ...
Selecting previously unselected package xdelta3.
Preparing to unpack .../xdelta3_3.0.8-dfsg-1ubuntu2_amd64.deb ...
Unpacking xdelta3 (3.0.8-dfsg-1ubuntu2) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.10) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up ebtables (2.0.10.4-3.4ubuntu2.16.04.2) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up libuv1:amd64 (1.8.0-1) ...
Setting up lxcfs (3.0.3-0ubuntu1~16.04.1) ...
Setting up lxd-client (3.0.3-0ubuntu1~16.04.1) ...
Setting up xdelta3 (3.0.8-dfsg-1ubuntu2) ...
Setting up liblxc1 (3.0.3-0ubuntu1~16.04.1) ...
Setting up liblxc-common (3.0.3-0ubuntu1~16.04.1) ...
Installing new version of config file /etc/apparmor.d/abstractions/lxc/container-base ...
Installing new version of config file /etc/apparmor.d/abstractions/lxc/start-container ...
Installing new version of config file /etc/apparmor.d/lxc/lxc-default-cgns ...
Installing new version of config file /etc/apparmor.d/lxc/lxc-default-with-nesting ...
Setting up lxd (3.0.3-0ubuntu1~16.04.1) ...
Installing new version of config file /etc/init.d/lxd ...
Removing obsolete conffile /etc/init/lxd.conf ...
Setting up lxd dnsmasq configuration.
Old bridge configuration detected in /etc/default/lxd-bridge, upgrading
Unsetting deprecated profile options
No bridge configuration to convert
Cleaning up lxd-bridge state files
Processing triggers for systemd (229-4ubuntu21.10) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Yes after the upgrade at 00:06h my containers were still running because I did not shut them down prior to upgrade LXD. Few minutes after 00:13:57h I went to see why it took so long to reboot and gave the system a hard reset.
Perhaps this explains the log?
Upgrade event:
2018-12-22 14:30:42 startup packages configure
2018-12-25 00:05:54 startup archives unpack
2018-12-25 00:05:55 upgrade liblxc1:amd64 2.0.8-0ubuntu1~16.04.2 3.0.3-0ubuntu1~16.04.1
2018-12-25 00:05:55 status triggers-pending libc-bin:amd64 2.23-0ubuntu10
2018-12-25 00:05:55 status half-configured liblxc1:amd64 2.0.8-0ubuntu1~16.04.2
2018-12-25 00:05:55 status unpacked liblxc1:amd64 2.0.8-0ubuntu1~16.04.2
….
2018-12-25 00:06:12 status installed lxd:amd64 3.0.3-0ubuntu1~16.04.1
2018-12-25 00:06:12 trigproc systemd:amd64 229-4ubuntu21.10 <none>
2018-12-25 00:06:12 status half-configured systemd:amd64 229-4ubuntu21.10
2018-12-25 00:06:12 status installed systemd:amd64 229-4ubuntu21.10
2018-12-25 00:06:12 trigproc ureadahead:amd64 0.100.0-19 <none>
2018-12-25 00:06:12 status half-configured ureadahead:amd64 0.100.0-19
2018-12-25 00:06:12 status installed ureadahead:amd64 0.100.0-19
2018-12-25 00:06:12 trigproc libc-bin:amd64 2.23-0ubuntu10 <none>
2018-12-25 00:06:12 status half-configured libc-bin:amd64 2.23-0ubuntu10
2018-12-25 00:06:12 status installed libc-bin:amd64 2.23-0ubuntu10
2018-12-25 00:06:12 startup packages configure
Reboot event:
server:~$ last -x | less
user pts/0 192.168.0.109 Wed Dec 26 13:04 still logged in
runlevel (to lvl 5) 4.4.0-141-generi Wed Dec 26 13:04 still running
reboot system boot 4.4.0-141-generi Wed Dec 26 13:04 still running
shutdown system down 4.4.0-141-generi Tue Dec 25 16:01 - 13:04 (21:02)
user tty1 Tue Dec 25 16:00 - down (00:01)
user pts/0 192.168.0.109 Tue Dec 25 08:29 - 12:53 (04:23)
runlevel (to lvl 5) 4.4.0-141-generi Tue Dec 25 08:27 - 16:01 (07:34)
reboot system boot 4.4.0-141-generi Tue Dec 25 08:27 - 16:01 (07:34)
user tty1 Tue Dec 25 03:25 - crash (05:02)
user pts/0 192.168.0.109 Tue Dec 25 00:14 - 03:16 (03:01)
runlevel (to lvl 5) 4.4.0-141-generi Tue Dec 25 00:13 - 08:27 (08:13)
reboot system boot 4.4.0-141-generi Tue Dec 25 00:13 - 16:01 (15:48)
user pts/7 192.168.0.109 Mon Dec 24 20:56 - 00:10 (03:14)
I started over a year ago with my first container and installed an basic configuration with zfs.
The zfs drive is a separate partition on the host. Nothing special…
This has been sorted out offline. The issue appears to be that LXD upgraded properly and everything worked just fine post-upgrade, but the newly upgraded database failed to sync to disk during host shutdown, leading to it being corrupted.
Thankfully the automated pre-upgrade database backup was intact and so we managed to replay the upgrade, getting things working just fine again.
The openvpn container in LXD was not passing traffic to openvpn clients any more after the upgrade from 2.0.11 to 3.0.3. The openvpn clients were connecting but browsing was not possible! Me being noob and struggled for days to find what was going on.
Somehow the kernel module ip_tables on the host was not passed to the container any more. After adding the module manually to the config from the container it functioned again. Probably this is by design and not a bug in LXD but I had to install openvpn from scratch to find out UFW was not working.
root@VPN3:~# ufw allow 1194/udp
ERROR: initcaps
[Errno 2] modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-141-generic/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0-141-generic
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
Hopefully this note will help others like me
To fix this I did: lxc config edit <contaner name>
and added this line:
config:
...
linux.kernel_modules: ip_tables
...
But we are not out of the woods… yet. Openvpn clients are not able to access my private webservices. I had to add this config line to all my containers!