Lost database after upgrade LXD 2.0.11 > 3.0.3


(LXDBlast) #1

After upgrade LXD 2.0.11 > 3.0.3 on Ubuntu server 16.04.5 my containers won’t start anymore.

The zfs pool is present but LXD is not configured to use it anymore. It looks like the database is purged. Perhaps the upgrade is designed to do this but I realy like to restore the containers!

server:~$ sudo zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
lxd   59.5G  10.5G  49.0G         -    29%    17%  1.00x  ONLINE  -

server:~$ sudo zfs list
NAME                                                                          USED  AVAIL  REFER  MOUNTPOINT
lxd                                                                          10.5G  47.2G    19K  none
lxd/containers                                                               7.27G  47.2G    19K  none
lxd/containers/DC1                                                            725M  47.2G   586M  /var/lib/lxd/storage-pools/lxd/containers/DC1
lxd/containers/DOC1                                                          1.75G  47.2G  1.12G  /var/lib/lxd/storage-pools/lxd/containers/DOC1
lxd/containers/DOM1                                                           920M  47.2G  1.04G  /var/lib/lxd/storage-pools/lxd/containers/DOM1
lxd/containers/MP1                                                            823M  47.2G   574M  /var/lib/lxd/storage-pools/lxd/containers/MP1
lxd/containers/NAS1                                                          1.18G  47.2G   922M  /var/lib/lxd/storage-pools/lxd/containers/NAS1
lxd/containers/NC1                                                           1.09G  47.2G   810M  /var/lib/lxd/storage-pools/lxd/containers/NC1
lxd/containers/VPN1                                                           439M  47.2G   409M  /var/lib/lxd/storage-pools/lxd/containers/VPN1
lxd/containers/VPN2                                                           431M  47.2G   406M  /var/lib/lxd/storage-pools/lxd/containers/VPN2
lxd/images                                                                   3.17G  47.2G    19K  none
lxd/images/00847ef8c06ebd000cccb94f7b7b58e57371fe2d6b6c97efe9fd8ab9b1dbb1f2   683M  47.2G   683M  none
lxd/images/2154f572caa16c83952901e6b5489baf8a18da5598603ba04834eb221941f4bc   414M  47.2G   414M  none
lxd/images/62f5ee6a7a3a1a07bc99dacc3c21e56933843fe5f445631fbe584fc3b944e5d1   415M  47.2G   415M  none
lxd/images/dceed4767ee8f9b662ebe9fb5a14ec609603aedac85a0e6632625e63ad62f68e   722M  47.2G   722M  none
lxd/images/e81ebee647abdc4499cd791d41d7b617657cb6701240ece21d398ba8478c14da   423M  47.2G   423M  none
lxd/images/fb1ba7cdfad1d9f2e10513bf00ca9085954bf8c8f108bb1017653ba4de6208de   593M  47.2G   593M  none

server:~$ sudo zpool status
  pool: lxd
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sun Jun 10 00:24:20 2018
config:

        NAME         STATE     READ WRITE CKSUM
        lxd          ONLINE       0     0     0
          nvme0n1p8  ONLINE       0     0     0

But the containers configurations are lost:

server:~$ lxc list
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:18.04

+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

server:~$ lxc image list
+-------+-------------+--------+-------------+------+------+-------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE |
+-------+-------------+--------+-------------+------+------+-------------+

server:~$ lxc storage list
+------+-------------+--------+--------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+------+-------------+--------+--------+---------+

server:~$ lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- candid_authentication
- candid_config
- candid_config_key
- usb_optional_vendorid
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFQDCCGLmdquespda53FeXnjB
    removed
    A5kveQ==
    -----END CERTIFICATE-----
  certificate_fingerprint: 0c2abedaa47bdeae27026cb8f388eabe5d598a5cab17e184797c25faaf6cca50
  driver: lxc
  driver_version: 3.0.3
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.4.0-141-generic
  server: lxd
  server_pid: 2410
  server_version: 3.0.3
  storage: ""
  storage_version: ""
  server_clustered: false
  server_name: server
  project: ""

Issue is my storage is not set, how to set this?

I succesfully mounted the containes:

server:~$ sudo zfs mount lxd/containers/DC1

And tried to import the containers, but no luck:

server:~$ sudo lxd import DC1
Error: open /var/lib/lxd/storage-pools/lxd/containers/DC1/backup.yaml: no such file or directory

Running LXD init won’t let me add an existing zfs pool:

server:~$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: n
Do you want to configure a new storage pool? (yes/no) [default=yes]: n
Would you like to connect to a MAAS server? (yes/no) [default=no]: n
Would you like to create a new local network bridge? (yes/no) [default=yes]: n
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]: y
Name of the existing bridge or host interface: br0
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

server:~$ lxc config show
config: {}

server:~$ lxc profile show default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
name: default
used_by: []

Any help to restore my containers is greatly appreciated


(Stéphane Graber) #2

Hmm, that’s weird, it’s an upgrade path we test daily so I’m unsure what’s caused problems in your case.

/var/log/lxd/lxd.log may have some more details (also look for the rotated ones, like lxd.log.1)

Can you show sudo find /var/lib/lxd/ -mount?

The upgrade does make a copy of the old database, so even if things went very badly, it should be possible to clean things up and manually re-trigger the update.

lxd import above didn’t work because LXD 2.0.x containers don’t contain the backup data that’s needed, this was introduced in a later 2.x release. That’s not to say they can’t be imported, but it’d be a much more manual process. Getting the database to behave should be considerably easier.


(LXDBlast) #3

Well, the upgrade did not give errors. But I did not stop the containers before the upgrade. This led me in the situation that the server (LXD-host) could not reboot. It did hang at closing the LXC containers. I forced a hard reset at that moment.

server:~$ sudo apt install -t xenial-backports lxd lxd-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ebtables liblxc-common liblxc1 libuv1 lxcfs xdelta3
Suggested packages:
  criu lxd-tools
The following packages will be REMOVED:
  lxc-common
The following NEW packages will be installed:
  ebtables liblxc-common libuv1 xdelta3
The following packages will be upgraded:
  liblxc1 lxcfs lxd lxd-client
4 upgraded, 4 newly installed, 1 to remove and 0 not upgraded.
Need to get 9,186 kB of archives.
After this operation, 6,981 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 liblxc1 amd64 3.0.3-0ubuntu1~16.04.1 [264 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 liblxc-common amd64 3.0.3-0ubuntu1~16.04.1 [110 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ebtables amd64 2.0.10.4-3.4ubuntu2.16.04.2 [79.9 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 libuv1 amd64 1.8.0-1 [57.4 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 lxcfs amd64 3.0.3-0ubuntu1~16.04.1 [38.6 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 lxd amd64 3.0.3-0ubuntu1~16.04.1 [5,542 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu xenial-backports/main amd64 lxd-client amd64 3.0.3-0ubuntu1~16.04.1 [3,027 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 xdelta3 amd64 3.0.8-dfsg-1ubuntu2 [67.1 kB]
Fetched 9,186 kB in 1s (5,906 kB/s)
(Reading database ... 93684 files and directories currently installed.)
Preparing to unpack .../liblxc1_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Unpacking liblxc1 (3.0.3-0ubuntu1~16.04.1) over (2.0.8-0ubuntu1~16.04.2) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
(Reading database ... 93684 files and directories currently installed.)
Removing lxc-common (2.0.8-0ubuntu1~16.04.2) ...
Selecting previously unselected package liblxc-common.
(Reading database ... 93663 files and directories currently installed.)
Preparing to unpack .../liblxc-common_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Unpacking liblxc-common (3.0.3-0ubuntu1~16.04.1) ...
Selecting previously unselected package ebtables.
Preparing to unpack .../ebtables_2.0.10.4-3.4ubuntu2.16.04.2_amd64.deb ...
Unpacking ebtables (2.0.10.4-3.4ubuntu2.16.04.2) ...
Selecting previously unselected package libuv1:amd64.
Preparing to unpack .../libuv1_1.8.0-1_amd64.deb ...
Unpacking libuv1:amd64 (1.8.0-1) ...
Preparing to unpack .../lxcfs_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Unpacking lxcfs (3.0.3-0ubuntu1~16.04.1) over (2.0.8-0ubuntu1~16.04.2) ...
Preparing to unpack .../lxd_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
Unpacking lxd (3.0.3-0ubuntu1~16.04.1) over (2.0.11-0ubuntu1~16.04.4) ...
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
Preparing to unpack .../lxd-client_3.0.3-0ubuntu1~16.04.1_amd64.deb ...
Unpacking lxd-client (3.0.3-0ubuntu1~16.04.1) over (2.0.11-0ubuntu1~16.04.4) ...
Selecting previously unselected package xdelta3.
Preparing to unpack .../xdelta3_3.0.8-dfsg-1ubuntu2_amd64.deb ...
Unpacking xdelta3 (3.0.8-dfsg-1ubuntu2) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.10) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up ebtables (2.0.10.4-3.4ubuntu2.16.04.2) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up libuv1:amd64 (1.8.0-1) ...
Setting up lxcfs (3.0.3-0ubuntu1~16.04.1) ...
Setting up lxd-client (3.0.3-0ubuntu1~16.04.1) ...
Setting up xdelta3 (3.0.8-dfsg-1ubuntu2) ...
Setting up liblxc1 (3.0.3-0ubuntu1~16.04.1) ...
Setting up liblxc-common (3.0.3-0ubuntu1~16.04.1) ...
Installing new version of config file /etc/apparmor.d/abstractions/lxc/container-base ...
Installing new version of config file /etc/apparmor.d/abstractions/lxc/start-container ...
Installing new version of config file /etc/apparmor.d/lxc/lxc-default-cgns ...
Installing new version of config file /etc/apparmor.d/lxc/lxc-default-with-nesting ...
Setting up lxd (3.0.3-0ubuntu1~16.04.1) ...
Installing new version of config file /etc/init.d/lxd ...
Removing obsolete conffile /etc/init/lxd.conf ...
Setting up lxd dnsmasq configuration.

Old bridge configuration detected in /etc/default/lxd-bridge, upgrading
Unsetting deprecated profile options
No bridge configuration to convert
Cleaning up lxd-bridge state files

Processing triggers for systemd (229-4ubuntu21.10) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...

The output of of all mounted is:

server:~$ sudo find /var/lib/lxd/ -mount
[sudo] password for ramon:
/var/lib/lxd/
/var/lib/lxd/snapshots
/var/lib/lxd/images
/var/lib/lxd/server.key
/var/lib/lxd/devices
/var/lib/lxd/devices/NAS1
/var/lib/lxd/devices/NAS1/disk.md0.storage0
/var/lib/lxd/devices/DC1
/var/lib/lxd/devices/VPN2
/var/lib/lxd/devices/MP1
/var/lib/lxd/devices/MP1/disk.md4.storage0-music
/var/lib/lxd/devices/MP1/disk.md3.storage0-photos
/var/lib/lxd/devices/MP1/disk.md1.storage0-movies
/var/lib/lxd/devices/MP1/disk.md2.storage0-emby-tmp
/var/lib/lxd/devices/MP1/disk.md0.storage0-encode
/var/lib/lxd/devices/MP1/unix.-dev-dri-renderD128.dev-dri-renderD128
/var/lib/lxd/devices/DOC1
/var/lib/lxd/devices/NC1
/var/lib/lxd/devices/DOM1
/var/lib/lxd/devices/VPN1
/var/lib/lxd/devlxd
/var/lib/lxd/networks
/var/lib/lxd/unix.socket
/var/lib/lxd/security
/var/lib/lxd/security/apparmor
/var/lib/lxd/security/apparmor/profiles
/var/lib/lxd/security/apparmor/profiles/lxd-VPN1
/var/lib/lxd/security/apparmor/profiles/lxd-MP1
/var/lib/lxd/security/apparmor/profiles/lxd-DC1
/var/lib/lxd/security/apparmor/profiles/lxd-DOC1
/var/lib/lxd/security/apparmor/profiles/lxd-NAS1
/var/lib/lxd/security/apparmor/profiles/lxd-DOM1
/var/lib/lxd/security/apparmor/profiles/lxd-VPN2
/var/lib/lxd/security/apparmor/profiles/lxd-NC1
/var/lib/lxd/security/apparmor/cache
/var/lib/lxd/security/apparmor/cache/lxd-VPN1
/var/lib/lxd/security/apparmor/cache/lxd-MP1
/var/lib/lxd/security/apparmor/cache/lxd-DC1
/var/lib/lxd/security/apparmor/cache/.features
/var/lib/lxd/security/apparmor/cache/lxd-DOC1
/var/lib/lxd/security/apparmor/cache/lxd-NAS1
/var/lib/lxd/security/apparmor/cache/lxd-DOM1
/var/lib/lxd/security/apparmor/cache/lxd-VPN2
/var/lib/lxd/security/apparmor/cache/lxd-NC1
/var/lib/lxd/security/seccomp
/var/lib/lxd/security/seccomp/NAS1
/var/lib/lxd/security/seccomp/DC1
/var/lib/lxd/security/seccomp/VPN2
/var/lib/lxd/security/seccomp/MP1
/var/lib/lxd/security/seccomp/DOC1
/var/lib/lxd/security/seccomp/NC1
/var/lib/lxd/security/seccomp/DOM1
/var/lib/lxd/security/seccomp/VPN1
/var/lib/lxd/containers
/var/lib/lxd/containers/NAS1
/var/lib/lxd/containers/DC1
/var/lib/lxd/containers/VPN2
/var/lib/lxd/containers/MP1
/var/lib/lxd/containers/DOC1
/var/lib/lxd/containers/NC1
/var/lib/lxd/containers/lxc-monitord.log
/var/lib/lxd/containers/DOM1
/var/lib/lxd/containers/VPN1
/var/lib/lxd/database
/var/lib/lxd/database/global
/var/lib/lxd/database/global/snapshots
/var/lib/lxd/database/global/logs.db
/var/lib/lxd/database/global/db.bin
/var/lib/lxd/database/local.db.bak
/var/lib/lxd/database/local.db
/var/lib/lxd/storage-pools
/var/lib/lxd/storage-pools/lxd
/var/lib/lxd/storage-pools/lxd/snapshots
/var/lib/lxd/storage-pools/lxd/snapshots/NAS1
/var/lib/lxd/storage-pools/lxd/snapshots/NAS1/backup20180429.nas1
/var/lib/lxd/storage-pools/lxd/snapshots/DC1
/var/lib/lxd/storage-pools/lxd/snapshots/DC1/backup20180429.dc1
/var/lib/lxd/storage-pools/lxd/snapshots/VPN2
/var/lib/lxd/storage-pools/lxd/snapshots/VPN2/backup20180429.vpn2
/var/lib/lxd/storage-pools/lxd/snapshots/MP1
/var/lib/lxd/storage-pools/lxd/snapshots/MP1/backup20180520.mp1.emby-3.4.1.0
/var/lib/lxd/storage-pools/lxd/snapshots/DOC1
/var/lib/lxd/storage-pools/lxd/snapshots/DOC1/backup20180429.doc1
/var/lib/lxd/storage-pools/lxd/snapshots/NC1
/var/lib/lxd/storage-pools/lxd/snapshots/NC1/backup20180429.nc1
/var/lib/lxd/storage-pools/lxd/snapshots/VPN1
/var/lib/lxd/storage-pools/lxd/snapshots/VPN1/backup20171001.vpn1
/var/lib/lxd/storage-pools/lxd/snapshots/VPN1/backup20180429.vpn1
/var/lib/lxd/storage-pools/lxd/images
/var/lib/lxd/storage-pools/lxd/images/00847ef8c06ebd000cccb94f7b7b58e57371fe2d6b6c97efe9fd8ab9b1dbb1f2
/var/lib/lxd/storage-pools/lxd/images/fb1ba7cdfad1d9f2e10513bf00ca9085954bf8c8f108bb1017653ba4de6208de
/var/lib/lxd/storage-pools/lxd/images/62f5ee6a7a3a1a07bc99dacc3c21e56933843fe5f445631fbe584fc3b944e5d1
/var/lib/lxd/storage-pools/lxd/images/dceed4767ee8f9b662ebe9fb5a14ec609603aedac85a0e6632625e63ad62f68e
/var/lib/lxd/storage-pools/lxd/images/e81ebee647abdc4499cd791d41d7b617657cb6701240ece21d398ba8478c14da
/var/lib/lxd/storage-pools/lxd/images/2154f572caa16c83952901e6b5489baf8a18da5598603ba04834eb221941f4bc
/var/lib/lxd/storage-pools/lxd/containers
/var/lib/lxd/storage-pools/lxd/containers/NAS1
/var/lib/lxd/storage-pools/lxd/containers/DC1
/var/lib/lxd/storage-pools/lxd/containers/VPN2
/var/lib/lxd/storage-pools/lxd/containers/MP1
/var/lib/lxd/storage-pools/lxd/containers/DOC1
/var/lib/lxd/storage-pools/lxd/containers/NC1
/var/lib/lxd/storage-pools/lxd/containers/DOM1
/var/lib/lxd/storage-pools/lxd/containers/VPN1
/var/lib/lxd/server.crt
/var/lib/lxd/disks
/var/lib/lxd/shmounts

(LXDBlast) #4

The log:

server:/var/log/lxd# ls
DC1  DOC1  DOM1  lxd.log  lxd.log.1  MP1  NAS1  NC1  VPN1  VPN2
root@server:/var/log/lxd# cat lxd.log
lvl=info msg="Updating images" t=2018-12-24T08:40:48+0100
lvl=info msg="Done updating images" t=2018-12-24T08:40:48+0100
lvl=info msg="Expiring log files" t=2018-12-24T14:40:48+0100
lvl=info msg="Done expiring log files" t=2018-12-24T14:40:48+0100
lvl=info msg="Pruning expired images" t=2018-12-24T14:40:48+0100
lvl=info msg="Done pruning expired images" t=2018-12-24T14:40:48+0100
lvl=info msg="Updating images" t=2018-12-24T14:40:48+0100
lvl=info msg="Done updating images" t=2018-12-24T14:40:48+0100
lvl=info msg="Updating instance types" t=2018-12-24T14:40:55+0100
lvl=info msg="Done updating instance types" t=2018-12-24T14:40:59+0100
lvl=info msg="Updating images" t=2018-12-24T20:40:48+0100
lvl=info msg="Done updating images" t=2018-12-24T20:40:48+0100
lvl=info msg="Received 'terminated signal', exiting." t=2018-12-25T00:05:55+0100
lvl=info msg="Stopping REST API handler:" t=2018-12-25T00:05:55+0100
lvl=info msg=" - skipping socket-activated socket" socket=/var/lib/lxd/unix.socket t=2018-12-25T00:05:55+0100
lvl=info msg="Stopping /dev/lxd handler" t=2018-12-25T00:05:55+0100
lvl=info msg="Stopped /dev/lxd handler" t=2018-12-25T00:05:55+0100
lvl=info msg="Closing the database" t=2018-12-25T00:05:55+0100
lvl=info msg="Saving simplestreams cache" t=2018-12-25T00:05:55+0100
lvl=info msg="Saved simplestreams cache" t=2018-12-25T00:05:55+0100
t=2018-12-25T00:05:59+0100 lvl=info msg="LXD 3.0.3 is starting in normal mode" path=/var/lib/lxd
t=2018-12-25T00:05:59+0100 lvl=info msg="Kernel uid/gid map:"
t=2018-12-25T00:05:59+0100 lvl=info msg=" - u 0 0 4294967295"
t=2018-12-25T00:05:59+0100 lvl=info msg=" - g 0 0 4294967295"
t=2018-12-25T00:05:59+0100 lvl=info msg="Configured LXD uid/gid map:"
t=2018-12-25T00:05:59+0100 lvl=info msg=" - u 0 100000 65536"
t=2018-12-25T00:05:59+0100 lvl=info msg=" - g 0 100000 65536"
t=2018-12-25T00:05:59+0100 lvl=info msg="Kernel features:"
t=2018-12-25T00:05:59+0100 lvl=info msg=" - netnsid-based network retrieval: no"
t=2018-12-25T00:05:59+0100 lvl=info msg=" - unprivileged file capabilities: yes"
t=2018-12-25T00:05:59+0100 lvl=info msg="Initializing local database"
t=2018-12-25T00:05:59+0100 lvl=info msg="Renaming local database file from lxd.db to database/local.db"
t=2018-12-25T00:05:59+0100 lvl=info msg="Updating the LXD database schema. Backup made as \"local.db.bak\""
t=2018-12-25T00:05:59+0100 lvl=info msg="Starting /dev/lxd handler:"
t=2018-12-25T00:05:59+0100 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2018-12-25T00:05:59+0100 lvl=info msg="REST API daemon:"
t=2018-12-25T00:05:59+0100 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/lib/lxd/unix.socket
t=2018-12-25T00:05:59+0100 lvl=info msg="Initializing global database"
t=2018-12-25T00:05:59+0100 lvl=info msg="Migrating data from local to global database"
t=2018-12-25T00:05:59+0100 lvl=info msg="Initializing storage pools"
t=2018-12-25T00:05:59+0100 lvl=info msg="Applying patch: network_permissions"
t=2018-12-25T00:05:59+0100 lvl=info msg="Applying patch: storage_api"
t=2018-12-25T00:06:09+0100 lvl=warn msg="Failed to unmount ZFS filesystem via zfs unmount, trying lazy umount (MNT_DETACH)..."
t=2018-12-25T00:06:10+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DC1
t=2018-12-25T00:06:10+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DOC1
t=2018-12-25T00:06:10+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DOM1
t=2018-12-25T00:06:10+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=MP1
t=2018-12-25T00:06:10+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=NAS1
t=2018-12-25T00:06:10+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=NC1
t=2018-12-25T00:06:10+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=VPN1
t=2018-12-25T00:06:10+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=VPN2
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_api_v1"
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_api_dir_cleanup"
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_api_lvm_keys"
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_api_keys"
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_api_update_storage_configs"
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_api_lxd_on_btrfs"
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_api_lvm_detect_lv_size"
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_api_insert_zfs_driver"
t=2018-12-25T00:06:10+0100 lvl=info msg="Applying patch: storage_zfs_noauto"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: storage_zfs_volume_size"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: network_dnsmasq_hosts"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: storage_api_dir_bind_mount"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: storage_api_ceph_size_remove"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: devices_new_naming_scheme"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: storage_api_permissions"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: container_config_regen"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: lvm_node_specific_config_keys"
t=2018-12-25T00:06:11+0100 lvl=info msg="Applying patch: candid_rename_config_key"
t=2018-12-25T00:06:11+0100 lvl=info msg="Initializing networks"
t=2018-12-25T00:06:11+0100 lvl=info msg="Pruning leftover image files"
t=2018-12-25T00:06:11+0100 lvl=info msg="Done pruning leftover image files"
t=2018-12-25T00:06:11+0100 lvl=info msg="Loading daemon configuration"
t=2018-12-25T00:06:11+0100 lvl=info msg="Pruning expired images"
t=2018-12-25T00:06:11+0100 lvl=info msg="Done pruning expired images"
t=2018-12-25T00:06:11+0100 lvl=info msg="Expiring log files"
t=2018-12-25T00:06:11+0100 lvl=info msg="Done expiring log files"
t=2018-12-25T00:06:11+0100 lvl=info msg="Updating instance types"
t=2018-12-25T00:06:11+0100 lvl=info msg="Done updating instance types"
t=2018-12-25T00:06:11+0100 lvl=info msg="Updating images"
t=2018-12-25T00:06:11+0100 lvl=info msg="Done updating images"
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DC1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=NAS1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=NC1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=VPN1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=VPN2
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DOC1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=MP1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DOM1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DC1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=NAS1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=NC1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=VPN1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=VPN2
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DOC1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=MP1
t=2018-12-25T00:06:11+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DOM1
t=2018-12-25T00:06:11+0100 lvl=info msg="Received 'terminated signal', exiting"
t=2018-12-25T00:06:11+0100 lvl=info msg="Starting shutdown sequence"
t=2018-12-25T00:06:11+0100 lvl=info msg="Stopping REST API handler:"
t=2018-12-25T00:06:11+0100 lvl=info msg=" - closing socket" socket=/var/lib/lxd/unix.socket
t=2018-12-25T00:06:11+0100 lvl=info msg="Stopping /dev/lxd handler"
t=2018-12-25T00:06:11+0100 lvl=info msg=" - closing socket" socket=/var/lib/lxd/devlxd/sock
t=2018-12-25T00:06:11+0100 lvl=warn msg="Failed to update instance types: Get https://images.linuxcontainers.org/meta/instance-types/.yaml: context canceled"
t=2018-12-25T00:06:11+0100 lvl=info msg="Closing the database"
t=2018-12-25T00:06:11+0100 lvl=info msg="Stopping REST API handler:"
t=2018-12-25T00:06:11+0100 lvl=info msg="Stopping /dev/lxd handler"
t=2018-12-25T00:06:11+0100 lvl=info msg="Stopping REST API handler:"
t=2018-12-25T00:06:11+0100 lvl=info msg="Stopping /dev/lxd handler"
t=2018-12-25T00:06:11+0100 lvl=info msg="Saving simplestreams cache"
t=2018-12-25T00:06:11+0100 lvl=info msg="Saved simplestreams cache"
t=2018-12-25T00:06:12+0100 lvl=info msg="LXD 3.0.3 is starting in normal mode" path=/var/lib/lxd
t=2018-12-25T00:06:12+0100 lvl=info msg="Kernel uid/gid map:"
t=2018-12-25T00:06:12+0100 lvl=info msg=" - u 0 0 4294967295"
t=2018-12-25T00:06:12+0100 lvl=info msg=" - g 0 0 4294967295"
t=2018-12-25T00:06:12+0100 lvl=info msg="Configured LXD uid/gid map:"
t=2018-12-25T00:06:12+0100 lvl=info msg=" - u 0 100000 65536"
t=2018-12-25T00:06:12+0100 lvl=info msg=" - g 0 100000 65536"
t=2018-12-25T00:06:12+0100 lvl=info msg="Kernel features:"
t=2018-12-25T00:06:12+0100 lvl=info msg=" - netnsid-based network retrieval: no"
t=2018-12-25T00:06:12+0100 lvl=info msg=" - unprivileged file capabilities: yes"
t=2018-12-25T00:06:12+0100 lvl=info msg="Initializing local database"
t=2018-12-25T00:06:12+0100 lvl=info msg="Starting /dev/lxd handler:"
t=2018-12-25T00:06:12+0100 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2018-12-25T00:06:12+0100 lvl=info msg="REST API daemon:"
t=2018-12-25T00:06:12+0100 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/lib/lxd/unix.socket
t=2018-12-25T00:06:12+0100 lvl=info msg="Initializing global database"
t=2018-12-25T00:06:12+0100 lvl=info msg="Initializing storage pools"
t=2018-12-25T00:06:12+0100 lvl=info msg="Initializing networks"
t=2018-12-25T00:06:12+0100 lvl=info msg="Pruning leftover image files"
t=2018-12-25T00:06:12+0100 lvl=info msg="Done pruning leftover image files"
t=2018-12-25T00:06:12+0100 lvl=info msg="Loading daemon configuration"
t=2018-12-25T00:06:12+0100 lvl=info msg="Pruning expired images"
t=2018-12-25T00:06:12+0100 lvl=info msg="Done pruning expired images"
t=2018-12-25T00:06:12+0100 lvl=info msg="Updating instance types"
t=2018-12-25T00:06:12+0100 lvl=info msg="Done updating instance types"
t=2018-12-25T00:06:12+0100 lvl=info msg="Updating images"
t=2018-12-25T00:06:12+0100 lvl=info msg="Done updating images"
t=2018-12-25T00:06:12+0100 lvl=info msg="Expiring log files"
t=2018-12-25T00:06:12+0100 lvl=info msg="Done expiring log files"
t=2018-12-25T00:10:44+0100 lvl=info msg="Asked to shutdown by API, shutting down containers"
t=2018-12-25T00:10:44+0100 lvl=info msg="Shutting down container" action=shutdown created=2018-04-15T20:43:05+0200 ephemeral=false name=DOC1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Shutting down container" action=shutdown created=2018-12-22T15:16:07+0100 ephemeral=false name=DOM1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=eror msg="Failed shutting down container" action=shutdown created=2018-04-15T20:43:05+0200 ephemeral=false name=DOC1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Shutting down container" action=shutdown created=2017-11-21T11:46:43+0100 ephemeral=false name=DC1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Stopping container" action=stop created=2018-04-15T20:43:05+0200 ephemeral=false name=DOC1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Shutting down container" action=shutdown created=2017-11-21T13:21:53+0100 ephemeral=false name=VPN2 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=eror msg="Failed shutting down container" action=shutdown created=2017-11-21T11:46:43+0100 ephemeral=false name=DC1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Stopping container" action=stop created=2017-11-21T11:46:43+0100 ephemeral=false name=DC1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=eror msg="Failed shutting down container" action=shutdown created=2017-11-21T13:21:53+0100 ephemeral=false name=VPN2 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Stopping container" action=stop created=2017-11-21T13:21:53+0100 ephemeral=false name=VPN2 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Shutting down container" action=shutdown created=2018-05-05T23:13:11+0200 ephemeral=false name=MP1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Shutting down container" action=shutdown created=2017-11-21T11:48:15+0100 ephemeral=false name=NAS1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=eror msg="Failed shutting down container" action=shutdown created=2017-11-21T11:48:15+0100 ephemeral=false name=NAS1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Stopping container" action=stop created=2017-11-21T11:48:15+0100 ephemeral=false name=NAS1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=eror msg="Failed shutting down container" action=shutdown created=2018-05-05T23:13:11+0200 ephemeral=false name=MP1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Shutting down container" action=shutdown created=2017-11-21T13:21:31+0100 ephemeral=false name=VPN1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Shutting down container" action=shutdown created=2017-11-21T12:27:53+0100 ephemeral=false name=NC1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=eror msg="Failed shutting down container" action=shutdown created=2018-12-22T15:16:07+0100 ephemeral=false name=DOM1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Stopping container" action=stop created=2018-12-22T15:16:07+0100 ephemeral=false name=DOM1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=eror msg="Failed shutting down container" action=shutdown created=2017-11-21T13:21:31+0100 ephemeral=false name=VPN1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=eror msg="Failed shutting down container" action=shutdown created=2017-11-21T12:27:53+0100 ephemeral=false name=NC1 timeout=30s used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Stopping container" action=stop created=2017-11-21T13:21:31+0100 ephemeral=false name=VPN1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:44+0100 lvl=info msg="Stopping container" action=stop created=2017-11-21T12:27:53+0100 ephemeral=false name=NC1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:45+0100 lvl=info msg="Stopped container" action=stop created=2017-11-21T13:21:53+0100 ephemeral=false name=VPN2 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:45+0100 lvl=info msg="Stopped container" action=stop created=2017-11-21T12:27:53+0100 ephemeral=false name=NC1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:45+0100 lvl=info msg="Stopped container" action=stop created=2017-11-21T13:21:31+0100 ephemeral=false name=VPN1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:45+0100 lvl=info msg="Stopped container" action=stop created=2017-11-21T11:46:43+0100 ephemeral=false name=DC1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:45+0100 lvl=info msg="Stopped container" action=stop created=2018-12-22T15:16:07+0100 ephemeral=false name=DOM1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:45+0100 lvl=info msg="Stopped container" action=stop created=2018-04-15T20:43:05+0200 ephemeral=false name=DOC1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:10:45+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DC1
t=2018-12-25T00:10:45+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=NC1
t=2018-12-25T00:10:45+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=VPN2
t=2018-12-25T00:10:45+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=VPN1
t=2018-12-25T00:10:45+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DOC1
t=2018-12-25T00:10:45+0100 lvl=warn msg="Unable to update backup.yaml at this time" name=DOM1
t=2018-12-25T00:10:49+0100 lvl=info msg="Stopping container" action=stop created=2018-05-05T23:13:11+0200 ephemeral=false name=MP1 stateful=false used=0001-01-01T00:00:00+0000
t=2018-12-25T00:13:57+0100 lvl=info msg="LXD 3.0.3 is starting in normal mode" path=/var/lib/lxd
t=2018-12-25T00:13:57+0100 lvl=info msg="Kernel uid/gid map:"
t=2018-12-25T00:13:57+0100 lvl=info msg=" - u 0 0 4294967295"
t=2018-12-25T00:13:57+0100 lvl=info msg=" - g 0 0 4294967295"
t=2018-12-25T00:13:57+0100 lvl=info msg="Configured LXD uid/gid map:"
t=2018-12-25T00:13:57+0100 lvl=info msg=" - u 0 100000 65536"
t=2018-12-25T00:13:57+0100 lvl=info msg=" - g 0 100000 65536"
t=2018-12-25T00:13:57+0100 lvl=info msg="Kernel features:"
t=2018-12-25T00:13:57+0100 lvl=info msg=" - netnsid-based network retrieval: no"
t=2018-12-25T00:13:57+0100 lvl=info msg=" - unprivileged file capabilities: yes"
t=2018-12-25T00:13:57+0100 lvl=info msg="Initializing local database"
t=2018-12-25T00:13:57+0100 lvl=info msg="Starting /dev/lxd handler:"
t=2018-12-25T00:13:57+0100 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2018-12-25T00:13:57+0100 lvl=info msg="REST API daemon:"
t=2018-12-25T00:13:57+0100 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/lib/lxd/unix.socket
t=2018-12-25T00:13:57+0100 lvl=info msg="Initializing global database"
t=2018-12-25T00:13:57+0100 lvl=info msg="Initializing storage pools"
t=2018-12-25T00:13:57+0100 lvl=info msg="Initializing networks"
t=2018-12-25T00:13:57+0100 lvl=info msg="Pruning leftover image files"
t=2018-12-25T00:13:57+0100 lvl=info msg="Done pruning leftover image files"
t=2018-12-25T00:13:57+0100 lvl=info msg="Loading daemon configuration"
t=2018-12-25T00:13:57+0100 lvl=info msg="Pruning expired images"
t=2018-12-25T00:13:57+0100 lvl=info msg="Done pruning expired images"
t=2018-12-25T00:13:57+0100 lvl=info msg="Updating instance types"
t=2018-12-25T00:13:57+0100 lvl=info msg="Done updating instance types"
t=2018-12-25T00:13:57+0100 lvl=info msg="Updating images"
t=2018-12-25T00:13:57+0100 lvl=info msg="Done updating images"
t=2018-12-25T00:13:57+0100 lvl=info msg="Expiring log files"
t=2018-12-25T00:13:57+0100 lvl=info msg="Done expiring log files"
t=2018-12-25T00:48:30+0100 lvl=info msg="Received 'terminated signal', exiting"
t=2018-12-25T00:48:30+0100 lvl=info msg="Starting shutdown sequence"
t=2018-12-25T00:48:30+0100 lvl=info msg="Stopping REST API handler:"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - closing socket" socket=/var/lib/lxd/unix.socket
t=2018-12-25T00:48:30+0100 lvl=info msg="Stopping /dev/lxd handler"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - closing socket" socket=/var/lib/lxd/devlxd/sock
t=2018-12-25T00:48:30+0100 lvl=info msg="Closing the database"
t=2018-12-25T00:48:30+0100 lvl=info msg="LXD 3.0.3 is starting in normal mode" path=/var/lib/lxd
t=2018-12-25T00:48:30+0100 lvl=info msg="Kernel uid/gid map:"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - u 0 0 4294967295"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - g 0 0 4294967295"
t=2018-12-25T00:48:30+0100 lvl=info msg="Configured LXD uid/gid map:"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - u 0 100000 65536"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - g 0 100000 65536"
t=2018-12-25T00:48:30+0100 lvl=info msg="Kernel features:"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - netnsid-based network retrieval: no"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - unprivileged file capabilities: yes"
t=2018-12-25T00:48:30+0100 lvl=info msg="Initializing local database"
t=2018-12-25T00:48:30+0100 lvl=info msg="Starting /dev/lxd handler:"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2018-12-25T00:48:30+0100 lvl=info msg="REST API daemon:"
t=2018-12-25T00:48:30+0100 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/lib/lxd/unix.socket
t=2018-12-25T00:48:30+0100 lvl=info msg="Initializing global database"
t=2018-12-25T00:48:30+0100 lvl=info msg="Initializing storage pools"
t=2018-12-25T00:48:30+0100 lvl=info msg="Initializing networks"
t=2018-12-25T00:48:30+0100 lvl=info msg="Pruning leftover image files"
t=2018-12-25T00:48:30+0100 lvl=info msg="Done pruning leftover image files"
t=2018-12-25T00:48:30+0100 lvl=info msg="Loading daemon configuration"
t=2018-12-25T00:48:30+0100 lvl=info msg="Pruning expired images"
t=2018-12-25T00:48:30+0100 lvl=info msg="Done pruning expired images"
t=2018-12-25T00:48:30+0100 lvl=info msg="Updating images"
t=2018-12-25T00:48:30+0100 lvl=info msg="Done updating images"
t=2018-12-25T00:48:30+0100 lvl=info msg="Updating instance types"
t=2018-12-25T00:48:30+0100 lvl=info msg="Done updating instance types"
t=2018-12-25T00:48:30+0100 lvl=info msg="Expiring log files"
t=2018-12-25T00:48:30+0100 lvl=info msg="Done expiring log files"
t=2018-12-25T03:25:30+0100 lvl=info msg="Asked to shutdown by API, shutting down containers"
t=2018-12-25T03:25:30+0100 lvl=info msg="Starting shutdown sequence"
t=2018-12-25T03:25:30+0100 lvl=info msg="Stopping REST API handler:"
t=2018-12-25T03:25:30+0100 lvl=info msg=" - closing socket" socket=/var/lib/lxd/unix.socket
t=2018-12-25T03:25:30+0100 lvl=info msg="Stopping /dev/lxd handler"
t=2018-12-25T03:25:30+0100 lvl=info msg=" - closing socket" socket=/var/lib/lxd/devlxd/sock
t=2018-12-25T03:25:30+0100 lvl=info msg="Closing the database"
t=2018-12-25T03:25:30+0100 lvl=info msg="Stopping REST API handler:"
t=2018-12-25T03:25:30+0100 lvl=info msg="Stopping /dev/lxd handler"
t=2018-12-25T03:25:30+0100 lvl=info msg="Stopping REST API handler:"
t=2018-12-25T03:25:30+0100 lvl=info msg="Stopping /dev/lxd handler"
t=2018-12-25T03:25:30+0100 lvl=info msg="Unmounting temporary filesystems"
t=2018-12-25T03:25:30+0100 lvl=info msg="Done unmounting temporary filesystems"
t=2018-12-25T03:25:30+0100 lvl=info msg="Saving simplestreams cache"
t=2018-12-25T03:25:30+0100 lvl=info msg="Saved simplestreams cache"

Realy hope the database is usable. How should I proceed?

server:/var/lib/lxd/database# ls -la
total 168
drwx------  3 root root     4096 Dec 25 00:06 .
drwxr-xr-x 13 lxd  nogroup  4096 Dec 25 08:27 ..
drwxr-x---  3 root root     4096 Dec 25 08:27 global
-rw-r--r--  1 root root    83968 Dec 25 00:06 local.db
-rw-r--r--  1 root root    69632 Dec 25 00:05 local.db.bak

(Stéphane Graber) #5

So the log seems to indicate that at 10:49 on the 25th, the system was shutdown with a bunch of containers properly running on the upgraded LXD 3.0.3.

When the system came back up at 13:57, the database didn’t contain any containers.

Do you have a filesystem mount over /var/lib/lxd or a similar storage setup that’d explain this behavior?

Otherwise, can you send a tarball of /var/lib/lxd/database to stgraber at ubuntu dot com so I can take a look at what your database looks like?


(LXDBlast) #6

Yes after the upgrade at 00:06h my containers were still running because I did not shut them down prior to upgrade LXD. Few minutes after 00:13:57h I went to see why it took so long to reboot and gave the system a hard reset.

Perhaps this explains the log?

Upgrade event:

2018-12-22 14:30:42 startup packages configure
2018-12-25 00:05:54 startup archives unpack
2018-12-25 00:05:55 upgrade liblxc1:amd64 2.0.8-0ubuntu1~16.04.2 3.0.3-0ubuntu1~16.04.1
2018-12-25 00:05:55 status triggers-pending libc-bin:amd64 2.23-0ubuntu10
2018-12-25 00:05:55 status half-configured liblxc1:amd64 2.0.8-0ubuntu1~16.04.2
2018-12-25 00:05:55 status unpacked liblxc1:amd64 2.0.8-0ubuntu1~16.04.2
….
2018-12-25 00:06:12 status installed lxd:amd64 3.0.3-0ubuntu1~16.04.1
2018-12-25 00:06:12 trigproc systemd:amd64 229-4ubuntu21.10 <none>
2018-12-25 00:06:12 status half-configured systemd:amd64 229-4ubuntu21.10
2018-12-25 00:06:12 status installed systemd:amd64 229-4ubuntu21.10
2018-12-25 00:06:12 trigproc ureadahead:amd64 0.100.0-19 <none>
2018-12-25 00:06:12 status half-configured ureadahead:amd64 0.100.0-19
2018-12-25 00:06:12 status installed ureadahead:amd64 0.100.0-19
2018-12-25 00:06:12 trigproc libc-bin:amd64 2.23-0ubuntu10 <none>
2018-12-25 00:06:12 status half-configured libc-bin:amd64 2.23-0ubuntu10
2018-12-25 00:06:12 status installed libc-bin:amd64 2.23-0ubuntu10
2018-12-25 00:06:12 startup packages configure

Reboot event:

server:~$ last -x | less
user    pts/0        192.168.0.109    Wed Dec 26 13:04   still logged in
runlevel (to lvl 5)   4.4.0-141-generi Wed Dec 26 13:04   still running
reboot   system boot  4.4.0-141-generi Wed Dec 26 13:04   still running
shutdown system down  4.4.0-141-generi Tue Dec 25 16:01 - 13:04  (21:02)
user    tty1                          Tue Dec 25 16:00 - down   (00:01)
user    pts/0        192.168.0.109    Tue Dec 25 08:29 - 12:53  (04:23)
runlevel (to lvl 5)   4.4.0-141-generi Tue Dec 25 08:27 - 16:01  (07:34)
reboot   system boot  4.4.0-141-generi Tue Dec 25 08:27 - 16:01  (07:34)
user    tty1                          Tue Dec 25 03:25 - crash  (05:02)
user    pts/0        192.168.0.109    Tue Dec 25 00:14 - 03:16  (03:01)
runlevel (to lvl 5)   4.4.0-141-generi Tue Dec 25 00:13 - 08:27  (08:13)
reboot   system boot  4.4.0-141-generi Tue Dec 25 00:13 - 16:01  (15:48)
user    pts/7        192.168.0.109    Mon Dec 24 20:56 - 00:10  (03:14)

I started over a year ago with my first container and installed an basic configuration with zfs.
The zfs drive is a separate partition on the host. Nothing special…

I send you the requested tarball.


(Stéphane Graber) #7

This has been sorted out offline. The issue appears to be that LXD upgraded properly and everything worked just fine post-upgrade, but the newly upgraded database failed to sync to disk during host shutdown, leading to it being corrupted.

Thankfully the automated pre-upgrade database backup was intact and so we managed to replay the upgrade, getting things working just fine again.


(LXDBlast) #8

Systems are up and running. Greatly appreciate your assistance time and effort in this!


(LXDBlast) #9

Hi stgraber,

I like to give an update on the LXD upgrade.

The openvpn container in LXD was not passing traffic to openvpn clients any more after the upgrade from 2.0.11 to 3.0.3. The openvpn clients were connecting but browsing was not possible! Me being noob and struggled for days to find what was going on.

Somehow the kernel module ip_tables on the host was not passed to the container any more. After adding the module manually to the config from the container it functioned again. Probably this is by design and not a bug in LXD but I had to install openvpn from scratch to find out UFW was not working.

root@VPN3:~# ufw allow 1194/udp
ERROR: initcaps
[Errno 2] modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-141-generic/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0-141-generic
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

Hopefully this note will help others like me :wink:

To fix this I did:
lxc config edit <contaner name>

and added this line:

config:
  ...
  linux.kernel_modules: ip_tables
  ...

But we are not out of the woods… yet. Openvpn clients are not able to access my private webservices. I had to add this config line to all my containers!