My LXD 2.0 stopped working today, after an apt update of Ub 16.04

karjala@my-laptop:~$ enter remote:databases
Error: Error opening startup config file: “loading config file for the container failed”
error: EOF

Maybe I have to reboot? I wonder if the containers will be lifted after rebooting though, that’s why I’m a bit reluctant to do that too.

The same error appears if I first ssh into the remote machine, and then try to do anything, such as:

lxc exec databases bash

Sorry “enter” is a script I wrote that just does lxc exec remote:databases bash

Today’s apt dist-upgrade changed some lxc-* packages

Here’s the “tig --all” output of my /etc directory (I run etckeeper):

-liblxc1 2.1.1-0ubuntu1~ubuntu16.04.1~ppa1 amd64
+liblxc-common 3.0.1-0ubuntu1~16.04.1 amd64
+liblxc1 3.0.1-0ubuntu1~16.04.1 amd64
-lxcfs 2.0.8-1ubuntu2~ubuntu16.04.1~ppa1 amd64
-lxd 2.21-0ubuntu3~16.04.2 amd64
-lxd-client 2.21-0ubuntu3~16.04.2 amd64
+lxcfs 3.0.1-0ubuntu2~16.04.1 amd64
+lxd 3.0.1-0ubuntu1~16.04.2 amd64
+lxd-client 3.0.1-0ubuntu1~16.04.2 amd64

Oops… Seems that LXD got automatically upgraded from 2 to 3, without me asking? Was that supposed to happen??

Yes, you are using the backports pocket (xenial-backports) which is why you were on LXD 2.21 rather than LXD 2.0.11 (which would be the version in xenial-updates).

xenial-backports was updated to LXD 3.0 last week and will now remain in sync with the version of LXD in bionic-updates until Ubuntu 16.04 hits EOL in 2021.

Thanks for the reply!

What should I do now to make it work again? The information is all over the place, I don’t know where to search.

@stgraber ^^

I can’t find the information here:

Surely there must be a way to make it work… Please help me, someone?

Please help me, my containers are not working properly

Can you restart that container? If so, that should generate a config that your newly updated liblxc will understand.

The output of lxc config show --expanded databases would also let us confirm you don’t have anything weird (“raw.lxc” mostly) that’d fail with the upgrade.

This is the output of the command you requested. Does this mean I can safely restart?

root@abc ~ # lxc config show --expanded databases
architecture: x86_64
  image.architecture: amd64
  image.description: ubuntu 16.04 LTS amd64 (release) (20170619.1)
  image.label: release
  image.os: ubuntu
  image.release: xenial
  image.serial: "20170619.1"
  image.version: "16.04"
  volatile.base_image: blablablablablablablablabla
  volatile.eth0.hwaddr: 00:16:3e:7a:aa:de eth0
  volatile.idmap.base: "0" '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
    nictype: bridged
    parent: lxdbr0
    type: nic
    path: /
    pool: default
    type: disk
ephemeral: false
- default
stateful: false
description: ""

Yeah, not seeing any weird config in there. What does lxc restart databases get you?

root@abc ~ # lxc restart databases
Error: shutting down the container failed

What if I restarted the whole machine?

Should I reboot the host itself?

A restart of the machine would fix it, yes.

I’m just trying to figure out what’s still wrong as this particular issue should have been addressed by a recent fix by @brauner.

Before you reboot, can you please include:

  • lxc info
  • cat /var/log/lxd/databases/lxc.conf
  • cat /var/log/lxd/databases/lxc.log

That should help us reproduce the upgrade issue and see why @brauner’s fix wasn’t sufficient in your case.

root@abc ~ # lxc info
  core.https_address: '[::]:8443'
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
api_status: stable
api_version: "1.0"
auth: trusted
public: false
- tls
  - '[not::telling::you]:8443'
  - x86_64
  - i686
  certificate: |
    -----END CERTIFICATE-----
  certificate_fingerprint: abcdef
  driver: lxc
  driver_version: 3.0.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.13.0-45-generic
  server: lxd
  server_pid: 15184
  server_version: 3.0.1
  storage: zfs
  server_clustered: false
  server_name: immense
root@abc ~ # cat /var/log/lxd/databases/lxc.conf
lxc.logfile = /var/log/lxd/databases/lxc.log
lxc.log.level = warn
lxc.cap.drop = sys_time sys_module sys_rawio = proc:rw sys:rw
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional
lxc.include = /usr/share/lxc/config/common.conf.d/
lxc.arch = linux64
lxc.hook.pre-start = /usr/bin/lxd callhook /var/lib/lxd 5 start = /usr/bin/lxd callhook /var/lib/lxd 5 stop
lxc.tty.max = 0 = databases
lxc.mount.entry = /var/lib/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.apparmor.profile = lxd-databases_</var/lib/lxd>//&:lxd-databases_<var-lib-lxd>:
lxc.seccomp.profile = /var/lib/lxd/security/seccomp/databases
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.rootfs.path = dir:/var/lib/lxd/containers/databases/rootfs = veth = up = lxdbr0 = abcdef = eth0
lxc.mount.entry = /var/lib/lxd/shmounts/databases dev/.lxd-mounts none bind,create=dir 0 0