karjala
(Alexander Karelas)
July 6, 2018, 4:41am
1
karjala@my-laptop:~$ enter remote:databases
Error: Error opening startup config file: “loading config file for the container failed”
error: EOF
Maybe I have to reboot? I wonder if the containers will be lifted after rebooting though, that’s why I’m a bit reluctant to do that too.
karjala
(Alexander Karelas)
July 6, 2018, 4:42am
2
The same error appears if I first ssh into the remote machine, and then try to do anything, such as:
lxc exec databases bash
karjala
(Alexander Karelas)
July 6, 2018, 4:42am
3
Sorry “enter” is a script I wrote that just does lxc exec remote:databases bash
karjala
(Alexander Karelas)
July 6, 2018, 4:43am
4
Today’s apt dist-upgrade changed some lxc-* packages
karjala
(Alexander Karelas)
July 6, 2018, 4:47am
5
Here’s the “tig --all” output of my /etc directory (I run etckeeper):
-liblxc1 2.1.1-0ubuntu1~ubuntu16.04.1~ppa1 amd64
+liblxc-common 3.0.1-0ubuntu1~16.04.1 amd64
+liblxc1 3.0.1-0ubuntu1~16.04.1 amd64
-lxcfs 2.0.8-1ubuntu2~ubuntu16.04.1~ppa1 amd64
-lxd 2.21-0ubuntu3~16.04.2 amd64
-lxd-client 2.21-0ubuntu3~16.04.2 amd64
+lxcfs 3.0.1-0ubuntu2~16.04.1 amd64
+lxd 3.0.1-0ubuntu1~16.04.2 amd64
+lxd-client 3.0.1-0ubuntu1~16.04.2 amd64
karjala
(Alexander Karelas)
July 6, 2018, 4:48am
6
Oops… Seems that LXD got automatically upgraded from 2 to 3, without me asking? Was that supposed to happen??
stgraber
(Stéphane Graber)
July 6, 2018, 5:21am
7
Yes, you are using the backports pocket (xenial-backports) which is why you were on LXD 2.21 rather than LXD 2.0.11 (which would be the version in xenial-updates).
xenial-backports was updated to LXD 3.0 last week and will now remain in sync with the version of LXD in bionic-updates until Ubuntu 16.04 hits EOL in 2021.
karjala
(Alexander Karelas)
July 6, 2018, 5:24am
8
Thanks for the reply!
What should I do now to make it work again? The information is all over the place, I don’t know where to search.
karjala
(Alexander Karelas)
July 6, 2018, 5:31am
10
I can’t find the information here: https://lxd.readthedocs.io/en/latest/#installing-lxd-from-packages
Surely there must be a way to make it work… Please help me, someone?
karjala
(Alexander Karelas)
July 6, 2018, 5:36am
11
Please help me, my containers are not working properly
stgraber
(Stéphane Graber)
July 6, 2018, 5:38am
12
Can you restart that container? If so, that should generate a config that your newly updated liblxc will understand.
The output of lxc config show --expanded databases
would also let us confirm you don’t have anything weird (“raw.lxc” mostly) that’d fail with the upgrade.
karjala
(Alexander Karelas)
July 6, 2018, 5:46am
13
This is the output of the command you requested. Does this mean I can safely restart?
root@abc ~ # lxc config show --expanded databases
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 16.04 LTS amd64 (release) (20170619.1)
image.label: release
image.os: ubuntu
image.release: xenial
image.serial: "20170619.1"
image.version: "16.04"
volatile.base_image: blablablablablablablablabla
volatile.eth0.hwaddr: 00:16:3e:7a:aa:de
volatile.eth0.name: eth0
volatile.idmap.base: "0"
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
volatile.last_state.power: RUNNING
devices:
eth0:
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
stgraber
(Stéphane Graber)
July 6, 2018, 5:48am
14
Yeah, not seeing any weird config in there. What does lxc restart databases
get you?
karjala
(Alexander Karelas)
July 6, 2018, 5:49am
15
root@abc ~ # lxc restart databases
Error: shutting down the container failed
karjala
(Alexander Karelas)
July 6, 2018, 5:50am
16
What if I restarted the whole machine?
karjala
(Alexander Karelas)
July 6, 2018, 5:51am
17
Should I reboot the host itself?
stgraber
(Stéphane Graber)
July 6, 2018, 5:53am
18
A restart of the machine would fix it, yes.
I’m just trying to figure out what’s still wrong as this particular issue should have been addressed by a recent fix by @brauner .
Before you reboot, can you please include:
lxc info
cat /var/log/lxd/databases/lxc.conf
cat /var/log/lxd/databases/lxc.log
That should help us reproduce the upgrade issue and see why @brauner ’s fix wasn’t sufficient in your case.
karjala
(Alexander Karelas)
July 6, 2018, 5:55am
19
root@abc ~ # lxc info
config:
core.https_address: '[::]:8443'
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses:
- not.telling.you:8443
- '[not::telling::you]:8443'
- not.telling.you:8443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
...............................
-----END CERTIFICATE-----
certificate_fingerprint: abcdef
driver: lxc
driver_version: 3.0.1
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.13.0-45-generic
server: lxd
server_pid: 15184
server_version: 3.0.1
storage: zfs
storage_version: 0.6.5.11-1ubuntu3.3
server_clustered: false
server_name: immense
karjala
(Alexander Karelas)
July 6, 2018, 5:57am
20
root@abc ~ # cat /var/log/lxd/databases/lxc.conf
lxc.logfile = /var/log/lxd/databases/lxc.log
lxc.log.level = warn
lxc.cap.drop = sys_time sys_module sys_rawio
lxc.mount.auto = proc:rw sys:rw
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional
lxc.include = /usr/share/lxc/config/common.conf.d/
lxc.arch = linux64
lxc.hook.pre-start = /usr/bin/lxd callhook /var/lib/lxd 5 start
lxc.hook.post-stop = /usr/bin/lxd callhook /var/lib/lxd 5 stop
lxc.tty.max = 0
lxc.uts.name = databases
lxc.mount.entry = /var/lib/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.apparmor.profile = lxd-databases_</var/lib/lxd>//&:lxd-databases_<var-lib-lxd>:
lxc.seccomp.profile = /var/lib/lxd/security/seccomp/databases
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.rootfs.path = dir:/var/lib/lxd/containers/databases/rootfs
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = lxdbr0
lxc.net.0.hwaddr = abcdef
lxc.net.0.name = eth0
lxc.mount.entry = /var/lib/lxd/shmounts/databases dev/.lxd-mounts none bind,create=dir 0 0