LXD 3.18 (snap) - unable to run CT

ygol · November 10, 2019, 8:01am

following server reboot:
“lxc list” show me all ocntainer stopped

“cat /var/snap/lxd/common/lxd/logs/lxd.log”

t=2019-11-10T07:48:03+0000 lvl=info msg="LXD 3.18 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2019-11-10T07:48:03+0000 lvl=info msg="Kernel uid/gid map:"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - u 0 0 4294967295"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - g 0 0 4294967295"
t=2019-11-10T07:48:03+0000 lvl=info msg="Configured LXD uid/gid map:"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - u 0 1000000 1000000000"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - g 0 1000000 1000000000"
t=2019-11-10T07:48:03+0000 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
t=2019-11-10T07:48:03+0000 lvl=info msg="Kernel features:"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - netnsid-based network retrieval: no"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - uevent injection: no"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - seccomp listener: no"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - unprivileged file capabilities: yes"
t=2019-11-10T07:48:03+0000 lvl=info msg=" - shiftfs support: no"
t=2019-11-10T07:48:03+0000 lvl=info msg="Initializing local database"
t=2019-11-10T07:48:04+0000 lvl=info msg="Starting /dev/lxd handler:"
t=2019-11-10T07:48:04+0000 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2019-11-10T07:48:04+0000 lvl=info msg="REST API daemon:"
t=2019-11-10T07:48:04+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2019-11-10T07:48:04+0000 lvl=info msg=" - binding TCP socket" socket=[::]:8443
t=2019-11-10T07:48:04+0000 lvl=info msg="Initializing global database"
t=2019-11-10T07:48:04+0000 lvl=info msg="Initializing storage pools"
t=2019-11-10T07:48:04+0000 lvl=info msg="Initializing networks"
t=2019-11-10T07:48:05+0000 lvl=info msg="Pruning leftover image files"
t=2019-11-10T07:48:05+0000 lvl=info msg="Done pruning leftover image files"
t=2019-11-10T07:48:05+0000 lvl=info msg="Loading daemon configuration"
t=2019-11-10T07:48:05+0000 lvl=info msg="Pruning expired images"
t=2019-11-10T07:48:05+0000 lvl=info msg="Done pruning expired images"
t=2019-11-10T07:48:05+0000 lvl=info msg="Pruning expired container backups"
t=2019-11-10T07:48:05+0000 lvl=info msg="Done pruning expired container backups"
t=2019-11-10T07:48:05+0000 lvl=info msg="Expiring log files"
t=2019-11-10T07:48:05+0000 lvl=info msg="Done expiring log files"
t=2019-11-10T07:48:05+0000 lvl=info msg="Updating images"
t=2019-11-10T07:48:05+0000 lvl=info msg="Updating instance types"
t=2019-11-10T07:48:05+0000 lvl=info msg="Done updating images"
t=2019-11-10T07:48:05+0000 lvl=info msg="Done updating instance types"
t=2019-11-10T07:48:34+0000 lvl=info msg="Deleting container" created=2019-11-10T07:39:42+0000 ephemeral=false name=bionic project=default used=2019-11-10T07:40:34+0000
t=2019-11-10T07:48:34+0000 lvl=info msg="Deleted container" created=2019-11-10T07:39:42+0000 ephemeral=false name=bionic project=default used=2019-11-10T07:40:34+0000
t=2019-11-10T07:48:47+0000 lvl=info msg="Starting container" action=start created=2018-12-20T17:57:03+0000 ephemeral=false name=gitlab project=default stateful=false used=2019-11-09T16:36:59+0000
t=2019-11-10T07:48:47+0000 lvl=eror msg="Failed starting container" action=start created=2018-12-20T17:57:03+0000 ephemeral=false name=gitlab project=default stateful=false used=2019-11-09T16:36:59+0000
t=2019-11-10T07:48:48+0000 lvl=info msg="Container initiated stop" action=stop created=2018-12-20T17:57:03+0000 ephemeral=false name=gitlab project=default stateful=false used=2019-11-10T07:48:47+0000

“lxc storage list”

+---------+-------------+--------+--------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |             SOURCE             | USED BY |
+---------+-------------+--------+--------------------------------+---------+
| default |             | btrfs  | /var/snap/lxd/common/lxd/disks | 9       |
+---------+-------------+--------+--------------------------------+---------+

and one of the ct “lxc config show --expanded ygol”

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20181124)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20181124"
  image.version: "18.04"
  volatile.base_image: 7b58622614fa724290eb15c139501394c63641e81411c13d166825cc8c7fae45
  volatile.eth0.hwaddr: 00:16:3e:f0:08:a9
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

simos · November 10, 2019, 9:36am

Hi!

If you try to start a container manually, what error message do you get? Is the issue with autostarting only?

I did not notice a specific error in the logs.

Note that there are also per-container logs, at the same location as the LXD log.

ygol · November 10, 2019, 9:49am

“lxc start ygol”

 Error: Failed to run: /snap/lxd/current/bin/lxd forkstart ygol /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/ygol/lxc.conf:
 Try `lxc info --show-log ygol` for more info

“lxc info --show-log local:ygol”

Name: ygol
Location: none
Remote: unix://
Architecture: x86_64
Created: 2018/12/02 12:31 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc ygol 20191110094725.517 ERROR    dir - storage/dir.c:dir_mount:198 - Permission denied - Failed to mount "/var/snap/lxd/common/lxd/containers/ygol/rootfs" on "/var/snap/lxd/common/lxc/"
lxc ygol 20191110094725.517 ERROR    conf - conf.c:lxc_mount_rootfs:1353 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/ygol/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)"
lxc ygol 20191110094725.517 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3447 - Failed to setup rootfs for
lxc ygol 20191110094725.517 ERROR    conf - conf.c:lxc_setup:3550 - Failed to setup rootfs
lxc ygol 20191110094725.517 ERROR    start - start.c:do_start:1321 - Failed to setup container "ygol"
lxc ygol 20191110094725.517 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)
lxc ygol 20191110094725.518 WARN     network - network.c:lxc_delete_network_priv:3377 - Failed to rename interface with index 77 from "eth0" to its initial name "veth9d00c19a"
lxc ygol 20191110094725.518 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:873 - Received container state "ABORTING" instead of "RUNNING"
lxc ygol 20191110094725.518 ERROR    start - start.c:__lxc_start:2039 - Failed to spawn container "ygol"
lxc 20191110094725.636 WARN     commands - commands.c:lxc_cmd_rsp_recv:135 - Connection reset by peer - Failed to receive response for command "get_state"

Thank you

ygol · November 10, 2019, 9:57am

the following dir is empty:

“/var/snap/lxd/common/lxd/storage-pools/default”

All containers are now under:

/var/snap/lxd/common/lxd/disks/containers

I get also the same problem when creating a new CT

simos · November 10, 2019, 1:07pm

I do not see a hint as to what could be the problem. It says that it cannot perform the mount, etc, with permission denied but that could be due to many things.

Can you post information as to which Linux distribution you are running?
Also, post the output of lxd.check-kernel.
Most likely someone else can tell us the most common cases from the current error messages.

ygol · November 10, 2019, 1:22pm

cat /etc/*-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION=“Ubuntu 18.04.3 LTS”
NAME=“Ubuntu”
VERSION=“18.04.3 LTS (Bionic Beaver)”
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=“Ubuntu 18.04.3 LTS”
VERSION_ID=“18.04”
HOME_URL=“https://www.ubuntu.com/”
SUPPORT_URL=“https://help.ubuntu.com/”
BUG_REPORT_URL=“Bugs : Ubuntu”
PRIVACY_POLICY_URL=“https://www.ubuntu.com/legal/terms-and-policies/privacy-policy”
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

lxd.check-kernel

Kernel configuration not found at /proc/config.gz; searching…
Kernel configuration found at /boot/config-4.15.0-66-generic
— Namespaces —
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled

— Control groups —
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/memory
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/pids
/sys/fs/cgroup/devices
/sys/fs/cgroup/freezer
/sys/fs/cgroup/blkio
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/rdma

Cgroup v2 mount points:
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

— Misc —
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: enabled, loaded
CONFIG_NF_NAT_IPV6: enabled, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

— Checkpoint/Restore —
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /snap/lxd/12317/bin/lxc-checkconfig

ygol · November 13, 2019, 10:37am

@simos,
This thread discuss that their zfs volume was unmounted. I am using btrfs and
“lxc storage show default” shows:

config:
  source: /var/snap/lxd/common/lxd/disks
  volatile.initial_source: /var/snap/lxd/common/lxd/disks
description: ""
name: default
driver: btrfs
used_by:
- /1.0/containers/ansible-cee-cee-11798410
- /1.0/containers/egol
- /1.0/containers/etherpad
- /1.0/containers/first
- /1.0/containers/gitlab
- /1.0/containers/glr
- /1.0/containers/v12
- /1.0/containers/ygol
- /1.0/images/d6f281a2e523674bcd9822f3f61be337c51828fb0dc94c8a200ab216d12a0fff
- /1.0/profiles/default
status: Created
locations:
- none

would it be the same problem I’m experiencing?
How would I fix it?
Thanks again

ygol · December 8, 2019, 11:00am

Reading https://github.com/lxc/lxd/issues/5797

I changed the permission on “/var/snap/lxd/common/lxd/storage-pools/default” from 700 to 711

and it solved the problem.