Container away after move and restart

Thomas_P · August 6, 2020, 12:37pm

hi, i have to machines running lxd-containers. I upgraded one to the snap lxd installation. create a zfs-storage and copy my containers from the old to new installation to the new zfs storage with the -s copy parameter with the new zfs-pool name. The container started and ist running well for a while. Now, if I stop a container and want to run again i get this error:

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart nextcloud /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/nextcloud/lxc.conf:

This folder ist a symbolic link to

nextcloud -> /var/snap/lxd/common/lxd/storage-pools/zfs-pool1/containers/nextcloud

And this folder ist empty, only a backup.yaml is in there.

So where is my container and rootfs? And why appears this problem after stopping the container? I have other containers migrated the same way. From old to new with new zfs storage. all are running. I tried to stop another. And the result ist the same as with the “nextcloud” container shown above.

lxc --show-info

lxc nextcloud 20200806124538.566 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.nextcloud"
lxc nextcloud 20200806124538.568 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.nextcloud"
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.607 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806124538.608 ERROR    dir - storage/dir.c:dir_mount:152 - No such file or directory - Failed to mount "/var/snap/lxd/common/lxd/containers/nextcloud/rootfs" on "/var/snap/lxd/common/lxc/"
lxc nextcloud 20200806124538.608 ERROR    conf - conf.c:lxc_mount_rootfs:1257 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/nextcloud/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)"
lxc nextcloud 20200806124538.608 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3094 - Failed to setup rootfs for
lxc nextcloud 20200806124538.608 ERROR    conf - conf.c:lxc_setup:3194 - Failed to setup rootfs
lxc nextcloud 20200806124538.608 ERROR    start - start.c:do_start:1224 - Failed to setup container "nextcloud"
lxc nextcloud 20200806124538.608 ERROR    sync - sync.c:__sync_wait:41 - An error occurred in another process (expected sequence number 5)
lxc nextcloud 20200806124538.613 WARN     network - network.c:lxc_delete_network_priv:3185 - Failed to rename interface with index 0 from "eth0" to its initial name "veth05c18771"
lxc nextcloud 20200806124538.613 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:850 - Received container state "ABORTING" instead of "RUNNING"
lxc nextcloud 20200806124538.613 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "nextcloud"
lxc nextcloud 20200806124538.614 WARN     start - start.c:lxc_abort:1019 - No such process - Failed to send SIGKILL via pidfd 30 for process 3787597
lxc 20200806124538.785 WARN     commands - commands.c:lxc_cmd_rsp_recv:124 - Connection reset by peer - Failed to receive response for command "get_state"
lxc nextcloud 20200806124549.376 ERROR    conf - conf.c:run_buffer:324 - Script exited with status 1
lxc nextcloud 20200806124549.376 ERROR    start - start.c:lxc_end:965 - Failed to run lxc.hook.post-stop for container "nextcloud"

zfs -list | grep nextcloud

zfs-pool1/containers/nextcloud 58,3G 724G 58,3G /var/snap/lxd/common/lxd/storage-pools/zfs-pool1/containers/nextcloud

I think my data must be somewhere but where? The Folder ist empty

root@plsrv2:/var/snap/lxd/common/lxd/storage-pools/zfs-pool1/containers/nextcloud-web# ls -l
insgesamt 0

tomp · August 6, 2020, 1:50pm

This issue has come up a few times before and I’m struggling to resolve it.

If you remove the backup.yaml file in question your container should start. However the backup.yaml file should only be written when the ZFS volume is mounted (it isn’t at the moment which is why you cant see your container’s files).

If you can reproduce the issue reliably by stopping/starting the container we may be able to figure this one out more thoroughly.

Thomas_P · August 6, 2020, 5:11pm

If I remove the file and try to start

lxc start nextcloud

I got the same error and the backup.yaml is newly created.

 lxc info --show-log plsrv2:nextcloud

Name: nextcloud
Location: none
Remote: https://192.168.20.108:8443
Architecture: x86_64
Created: 2020/07/28 20:48 UTC
Status: Stopped
Type: container
Profiles: zfs-bridged

Log:

lxc nextcloud 20200806170908.309 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.nextcloud"
lxc nextcloud 20200806170908.323 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.nextcloud"
lxc nextcloud 20200806170908.559 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.559 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.559 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.559 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.560 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.561 ERROR    conf - conf.c:turn_into_dependent_mounts:3005 - Invalid argument - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc nextcloud 20200806170908.561 ERROR    dir - storage/dir.c:dir_mount:152 - No such file or directory - Failed to mount "/var/snap/lxd/common/lxd/containers/nextcloud/rootfs" on "/var/snap/lxd/common/lxc/"
lxc nextcloud 20200806170908.561 ERROR    conf - conf.c:lxc_mount_rootfs:1257 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/nextcloud/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)"
lxc nextcloud 20200806170908.561 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3094 - Failed to setup rootfs for
lxc nextcloud 20200806170908.561 ERROR    conf - conf.c:lxc_setup:3194 - Failed to setup rootfs
lxc nextcloud 20200806170908.561 ERROR    start - start.c:do_start:1224 - Failed to setup container "nextcloud"
lxc nextcloud 20200806170908.563 ERROR    sync - sync.c:__sync_wait:41 - An error occurred in another process (expected sequence number 5)
lxc nextcloud 20200806170908.615 WARN     network - network.c:lxc_delete_network_priv:3185 - Failed to rename interface with index 0 from "eth0" to its initial name "veth99f77889"
lxc nextcloud 20200806170908.616 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:850 - Received container state "ABORTING" instead of "RUNNING"
lxc nextcloud 20200806170908.616 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "nextcloud"
lxc nextcloud 20200806170908.616 WARN     start - start.c:lxc_abort:1019 - No such process - Failed to send SIGKILL via pidfd 30 for process 263395
lxc 20200806170908.221 WARN     commands - commands.c:lxc_cmd_rsp_recv:124 - Connection reset by peer - Failed to receive response for command "get_state"
lxc nextcloud 20200806170918.789 ERROR    conf - conf.c:run_buffer:324 - Script exited with status 1
lxc nextcloud 20200806170918.790 ERROR    start - start.c:lxc_end:965 - Failed to run lxc.hook.post-stop for container "nextcloud"

tomp · August 7, 2020, 8:44am

Please can you try enabling debug more in the snap:

sudo snap set lxd daemon.debug=true
sudo systemctl reload snap.lxd.daemon

And then reproducing the issue (after removing the file again) and send the last few hundred lines of the /var/snap/lxd/common/lxd/logs/lxd.log file.

Thomas_P · August 12, 2020, 4:19pm

Thats stupid. After stopping all Container on the system and reloading the daemon the container starts again.

Here is the Log

lxc 20200812161353.351 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/rdma
lxc 20200812161353.351 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.351 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: rdma
lxc 20200812161353.351 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc 20200812161353.351 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:46 - Running with hybrid cgroup layout
lxc nextcloud 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc nextcloud 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_limiting_cgroup"
lxc 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_limiting_cgroup" received response
lxc 20200812161353.351 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_limiting_cgroup" is 0
lxc 20200812161353.351 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_limiting_cgroup"
lxc nextcloud 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_cgroup" received response
lxc 20200812161353.351 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_cgroup"
lxc nextcloud 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_cgroup"
lxc nextcloud 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_state" received response
lxc 20200812161353.351 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_state" is 0
lxc 20200812161353.351 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_state"
lxc 20200812161353.351 DEBUG    commands - commands.c:lxc_cmd_get_state:726 - Container "nextcloud" is in "RUNNING" state
lxc nextcloud 20200812161353.351 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_state"
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:941 - basecginfo is:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:942 - 12:rdma:/
11:perf_event:/
10:blkio:/init.scope
9:devices:/init.scope
8:freezer:/
7:hugetlb:/
6:cpu,cpuacct:/init.scope
5:pids:/init.scope
4:net_cls,net_prio:/
3:cpuset:/
2:memory:/init.scope
1:name=systemd:/init.scope
0::/init.scope

lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 0: rdma
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 1: perf_event
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 2: blkio
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 3: devices
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 4: freezer
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 5: hugetlb
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 6: cpu
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 7: cpuacct
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 8: pids
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 9: net_cls
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 10: net_prio
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 11: cpuset
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 12: memory
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 13: cgroup2
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:948 - named subsystem 0: name=systemd
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3261 - No controllers are enabled for delegation in the unified hierarchy
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3279 - Writable cgroup hierarchies:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:922 -   Hierarchies:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   0: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/unified
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   1: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/systemd
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: name=systemd
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   2: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/memory
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: memory
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   3: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpuset
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpuset
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   4: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/net_cls,net_prio
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: net_cls
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: net_prio
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   5: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/pids
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: pids
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   6: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpu,cpuacct
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpu
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: cpuacct
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   7: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/hugetlb
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: hugetlb
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   8: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/freezer
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: freezer
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   9: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/devices
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: devices
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   10: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/blkio
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: blkio
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   11: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/perf_event
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: perf_event
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   12: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/rdma
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: rdma
lxc 20200812161353.352 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc 20200812161353.352 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:46 - Running with hybrid cgroup layout
lxc nextcloud 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc nextcloud 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_limiting_cgroup"
lxc 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_limiting_cgroup" received response
lxc 20200812161353.352 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_limiting_cgroup" is 0
lxc 20200812161353.352 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_limiting_cgroup"
lxc nextcloud 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_cgroup" received response
lxc 20200812161353.352 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_cgroup"
lxc nextcloud 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_cgroup"
lxc nextcloud 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_state" received response
lxc 20200812161353.352 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_state" is 0
lxc 20200812161353.352 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_state"
lxc 20200812161353.352 DEBUG    commands - commands.c:lxc_cmd_get_state:726 - Container "nextcloud" is in "RUNNING" state
lxc nextcloud 20200812161353.352 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_state"
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:941 - basecginfo is:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:942 - 12:rdma:/
11:perf_event:/
10:blkio:/init.scope
9:devices:/init.scope
8:freezer:/
7:hugetlb:/
6:cpu,cpuacct:/init.scope
5:pids:/init.scope
4:net_cls,net_prio:/
3:cpuset:/
2:memory:/init.scope
1:name=systemd:/init.scope
0::/init.scope

lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 0: rdma
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 1: perf_event
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 2: blkio
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 3: devices
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 4: freezer
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 5: hugetlb
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 6: cpu
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 7: cpuacct
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 8: pids
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 9: net_cls
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 10: net_prio
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 11: cpuset
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 12: memory
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 13: cgroup2
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:948 - named subsystem 0: name=systemd
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3261 - No controllers are enabled for delegation in the unified hierarchy
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3279 - Writable cgroup hierarchies:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:922 -   Hierarchies:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   0: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/unified
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   1: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/systemd
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: name=systemd
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   2: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/memory
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: memory
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   3: base_cgroup: /
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpuset
lxc 20200812161353.352 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpuset
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   4: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/net_cls,net_prio
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: net_cls
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: net_prio
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   5: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/pids
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: pids
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   6: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpu,cpuacct
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpu
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: cpuacct
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   7: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/hugetlb
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: hugetlb
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   8: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/freezer
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: freezer
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   9: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/devices
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: devices
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   10: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/blkio
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: blkio
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   11: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/perf_event
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: perf_event
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   12: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/rdma
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: rdma
lxc 20200812161353.353 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc 20200812161353.353 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:46 - Running with hybrid cgroup layout
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_limiting_cgroup"
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_limiting_cgroup" received response
lxc 20200812161353.353 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_limiting_cgroup" is 0
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_limiting_cgroup"
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_cgroup" received response
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_cgroup"
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_cgroup"
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_init_pid" received response
lxc 20200812161353.353 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_init_pid" is 0
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_init_pid"
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_init_pid"
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_init_pid" received response
lxc 20200812161353.353 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_init_pid" is 0
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_init_pid"
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_init_pid"
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_state" received response
lxc 20200812161353.353 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_state" is 0
lxc 20200812161353.353 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_state"
lxc 20200812161353.353 DEBUG    commands - commands.c:lxc_cmd_get_state:726 - Container "nextcloud" is in "RUNNING" state
lxc nextcloud 20200812161353.353 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_state"
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:941 - basecginfo is:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:942 - 12:rdma:/
11:perf_event:/
10:blkio:/init.scope
9:devices:/init.scope
8:freezer:/
7:hugetlb:/
6:cpu,cpuacct:/init.scope
5:pids:/init.scope
4:net_cls,net_prio:/
3:cpuset:/
2:memory:/init.scope
1:name=systemd:/init.scope
0::/init.scope

lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 0: rdma
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 1: perf_event
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 2: blkio
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 3: devices
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 4: freezer
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 5: hugetlb
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 6: cpu
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 7: cpuacct
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 8: pids
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 9: net_cls
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 10: net_prio
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 11: cpuset
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 12: memory
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 13: cgroup2
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:948 - named subsystem 0: name=systemd
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3261 - No controllers are enabled for delegation in the unified hierarchy
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3279 - Writable cgroup hierarchies:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:922 -   Hierarchies:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   0: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/unified
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   1: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/systemd
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: name=systemd
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   2: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/memory
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: memory
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   3: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpuset
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpuset
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   4: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/net_cls,net_prio
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: net_cls
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: net_prio
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   5: base_cgroup: /
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/pids
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.353 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: pids
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   6: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpu,cpuacct
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpu
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: cpuacct
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   7: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/hugetlb
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: hugetlb
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   8: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/freezer
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: freezer
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   9: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/devices
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: devices
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   10: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/blkio
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: blkio
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   11: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/perf_event
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: perf_event
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   12: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/rdma
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: rdma
lxc 20200812161353.354 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc 20200812161353.354 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:46 - Running with hybrid cgroup layout
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_limiting_cgroup"
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_limiting_cgroup" received response
lxc 20200812161353.354 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_limiting_cgroup" is 0
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_limiting_cgroup"
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_cgroup" received response
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_cgroup"
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_cgroup"
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_state" received response
lxc 20200812161353.354 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_state" is 0
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_state"
lxc 20200812161353.354 DEBUG    commands - commands.c:lxc_cmd_get_state:726 - Container "nextcloud" is in "RUNNING" state
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_state"
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:941 - basecginfo is:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:942 - 12:rdma:/
11:perf_event:/
10:blkio:/init.scope
9:devices:/init.scope
8:freezer:/
7:hugetlb:/
6:cpu,cpuacct:/init.scope
5:pids:/init.scope
4:net_cls,net_prio:/
3:cpuset:/
2:memory:/init.scope
1:name=systemd:/init.scope
0::/init.scope

lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 0: rdma
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 1: perf_event
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 2: blkio
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 3: devices
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 4: freezer
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 5: hugetlb
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 6: cpu
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 7: cpuacct
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 8: pids
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 9: net_cls
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 10: net_prio
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 11: cpuset
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 12: memory
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 13: cgroup2
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:948 - named subsystem 0: name=systemd
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3261 - No controllers are enabled for delegation in the unified hierarchy
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3279 - Writable cgroup hierarchies:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:922 -   Hierarchies:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   0: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/unified
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   1: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/systemd
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: name=systemd
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   2: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/memory
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: memory
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   3: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpuset
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpuset
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   4: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/net_cls,net_prio
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: net_cls
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: net_prio
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   5: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/pids
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: pids
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   6: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpu,cpuacct
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpu
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: cpuacct
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   7: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/hugetlb
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: hugetlb
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   8: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/freezer
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: freezer
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   9: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/devices
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: devices
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   10: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/blkio
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: blkio
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   11: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/perf_event
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: perf_event
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   12: base_cgroup: /
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/rdma
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc 20200812161353.354 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: rdma
lxc 20200812161353.354 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc 20200812161353.354 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:46 - Running with hybrid cgroup layout
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_limiting_cgroup"
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_limiting_cgroup" received response
lxc 20200812161353.354 DEBUG    commands - commands.c:lxc_cmd_rsp_recv:162 - Response data length for command "get_limiting_cgroup" is 0
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_limiting_cgroup"
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_accept:1644 - Accepted new client as fd 13 on command server fd 9
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_rsp_recv:125 - Command "get_cgroup" received response
lxc 20200812161353.354 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 55 for command "get_cgroup"
lxc nextcloud 20200812161353.354 TRACE    commands - commands.c:lxc_cmd_fd_cleanup:1556 - Closing client fd 13 for command "get_cgroup"

Was this a problem with the zfs system? I migrated to zfs because I have sevaral problems with btrfs, so I though give zfs a try, but I think the safest way would be to move back to completly “normal” filesystems for my containers.

tomp · August 13, 2020, 5:57am

I suspect its some sort of race condition in the lxd mounting code that is unmounting the volume whilst LXD is writing the backup file, thats why it rarely comes up.

tomp · August 17, 2020, 9:16am

Is that output you posted from /var/snap/lxd/common/lxd/logs/lxd.log or from lxc info --show-log, as the former would be more useful for debugging this issue.

Thomas_P · August 17, 2020, 11:25am

The log ist from the special container that wouldn’t start before:

lxc info --show-log nextcloud

I stopped all containers again. Set debug.mode = true, reload daemon and start one container (nextcloud). This is the content from the logfile in

/var/snap/lxd/common/lxd/logs/lxd.log

t=2020-08-17T13:20:23+0200 lvl=info msg="LXD 4.0.2 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2020-08-17T13:20:23+0200 lvl=info msg="Kernel uid/gid map:" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - u 0 0 4294967295" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - g 0 0 4294967295" 
t=2020-08-17T13:20:23+0200 lvl=info msg="Configured LXD uid/gid map:" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - u 0 1000000 1000000000" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - g 0 1000000 1000000000" 
t=2020-08-17T13:20:23+0200 lvl=info msg="Kernel features:" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - netnsid-based network retrieval: yes" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - pidfds: no" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - uevent injection: yes" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - seccomp listener: yes" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - seccomp listener continue syscalls: yes" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - cgroup layout: hybrid" 
t=2020-08-17T13:20:23+0200 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored" 
t=2020-08-17T13:20:23+0200 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - shiftfs support: disabled" 
t=2020-08-17T13:20:23+0200 lvl=info msg="Initializing local database" 
t=2020-08-17T13:20:23+0200 lvl=dbug msg="Initializing database gateway" 
t=2020-08-17T13:20:23+0200 lvl=dbug msg="Start database node" address= id=1 role=voter
t=2020-08-17T13:20:23+0200 lvl=info msg="Starting /dev/lxd handler:" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2020-08-17T13:20:23+0200 lvl=info msg="REST API daemon:" 
t=2020-08-17T13:20:23+0200 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2020-08-17T13:20:23+0200 lvl=info msg=" - binding TCP socket" socket=[::]:8443
t=2020-08-17T13:20:23+0200 lvl=info msg="Initializing global database" 
t=2020-08-17T13:20:23+0200 lvl=dbug msg="Dqlite: attempt 0: server 1: connected" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Firewall loaded driver \"nftables\"" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Initializing storage pools" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Initializing and checking storage pool \"default\"" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Mount started" driver=dir pool=default
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Mount finished" driver=dir pool=default
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Initializing and checking storage pool \"zfs-pool1\"" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Mount started" driver=zfs pool=zfs-pool1
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Mount finished" driver=zfs pool=zfs-pool1
t=2020-08-17T13:20:24+0200 lvl=info msg="Initializing daemon storage mounts" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Initializing networks" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="New task Operation: 15e98582-6a94-4213-b5f6-da9cdd2fda8f" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Pruning leftover image files" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Started task operation: 15e98582-6a94-4213-b5f6-da9cdd2fda8f" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Done pruning leftover image files" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Loading daemon configuration" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Success for task operation: 15e98582-6a94-4213-b5f6-da9cdd2fda8f" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Initialized inotify with file descriptor 3" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Started seccomp handler" path=/var/snap/lxd/common/lxd/seccomp.socket
t=2020-08-17T13:20:24+0200 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2020-08-17T13:20:24+0200 lvl=dbug msg="New task Operation: 1d5426fd-0d80-40cf-a26b-01906d27eb7b" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Pruning expired images" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Started task operation: 1d5426fd-0d80-40cf-a26b-01906d27eb7b" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Done pruning expired images" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2020-08-17T13:20:24+0200 lvl=dbug msg="New task Operation: 1200ecda-862c-4652-86d0-15e5e042ac66" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Success for task operation: 1d5426fd-0d80-40cf-a26b-01906d27eb7b" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Pruning expired instance backups" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Started task operation: 1200ecda-862c-4652-86d0-15e5e042ac66" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Done pruning expired instance backups" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Success for task operation: 1200ecda-862c-4652-86d0-15e5e042ac66" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="New task Operation: 9cdddaa9-63a2-4c7a-9d1e-1e60e90de582" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="New task Operation: 0d3efb53-33d3-4f31-94ab-80c07845eb14" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="New task Operation: 3d56b005-a773-4915-88b6-d92a89dc2a13" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Updating images" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Started task operation: 3d56b005-a773-4915-88b6-d92a89dc2a13" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Expiring log files" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Started task operation: 9cdddaa9-63a2-4c7a-9d1e-1e60e90de582" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Done updating images" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Updating instance types" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Started task operation: 0d3efb53-33d3-4f31-94ab-80c07845eb14" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Done updating instance types" 
t=2020-08-17T13:20:24+0200 lvl=info msg="Done expiring log files" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Processing image" alias=20.04 fp=97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540 protocol=simplestreams server=https://cloud-images.ubuntu.com/releases
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Connecting to a remote simplestreams server" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Success for task operation: 9cdddaa9-63a2-4c7a-9d1e-1e60e90de582" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Scheduler: network: veth717232f0 has been added: updating network priorities" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Scheduler: network: veth19d557cd has been added: updating network priorities" 
t=2020-08-17T13:20:24+0200 lvl=dbug msg="MountInstance started" driver=zfs instance=webtrees pool=zfs-pool1 project=default
t=2020-08-17T13:20:24+0200 lvl=dbug msg="MountInstance finished" driver=zfs instance=webtrees pool=zfs-pool1 project=default
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Stopping device" device=root instance=webtrees project=default
t=2020-08-17T13:20:24+0200 lvl=dbug msg="Stopping device" device=eth0 instance=webtrees project=default
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Clearing instance firewall static filters" dev=eth0 host_name=veth19d557cd hwaddr=00:16:3e:f1:f7:57 instance=webtrees ipv4=0.0.0.0 ipv6=:: parent=br0 project=default
t=2020-08-17T13:20:25+0200 lvl=eror msg="Failed to start instance 'webtrees': Common start logic: Failed to run: zfs mount zfs-pool1/containers/webtrees: cannot mount '/var/snap/lxd/common/lxd/storage-pools/zfs-pool1/containers/webtrees': directory is not empty" 
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Image already exists in the db" image=97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Image already exists on storage pool \"zfs-pool1\"" 
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Already up to date" fp=97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Processing image" alias=default fp=a92eaa65a5c5e53c6bf788b4443f4e5d2afac1665486247c336aa90959522bb6 protocol=simplestreams server=https://cloud-images.ubuntu.com/releases
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Connecting to a remote simplestreams server" 
t=2020-08-17T13:20:25+0200 lvl=dbug msg=Handling ip=@ method=GET url=/1.0 user=
t=2020-08-17T13:20:25+0200 lvl=dbug msg=Handling ip=@ method=GET url=/internal/ready user=
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Image already exists in the db" image=a92eaa65a5c5e53c6bf788b4443f4e5d2afac1665486247c336aa90959522bb6
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Already up to date" fp=a92eaa65a5c5e53c6bf788b4443f4e5d2afac1665486247c336aa90959522bb6
t=2020-08-17T13:20:25+0200 lvl=dbug msg="Success for task operation: 3d56b005-a773-4915-88b6-d92a89dc2a13" 
t=2020-08-17T13:20:32+0200 lvl=dbug msg="Success for task operation: 0d3efb53-33d3-4f31-94ab-80c07845eb14" 
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Found cert" name=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg=Handling ip=192.168.20.1:37932 method=GET url=/1.0 user=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Found cert" name=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Found cert" name=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg=Handling ip=192.168.20.1:37934 method=GET url=/1.0/instances/nextcloud user=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Found cert" name=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg=Handling ip=192.168.20.1:37936 method=GET url=/1.0/events user=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg="New event listener: 8552a5b3-d7de-43a2-9ac4-f770679cb197" 
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Found cert" name=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg=Handling ip=192.168.20.1:37938 method=PUT url=/1.0/instances/nextcloud/state user=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg="\n\t{\n\t\t\"action\": \"start\",\n\t\t\"timeout\": 0,\n\t\t\"force\": false,\n\t\t\"stateful\": false\n\t}" 
t=2020-08-17T13:20:34+0200 lvl=dbug msg="New task Operation: 94728dac-5e8f-48b2-82e9-cb9511655be1" 
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Started task operation: 94728dac-5e8f-48b2-82e9-cb9511655be1" 
t=2020-08-17T13:20:34+0200 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/94728dac-5e8f-48b2-82e9-cb9511655be1\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"94728dac-5e8f-48b2-82e9-cb9511655be1\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Starting container\",\n\t\t\t\"created_at\": \"2020-08-17T13:20:34.670563877+02:00\",\n\t\t\t\"updated_at\": \"2020-08-17T13:20:34.670563877+02:00\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/nextcloud\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"none\"\n\t\t}\n\t}" 
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Scheduler: network: vethf2e9cd38 has been added: updating network priorities" 
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Scheduler: network: vethe0d4b7df has been added: updating network priorities" 
t=2020-08-17T13:20:34+0200 lvl=dbug msg="MountInstance started" driver=zfs instance=nextcloud pool=zfs-pool1 project=default
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Found cert" name=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg=Handling ip=192.168.20.1:37940 method=GET url=/1.0/operations/94728dac-5e8f-48b2-82e9-cb9511655be1 user=22a4d8688f9e540c2393a2c8257ccf1b83fff50ce4b8bd3359c7d081cc2922d4
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Mounted ZFS dataset" dev=zfs-pool1/containers/nextcloud driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/zfs-pool1/containers/nextcloud pool=zfs-pool1
t=2020-08-17T13:20:34+0200 lvl=dbug msg="MountInstance finished" driver=zfs instance=nextcloud pool=zfs-pool1 project=default
t=2020-08-17T13:20:34+0200 lvl=dbug msg="UpdateInstanceBackupFile started" driver=zfs instance=nextcloud pool=zfs-pool1 project=default
t=2020-08-17T13:20:34+0200 lvl=dbug msg="UpdateInstanceBackupFile finished" driver=zfs instance=nextcloud pool=zfs-pool1 project=default
t=2020-08-17T13:20:34+0200 lvl=dbug msg="MountInstance started" driver=zfs instance=nextcloud pool=zfs-pool1 project=default
t=2020-08-17T13:20:34+0200 lvl=dbug msg="MountInstance finished" driver=zfs instance=nextcloud pool=zfs-pool1 project=default
t=2020-08-17T13:20:34+0200 lvl=info msg="Starting container" action=start created=2020-07-28T22:48:49+0200 ephemeral=false name=nextcloud project=default stateful=false used=2020-08-12T18:29:52+0200
t=2020-08-17T13:20:34+0200 lvl=dbug msg=Handling ip=@ method=GET url=/internal/containers/332/onstart user=
t=2020-08-17T13:20:34+0200 lvl=dbug msg="MountInstance started" driver=zfs instance=nextcloud pool=zfs-pool1 project=default
t=2020-08-17T13:20:34+0200 lvl=dbug msg="MountInstance finished" driver=zfs instance=nextcloud pool=zfs-pool1 project=default
t=2020-08-17T13:20:34+0200 lvl=dbug msg="Scheduler: container nextcloud started: re-balancing" 
t=2020-08-17T13:20:35+0200 lvl=info msg="Started container" action=start created=2020-07-28T22:48:49+0200 ephemeral=false name=nextcloud project=default stateful=false used=2020-08-12T18:29:52+0200
t=2020-08-17T13:20:35+0200 lvl=dbug msg="Success for task operation: 94728dac-5e8f-48b2-82e9-cb9511655be1" 
t=2020-08-17T13:20:35+0200 lvl=dbug msg="Event listener finished: 8552a5b3-d7de-43a2-9ac4-f770679cb197" 
t=2020-08-17T13:20:35+0200 lvl=dbug msg="Disconnected event listener: 8552a5b3-d7de-43a2-9ac4-f770679cb197"

Thomas_P · August 20, 2020, 8:42am

Now, the next container wouldn’t restart. Even after set to debug and reloading the deamon.
I have made a snapshot of the container and run an reboot in the container. After that, the “backup.yml” File ist there and the container doesn’t start. Then I set the log to debug (as in previous posts); reload the deamon, tests it. No Start. Set debug to false. reload deamon. Start. Doesn’t start.

Output Log is:

lxc info --show-log plsrv2:nodered
Name: nodered
Location: none
Remote: https://192.168.20.108:8443
Architecture: x86_64
Created: 2020/07/28 19:15 UTC
Status: Stopped
Type: container
Profiles: zfs-bridged

Log:

lxc nodered 20200820083904.739 TRACE    commands - commands.c:lxc_cmd:290 - Connection refused - Command "get_state" failed to connect command socket
lxc nodered 20200820083904.739 TRACE    start - start.c:lxc_init_handler:694 - Created anonymous pair {3,7} of unix sockets
lxc nodered 20200820083904.739 TRACE    commands - commands.c:lxc_cmd_init:1671 - Created abstract unix socket "/var/snap/lxd/common/lxd/containers/nodered/command"
lxc nodered 20200820083904.739 TRACE    start - start.c:lxc_init_handler:709 - Unix domain socket 9 for command server is ready
lxc nodered 20200820083904.739 INFO     lxccontainer - lxccontainer.c:do_lxcapi_start:969 - Set process title to [lxc monitor] /var/snap/lxd/common/lxd/containers nodered
lxc nodered 20200820083904.740 DEBUG    lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 2774604 exited
lxc nodered 20200820083904.740 INFO     start - start.c:lxc_check_inherited:286 - Closed inherited fd 4
lxc nodered 20200820083904.740 INFO     start - start.c:lxc_check_inherited:286 - Closed inherited fd 5
lxc nodered 20200820083904.740 INFO     start - start.c:lxc_check_inherited:286 - Closed inherited fd 6
lxc nodered 20200820083904.740 TRACE    start - start.c:lxc_start:2113 - Doing lxc_start
lxc nodered 20200820083904.740 INFO     lsm - lsm/lsm.c:lsm_init:30 - LSM security driver AppArmor
lxc nodered 20200820083904.740 TRACE    start - start.c:lxc_init:732 - Initialized LSM
lxc nodered 20200820083904.740 TRACE    start - start.c:lxc_serve_state_clients:438 - Set container state to STARTING
lxc nodered 20200820083904.740 TRACE    start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc nodered 20200820083904.740 TRACE    start - start.c:lxc_init:738 - Set container state to "STARTING"
lxc nodered 20200820083904.740 TRACE    start - start.c:lxc_init:794 - Set environment variables
lxc nodered 20200820083904.740 INFO     conf - conf.c:run_script_argv:343 - Executing script "/proc/2773932/exe callhook /var/snap/lxd/common/lxd 326 start" for container "nodered"
lxc nodered 20200820083904.740 TRACE    conf - conf.c:run_script_argv:386 - Set environment variable: LXC_HOOK_TYPE=pre-start
lxc nodered 20200820083904.740 TRACE    conf - conf.c:run_script_argv:391 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc nodered 20200820083904.822 TRACE    start - start.c:lxc_init:799 - Ran pre-start hooks
lxc nodered 20200820083904.822 TRACE    start - start.c:setup_signal_fd:331 - Created signal file descriptor 5
lxc nodered 20200820083904.822 TRACE    start - start.c:lxc_init:808 - Set up signal fd
lxc nodered 20200820083904.823 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:655 - No such device - The process does not have a controlling terminal
lxc nodered 20200820083904.823 DEBUG    terminal - terminal.c:lxc_terminal_create_log_file:827 - Using "/var/snap/lxd/common/lxd/logs/nodered/console.log" as terminal log file
lxc nodered 20200820083904.823 TRACE    terminal - terminal.c:lxc_terminal_create_ringbuf:808 - Allocated 131072 byte terminal ringbuffer
lxc nodered 20200820083904.823 TRACE    start - start.c:lxc_init:816 - Created console
lxc nodered 20200820083904.823 TRACE    conf - conf.c:userns_exec_mapped_root:4428 - Chowned 10(/dev/pts/11) to uid 1000000 and 1000000
lxc nodered 20200820083904.823 TRACE    terminal - terminal.c:lxc_terminal_map_ids:1176 - Chowned terminal 10(/dev/pts/11)
lxc nodered 20200820083904.823 TRACE    start - start.c:lxc_init:823 - Chowned console
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:941 - basecginfo is:
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:942 - 12:rdma:/
11:perf_event:/
10:blkio:/init.scope
9:devices:/init.scope
8:freezer:/
7:hugetlb:/
6:cpu,cpuacct:/init.scope
5:pids:/init.scope
4:net_cls,net_prio:/
3:cpuset:/
2:memory:/init.scope
1:name=systemd:/init.scope
0::/init.scope

lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 0: rdma
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 1: perf_event
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 2: blkio
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 3: devices
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 4: freezer
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 5: hugetlb
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 6: cpu
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 7: cpuacct
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 8: pids
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 9: net_cls
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 10: net_prio
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 11: cpuset
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 12: memory
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:945 - kernel subsystem 13: cgroup2
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:948 - named subsystem 0: name=systemd
lxc nodered 20200820083904.823 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3261 - No controllers are enabled for delegation in the unified hierarchy
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3279 - Writable cgroup hierarchies:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:922 -   Hierarchies:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   0: base_cgroup: /
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/unified
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   1: base_cgroup: /
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/systemd
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: name=systemd
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   2: base_cgroup: /
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/memory
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: memory
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   3: base_cgroup: /
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpuset
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpuset
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   4: base_cgroup: /
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/net_cls,net_prio
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: net_cls
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: net_prio
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   5: base_cgroup: /
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/pids
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: pids
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   6: base_cgroup: /
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/cpu,cpuacct
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: cpu
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       1: cpuacct
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   7: base_cgroup: /
lxc nodered 20200820083904.825 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/hugetlb
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: hugetlb
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   8: base_cgroup: /
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/freezer
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: freezer
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   9: base_cgroup: /
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/devices
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: devices
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   10: base_cgroup: /
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/blkio
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: blkio
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   11: base_cgroup: /
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/perf_event
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: perf_event
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:927 -   12: base_cgroup: /
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:928 -       mountpoint:  /sys/fs/cgroup/rdma
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:929 -       controllers:
lxc nodered 20200820083904.826 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:931 -       0: rdma
lxc nodered 20200820083904.826 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc nodered 20200820083904.826 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:46 - Running with hybrid cgroup layout
lxc nodered 20200820083904.826 TRACE    start - start.c:lxc_init:830 - Initialized cgroup driver
lxc nodered 20200820083904.826 TRACE    seccomp - seccomp.c:get_new_ctx:470 - Added arch 2 to main seccomp context
lxc nodered 20200820083904.826 TRACE    seccomp - seccomp.c:get_new_ctx:478 - Removed native arch from main seccomp context
lxc nodered 20200820083904.826 TRACE    seccomp - seccomp.c:get_new_ctx:470 - Added arch 3 to main seccomp context
lxc nodered 20200820083904.826 TRACE    seccomp - seccomp.c:get_new_ctx:478 - Removed native arch from main seccomp context
lxc nodered 20200820083904.826 TRACE    seccomp - seccomp.c:get_new_ctx:483 - Arch 4 already present in main seccomp context
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:790 - Processing "[all]"
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:790 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:do_resolve_add_rule:517 - Set seccomp rule to reject force umounts
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:987 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:do_resolve_add_rule:517 - Set seccomp rule to reject force umounts
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:996 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:do_resolve_add_rule:517 - Set seccomp rule to reject force umounts
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1006 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:do_resolve_add_rule:517 - Set seccomp rule to reject force umounts
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1016 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:790 - Processing "[all]"
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:790 - Processing "kexec_load errno 38"
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:987 - Added native rule for arch 0 for kexec_load action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:996 - Added compat rule for arch 1073741827 for kexec_load action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1006 - Added compat rule for arch 1073741886 for kexec_load action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1016 - Added native rule for arch -1073741762 for kexec_load action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:790 - Processing "open_by_handle_at errno 38"
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:987 - Added native rule for arch 0 for open_by_handle_at action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:996 - Added compat rule for arch 1073741827 for open_by_handle_at action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1006 - Added compat rule for arch 1073741886 for open_by_handle_at action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1016 - Added native rule for arch -1073741762 for open_by_handle_at action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:790 - Processing "init_module errno 38"
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:987 - Added native rule for arch 0 for init_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:996 - Added compat rule for arch 1073741827 for init_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1006 - Added compat rule for arch 1073741886 for init_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1016 - Added native rule for arch -1073741762 for init_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:790 - Processing "finit_module errno 38"
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:987 - Added native rule for arch 0 for finit_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:996 - Added compat rule for arch 1073741827 for finit_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1006 - Added compat rule for arch 1073741886 for finit_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1016 - Added native rule for arch -1073741762 for finit_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:790 - Processing "delete_module errno 38"
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:987 - Added native rule for arch 0 for delete_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:996 - Added compat rule for arch 1073741827 for delete_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1006 - Added compat rule for arch 1073741886 for delete_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1016 - Added native rule for arch -1073741762 for delete_module action 327718(errno)
lxc nodered 20200820083904.826 INFO     seccomp - seccomp.c:parse_config_v2:1020 - Merging compat seccomp contexts into main context
lxc nodered 20200820083904.826 TRACE    seccomp - seccomp.c:parse_config_v2:1030 - Merged first compat seccomp context into main context
lxc nodered 20200820083904.826 TRACE    seccomp - seccomp.c:parse_config_v2:1046 - Merged second compat seccomp context into main context
lxc nodered 20200820083904.826 TRACE    start - start.c:lxc_init:835 - Read seccomp policy
lxc nodered 20200820083904.826 TRACE    start - start.c:lxc_init:842 - Initialized LSM
lxc nodered 20200820083904.826 INFO     start - start.c:lxc_init:844 - Container "nodered" is initialized
lxc nodered 20200820083904.842 TRACE    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:552 - "cgroup.clone_children" was already set to "1"
lxc nodered 20200820083904.843 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.nodered"
lxc nodered 20200820083904.843 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1368 - The monitor process uses "lxc.monitor.nodered" as cgroup
lxc nodered 20200820083904.844 DEBUG    storage - storage/storage.c:get_storage_by_name:211 - Detected rootfs type "dir"
lxc nodered 20200820083904.844 TRACE    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:552 - "cgroup.clone_children" was already set to "1"
lxc nodered 20200820083904.844 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.nodered"
lxc nodered 20200820083904.844 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1471 - The container process uses "lxc.payload.nodered" as cgroup
lxc nodered 20200820083904.844 TRACE    start - start.c:lxc_spawn:1658 - Argument list too long - Failed to spawn container directly into target cgroup
lxc nodered 20200820083904.846 TRACE    start - start.c:lxc_spawn:1690 - Cloned child process 2774651
lxc nodered 20200820083904.846 TRACE    utils - utils.c:lxc_can_use_pidfd:1861 - Kernel supports pidfds
lxc nodered 20200820083904.846 INFO     start - start.c:lxc_spawn:1706 - Cloned CLONE_NEWUSER
lxc nodered 20200820083904.846 INFO     start - start.c:lxc_spawn:1706 - Cloned CLONE_NEWNS
lxc nodered 20200820083904.846 INFO     start - start.c:lxc_spawn:1706 - Cloned CLONE_NEWPID
lxc nodered 20200820083904.846 INFO     start - start.c:lxc_spawn:1706 - Cloned CLONE_NEWUTS
lxc nodered 20200820083904.847 INFO     start - start.c:lxc_spawn:1706 - Cloned CLONE_NEWIPC
lxc nodered 20200820083904.847 DEBUG    start - start.c:lxc_try_preserve_namespaces:167 - Preserved user namespace via fd 31
lxc nodered 20200820083904.847 DEBUG    start - start.c:lxc_try_preserve_namespaces:167 - Preserved mnt namespace via fd 32
lxc nodered 20200820083904.847 DEBUG    start - start.c:lxc_try_preserve_namespaces:167 - Preserved pid namespace via fd 33
lxc nodered 20200820083904.847 DEBUG    start - start.c:lxc_try_preserve_namespaces:167 - Preserved uts namespace via fd 34
lxc nodered 20200820083904.847 DEBUG    start - start.c:lxc_try_preserve_namespaces:167 - Preserved ipc namespace via fd 35
lxc nodered 20200820083904.847 DEBUG    conf - conf.c:lxc_map_ids:2757 - No newuidmap and newgidmap binary found. Trying to write directly with euid 0
lxc nodered 20200820083904.847 TRACE    conf - conf.c:lxc_map_ids:2823 - Wrote mapping "0 1000000 1000000000
"
lxc nodered 20200820083904.847 TRACE    conf - conf.c:lxc_map_ids:2823 - Wrote mapping "0 1000000 1000000000
"
lxc nodered 20200820083904.848 INFO     start - start.c:do_start:1091 - Unshared CLONE_NEWNET
lxc nodered 20200820083904.848 TRACE    conf - conf.c:get_minimal_idmap:3950 - Allocated minimal idmapping for ns uid 0 and ns gid 0
lxc nodered 20200820083904.849 TRACE    conf - conf.c:userns_exec_1:4016 - Establishing uid mapping for "2774652" in new user namespace: nsuid 1000000000 - hostid 0 - range 1
lxc nodered 20200820083904.849 TRACE    conf - conf.c:userns_exec_1:4016 - Establishing uid mapping for "2774652" in new user namespace: nsuid 0 - hostid 1000000 - range 1000000000
lxc nodered 20200820083904.849 TRACE    conf - conf.c:userns_exec_1:4016 - Establishing gid mapping for "2774652" in new user namespace: nsuid 1000000000 - hostid 0 - range 1
lxc nodered 20200820083904.849 TRACE    conf - conf.c:userns_exec_1:4016 - Establishing gid mapping for "2774652" in new user namespace: nsuid 0 - hostid 1000000 - range 1000000000
lxc nodered 20200820083904.849 DEBUG    conf - conf.c:lxc_map_ids:2757 - No newuidmap and newgidmap binary found. Trying to write directly with euid 0
lxc nodered 20200820083904.849 TRACE    conf - conf.c:lxc_map_ids:2823 - Wrote mapping "1000000000 0 1
0 1000000 1000000000
"
lxc nodered 20200820083904.849 TRACE    conf - conf.c:lxc_map_ids:2823 - Wrote mapping "1000000000 0 1
0 1000000 1000000000
"
lxc nodered 20200820083904.849 TRACE    conf - conf.c:run_userns_fn:3792 - Calling function "chown_cgroup_wrapper"
lxc nodered 20200820083904.849 NOTICE   utils - utils.c:lxc_setgroups:1366 - Dropped additional groups
lxc nodered 20200820083904.849 WARN     cgfsng - cgroups/cgfsng.c:fchowmodat:1573 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc nodered 20200820083904.852 DEBUG    start - start.c:lxc_spawn:1779 - Preserved net namespace via fd 4
lxc nodered 20200820083904.852 TRACE    start - start.c:lxc_spawn:1786 - Allocated new network namespace id
lxc nodered 20200820083904.852 DEBUG    network - network.c:instantiate_phys:821 - Instantiated phys "veth1f9b6958" with ifindex is "192"
lxc nodered 20200820083904.895 DEBUG    network - network.c:lxc_network_move_created_netdev_priv:3104 - Moved network device "veth1f9b6958" with ifindex 192 to network namespace of 2774651
lxc nodered 20200820083904.895 TRACE    network - network.c:lxc_network_send_to_child:3558 - Sent network device name "veth1f9b6958" to child
lxc nodered 20200820083904.895 TRACE    network - network.c:lxc_network_recv_from_parent:3585 - Received network device name "veth1f9b6958" from parent
lxc nodered 20200820083904.895 NOTICE   utils - utils.c:lxc_setgroups:1366 - Dropped additional groups
lxc nodered 20200820083904.895 NOTICE   utils - utils.c:lxc_switch_uid_gid:1344 - Switched to gid 0
lxc nodered 20200820083904.895 NOTICE   utils - utils.c:lxc_switch_uid_gid:1353 - Switched to uid 0
lxc nodered 20200820083904.895 INFO     start - start.c:do_start:1204 - Unshared CLONE_NEWCGROUP
lxc nodered 20200820083904.898 TRACE    conf - conf.c:turn_into_dependent_mounts:3010 - Turned all mount table entries into dependent mount
lxc nodered 20200820083904.898 DEBUG    storage - storage/storage.c:get_storage_by_name:211 - Detected rootfs type "dir"
lxc nodered 20200820083904.898 ERROR    dir - storage/dir.c:dir_mount:152 - No such file or directory - Failed to mount "/var/snap/lxd/common/lxd/containers/nodered/rootfs" on "/var/snap/lxd/common/lxc/"
lxc nodered 20200820083904.899 ERROR    conf - conf.c:lxc_mount_rootfs:1257 - Failed to mount rootfs "/var/snap/lxd/common/lxd/containers/nodered/rootfs" onto "/var/snap/lxd/common/lxc/" with options "(null)"
lxc nodered 20200820083904.899 ERROR    conf - conf.c:lxc_setup_rootfs_prepare_root:3094 - Failed to setup rootfs for
lxc nodered 20200820083904.899 ERROR    conf - conf.c:lxc_setup:3194 - Failed to setup rootfs
lxc nodered 20200820083904.899 ERROR    start - start.c:do_start:1224 - Failed to setup container "nodered"
lxc nodered 20200820083904.899 ERROR    sync - sync.c:__sync_wait:41 - An error occurred in another process (expected sequence number 5)
lxc nodered 20200820083904.903 WARN     network - network.c:lxc_delete_network_priv:3185 - Failed to rename interface with index 0 from "eth0" to its initial name "veth1f9b6958"
lxc nodered 20200820083904.904 DEBUG    network - network.c:lxc_delete_network:3665 - Deleted network devices
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_serve_state_socket_pair:503 - Sent container state "ABORTING" to 7
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_serve_state_clients:438 - Set container state to ABORTING
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc nodered 20200820083904.904 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:850 - Received container state "ABORTING" instead of "RUNNING"
lxc nodered 20200820083904.904 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "nodered"
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_serve_state_clients:438 - Set container state to ABORTING
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc nodered 20200820083904.904 WARN     start - start.c:lxc_abort:1019 - No such process - Failed to send SIGKILL via pidfd 30 for process 2774651
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_serve_state_clients:438 - Set container state to STOPPING
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_end:900 - Set environment variable LXC_USER_NS=/proc/2774605/fd/31
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_end:900 - Set environment variable LXC_MNT_NS=/proc/2774605/fd/32
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_end:900 - Set environment variable LXC_PID_NS=/proc/2774605/fd/33
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_end:900 - Set environment variable LXC_UTS_NS=/proc/2774605/fd/34
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_end:900 - Set environment variable LXC_IPC_NS=/proc/2774605/fd/35
lxc nodered 20200820083904.904 TRACE    start - start.c:lxc_end:900 - Set environment variable LXC_NET_NS=/proc/2774605/fd/4
lxc nodered 20200820083904.904 INFO     conf - conf.c:run_script_argv:343 - Executing script "/snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 326 stopns" for container "nodered"
lxc nodered 20200820083904.904 TRACE    conf - conf.c:run_script_argv:386 - Set environment variable: LXC_HOOK_TYPE=stop
lxc nodered 20200820083904.904 TRACE    conf - conf.c:run_script_argv:391 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc nodered 20200820083905.231 TRACE    conf - conf.c:get_minimal_idmap:3950 - Allocated minimal idmapping for ns uid 0 and ns gid 0
lxc nodered 20200820083905.238 TRACE    conf - conf.c:userns_exec_1:4016 - Establishing uid mapping for "2774676" in new user namespace: nsuid 1000000000 - hostid 0 - range 1
lxc nodered 20200820083905.238 TRACE    conf - conf.c:userns_exec_1:4016 - Establishing uid mapping for "2774676" in new user namespace: nsuid 0 - hostid 1000000 - range 1000000000
lxc nodered 20200820083905.238 TRACE    conf - conf.c:userns_exec_1:4016 - Establishing gid mapping for "2774676" in new user namespace: nsuid 1000000000 - hostid 0 - range 1
lxc nodered 20200820083905.238 TRACE    conf - conf.c:userns_exec_1:4016 - Establishing gid mapping for "2774676" in new user namespace: nsuid 0 - hostid 1000000 - range 1000000000
lxc nodered 20200820083905.239 DEBUG    conf - conf.c:lxc_map_ids:2757 - No newuidmap and newgidmap binary found. Trying to write directly with euid 0
lxc nodered 20200820083905.240 TRACE    conf - conf.c:lxc_map_ids:2823 - Wrote mapping "1000000000 0 1
0 1000000 1000000000
"
lxc nodered 20200820083905.240 TRACE    conf - conf.c:lxc_map_ids:2823 - Wrote mapping "1000000000 0 1
0 1000000 1000000000
"
lxc nodered 20200820083905.241 TRACE    conf - conf.c:run_userns_fn:3792 - Calling function "cgroup_tree_remove_wrapper"
lxc nodered 20200820083905.243 NOTICE   utils - utils.c:lxc_setgroups:1366 - Dropped additional groups
lxc nodered 20200820083905.270 TRACE    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:552 - "cgroup.clone_children" was already set to "1"
lxc nodered 20200820083905.537 TRACE    start - start.c:lxc_end:943 - Closed command socket
lxc 20200820083905.538 WARN     commands - commands.c:lxc_cmd_rsp_recv:124 - Connection reset by peer - Failed to receive response for command "get_state"
lxc nodered 20200820083905.538 TRACE    start - start.c:lxc_end:954 - Set container state to "STOPPED"
lxc 20200820083905.538 TRACE    commands - commands.c:lxc_cmd:298 - Opened new command socket connection fd 50 for command "get_state"
lxc nodered 20200820083905.790 INFO     conf - conf.c:run_script_argv:343 - Executing script "/snap/lxd/current/lxcfs/lxc.reboot.hook" for container "nodered"
lxc nodered 20200820083905.791 TRACE    conf - conf.c:run_script_argv:386 - Set environment variable: LXC_HOOK_TYPE=post-stop
lxc nodered 20200820083905.791 TRACE    conf - conf.c:run_script_argv:391 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc nodered 20200820083905.582 INFO     conf - conf.c:run_script_argv:343 - Executing script "/snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 326 stop" for container "nodered"
lxc nodered 20200820083905.582 TRACE    conf - conf.c:run_script_argv:386 - Set environment variable: LXC_HOOK_TYPE=post-stop
lxc nodered 20200820083905.582 TRACE    conf - conf.c:run_script_argv:391 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc nodered 20200820083915.624 DEBUG    conf - conf.c:run_buffer:313 - Script exec /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 326 stop produced output: Error: Failed to unmount '/var/snap/lxd/common/lxd/storage-pools/zfs-pool1/containers/nodered': invalid argument

lxc nodered 20200820083915.627 ERROR    conf - conf.c:run_buffer:324 - Script exited with status 1
lxc nodered 20200820083915.627 ERROR    start - start.c:lxc_end:965 - Failed to run lxc.hook.post-stop for container "nodered"
lxc 20200820083922.570 TRACE    commands - commands.c:lxc_cmd:290 - Connection refused - Command "get_state" failed to connect command socket
lxc 20200820083922.627 TRACE    commands - commands.c:lxc_cmd:290 - Connection refused - Command "get_state" failed to connect command socket
lxc 20200820083922.628 TRACE    commands - commands.c:lxc_cmd:290 - Connection refused - Command "get_state" failed to connect command socket

tomp · August 20, 2020, 8:54am

Does it start if you remove the backup.yaml file before trying to start the container?

Thomas_P · August 20, 2020, 9:01am

unfortunately no. I set deamon to debug. restart deamon. remove backup.yml. start the container.
Get this error (and log from above):

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart nodered /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/nodered/lxc.conf: 
Try `lxc info --show-log plsrv2:nodered` for more info

Than I set debug back to false. reload deamon. remove backup.yml. try to start again. No Start of the container (error as above)

tomp · August 20, 2020, 9:01am

Is it possible to get a login to the problem system for investigation?

tomp · August 20, 2020, 10:02am

On further investigation on the affected system the issue appeared to be a kernel issue with dmesg logging multiple I/O subsystem crashes mentioning clearing mounts.

The LXD snap mount namespace showed hundreds of mounts and the affected container had at least 20 mounts for the same directory. However they couldn’t be unmounted using umount or zfs unmount.

Have suggested a reboot to see if that fixes the issue.

@stgraber have you seen anything like that before?

Thomas_P · August 20, 2020, 12:06pm

Hi,

i shutdown all containers (exec into container and shutdown command). restartet the system and then I could start the container without issues…
I think about to move the cotainers away from zfs-storage to plain file storage.

@tomp
Thanks again for immediately examining the issue!

VoidAndAny · February 19, 2021, 9:06pm

Hi,
Same problem here.
Old LXC container migrated to LXD (copy of rootfs over a small LXD container), then container moved to another host (copy of a snapshot)

When I start container, only boot init process is created.

$ lxc info --show-log suivi
Name: suivi
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/19 19:24 UTC
Status: Stopped
Type: container
Profiles: lanprofile

Log:

lxc suivi 20210219205847.837 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1126 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor.suivi"
lxc suivi 20210219205847.842 WARN     cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1126 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.payload.suivi"
lxc suivi 20210219205847.886 ERROR    conf - conf.c:turn_into_dependent_mounts:3065 - No such file or directory - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc suivi 20210219205847.914 ERROR    utils - utils.c:__safe_mount_beneath_at:1106 - Function not implemented - Failed to open 30(dev)
lxc suivi 20210219205945.456 WARN     cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:1083 - Success - Failed to initialize cpuset /sys/fs/cgroup/cpuset//lxc.pivot/lxc.pivot

I don’t understand how you resolve the previous case, can you explain me please ?

I tried to mount zfs pool, delete backup.yaml, restart container, reboot host, nothing works

Yoann

VoidAndAny · February 19, 2021, 9:10pm

My profile :

 $ lxc profile show lanprofile
config: {}
description: Default LXD profile
devices:
  eth0:
    nictype: macvlan
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: lanprofile
used_by:
- /1.0/instances/dokuwiki
- /1.0/instances/suivi

thomaspz · February 20, 2021, 6:58am

After all, we didn’t solve the issue. I shutdown all containers, restart the host. Then the container with the problem was starting right and I use this state to move the container back from zfs storage to a non journaling filesystem with ext4. Till than it worked well. I think it was an issue with the zfs - storage or probably the HDD it self.

VoidAndAny · February 20, 2021, 8:26am

strange because I do this migration with 2 containers, first one no problem (but it’s not a container migrated from LXC), second can’t make it boot

My 2 LXD host are using ZFS