Not able to stop and delete a container

Hello,

I am not able to stop a LXC container, the command lxc delete ag-k8s-slave-1 is running but nothing happens. I tried several workaround such as:

  • sudo ps -f -p grep containers/ag-k8s-slave-1 /proc/*/mountinfo | sed 's-^/proc/--' | sed 's-/.*--' | tr '\n' ',' | sed 's/,$//'
  • There was several process, I ran nsenter -t <PID> -m -- umount /var/lib/lxd/storage-pools/lxd/containers/ag-k8s-slave-1 with the first PID only
  • After that, only zfs process were still running (see below). I don’t know how to stop a ZFS process, could you tell me if this is possible? Also, why am I getting several Invalid argument?
grep: /proc/40401/mountinfo: Invalid argument
/proc/40541/mountinfo:1582 672 0:135 /rootfs / rw,relatime - zfs syspool/lxd/containers/ag-k8s-slave-1 rw,xattr,posixacl
  • The output of lxc info --show-log ag-k8s-slave-1 is:
Name: ag-k8s-slave-1
Location: none
Remote: unix://
Architecture: x86_64
Created: 2019/09/21 16:29 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 17356
Resources:
  Processes: 5
  Disk usage:
    root: 1.41GB
  CPU usage:
    CPU usage (in seconds): 14301
  Memory usage:
    Memory (current): 954.48MB
    Memory (peak): 1.34GB

Log:

lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.922 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1319 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1456 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1319 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1456 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1319 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.923 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1456 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190923141041.896 ERROR    attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 43886
lxc ag-k8s-slave-1 20190923141043.358 ERROR    attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 45204
lxc ag-k8s-slave-1 20190923144335.971 ERROR    attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 23622
lxc ag-k8s-slave-1 20190923144350.645 ERROR    attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 28149
lxc ag-k8s-slave-1 20190923144616.886 ERROR    attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 21767
lxc ag-k8s-slave-1 20190923155424.847 ERROR    attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 14714
lxc ag-k8s-slave-1 20190923213455.212 ERROR    attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 40401

On the logs just above, why do I get values such as ag-k8s-slave-1-1, ag-k8s-slave-1-2… I tried to stop the container or restart or delete it several times, could it be the reason?

  • Last, I think the container might not be able to stop because of a kernel module: libceph. On my host machine the command dmesg output:
[349015.985763] libceph: mon2 10.0.3.12:6789 connect error
[349017.009300] libceph: connect 10.0.3.12:6789 error -101
[349017.009749] libceph: mon2 10.0.3.12:6789 connect error
[349018.993227] libceph: connect 10.0.3.12:6789 error -101
[349018.993667] libceph: mon2 10.0.3.12:6789 connect error
[349023.089100] libceph: connect 10.0.3.12:6789 error -101
[349023.089535] libceph: mon2 10.0.3.12:6789 connect error
[349031.024830] libceph: connect 10.0.3.12:6789 error -101
[349031.025261] libceph: mon2 10.0.3.12:6789 connect error

Do you know if it is possible to force a kernel module to stop in order to be able to stop the container?
I am not able to get inside the container:

lxc exec ag-k8s-slave-1 bash
Error: EOF

LXC version is 3.17
Kernel version is Linux n2 4.18.0-25-generic #26-Ubuntu SMP Mon Jun 24 09:32:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

The container is a privileged one to be able to run docker and kubernetes inside. Is it possible to run docker and k8s with a unprivileged container?

I can provide more logs if needed.

By advance thank you if anyone can help.