Hello,
I am not able to stop a LXC container, the command lxc delete ag-k8s-slave-1
is running but nothing happens. I tried several workaround such as:
- sudo ps -f -p
grep containers/ag-k8s-slave-1 /proc/*/mountinfo | sed 's-^/proc/--' | sed 's-/.*--' | tr '\n' ',' | sed 's/,$//'
- There was several process, I ran
nsenter -t <PID> -m -- umount /var/lib/lxd/storage-pools/lxd/containers/ag-k8s-slave-1
with the first PID only - After that, only zfs process were still running (see below). I don’t know how to stop a ZFS process, could you tell me if this is possible? Also, why am I getting several
Invalid argument
?
grep: /proc/40401/mountinfo: Invalid argument
/proc/40541/mountinfo:1582 672 0:135 /rootfs / rw,relatime - zfs syspool/lxd/containers/ag-k8s-slave-1 rw,xattr,posixacl
- The output of
lxc info --show-log ag-k8s-slave-1
is:
Name: ag-k8s-slave-1
Location: none
Remote: unix://
Architecture: x86_64
Created: 2019/09/21 16:29 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 17356
Resources:
Processes: 5
Disk usage:
root: 1.41GB
CPU usage:
CPU usage (in seconds): 14301
Memory usage:
Memory (current): 954.48MB
Memory (peak): 1.34GB
Log:
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.922 ERROR cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.monitor/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1319 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1456 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1319 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1456 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-1"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1319 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190921162938.923 ERROR cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1456 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc.payload/ag-k8s-slave-1-2"
lxc ag-k8s-slave-1 20190923141041.896 ERROR attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 43886
lxc ag-k8s-slave-1 20190923141043.358 ERROR attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 45204
lxc ag-k8s-slave-1 20190923144335.971 ERROR attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 23622
lxc ag-k8s-slave-1 20190923144350.645 ERROR attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 28149
lxc ag-k8s-slave-1 20190923144616.886 ERROR attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 21767
lxc ag-k8s-slave-1 20190923155424.847 ERROR attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 14714
lxc ag-k8s-slave-1 20190923213455.212 ERROR attach - attach.c:lxc_attach:1136 - No such file or directory - Failed to attach to mnt namespace of 40401
On the logs just above, why do I get values such as ag-k8s-slave-1-1
, ag-k8s-slave-1-2
… I tried to stop the container or restart or delete it several times, could it be the reason?
- Last, I think the container might not be able to stop because of a kernel module:
libceph
. On my host machine the commanddmesg
output:
[349015.985763] libceph: mon2 10.0.3.12:6789 connect error
[349017.009300] libceph: connect 10.0.3.12:6789 error -101
[349017.009749] libceph: mon2 10.0.3.12:6789 connect error
[349018.993227] libceph: connect 10.0.3.12:6789 error -101
[349018.993667] libceph: mon2 10.0.3.12:6789 connect error
[349023.089100] libceph: connect 10.0.3.12:6789 error -101
[349023.089535] libceph: mon2 10.0.3.12:6789 connect error
[349031.024830] libceph: connect 10.0.3.12:6789 error -101
[349031.025261] libceph: mon2 10.0.3.12:6789 connect error
Do you know if it is possible to force a kernel module to stop in order to be able to stop the container?
I am not able to get inside the container:
lxc exec ag-k8s-slave-1 bash
Error: EOF
LXC version is 3.17
Kernel version is Linux n2 4.18.0-25-generic #26-Ubuntu SMP Mon Jun 24 09:32:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
The container is a privileged one to be able to run docker and kubernetes inside. Is it possible to run docker and k8s with a unprivileged container?
I can provide more logs if needed.
By advance thank you if anyone can help.