After container stop cgroup is still alive

nobody · July 27, 2021, 1:26pm

Sup guys!
I have stopped proxy-26.ms container, but i see:

# ps ax | grep proxy-26.ms
15616 pts/7    S+     0:00 grep --color=auto proxy-26.ms
# cat /sys/fs/cgroup/memory/ns/proxy-26.ms/tasks 
# cat /sys/fs/cgroup/memory/ns/proxy-26.ms/cgroup.procs
# find /sys/fs/cgroup/ -name proxy-26.ms
/sys/fs/cgroup/freezer/ns/proxy-26.ms
/sys/fs/cgroup/cpuset/ns/proxy-26.ms
/sys/fs/cgroup/net_cls,net_prio/ns/proxy-26.ms
/sys/fs/cgroup/blkio/ns/proxy-26.ms
/sys/fs/cgroup/memory/ns/proxy-26.ms
/sys/fs/cgroup/cpu,cpuacct/ns/proxy-26.ms
/sys/fs/cgroup/rdma/ns/proxy-26.ms
/sys/fs/cgroup/perf_event/ns/proxy-26.ms
/sys/fs/cgroup/devices/ns/proxy-26.ms
/sys/fs/cgroup/pids/ns/proxy-26.ms
/sys/fs/cgroup/systemd/ns/proxy-26.ms
/sys/fs/cgroup/unified/ns/proxy-26.ms

When i start proxy-26.ms it spawns new cgroup ns/proxy-26.ms-1 - why? How it can be? What did i wrong again?

P.S. Temporarily fix find /sys/fs/cgroup/ -type d -name my_container | tac | xargs rmdir

stgraber · July 27, 2021, 10:01pm

This is an unusual configuration, LXC doesn’t use /ns/ in its cgroup path.
Can you show more of your configs and what you did to get this?

nobody · July 28, 2021, 6:47am

Hi, Stéphane! Very glad to see you again!
Yep, /ns/ i did it by myself, dunno why, just copypaste from proxmox. But i dont think this is the reason. I have it on hundereds of my CTs, but only several of them have the problem.

# cat /var/lib/lxc/proxy-26.ms/config

# Common and Puppet specified includes
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.include = /var/lib/lxc/proxy-26.ms/conf.d

# Distribution configuration
lxc.arch = x86_64

# Container specific configuration
lxc.rootfs.path = loop:/var/lib/lxc/proxy-26.ms/rootdev
lxc.uts.name = proxy-26.ms

# Local additions
lxc.monitor.unshare = 1
lxc.environment = TERM=linux
lxc.tty.max = 2
lxc.autodev = 1
lxc.start.auto = 1
lxc.log.level = 4
lxc.log.syslog = daemon
lxc.prlimit.nofile = 10000

# Apparmor section
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 0
lxc.apparmor.raw = deny mount -> /proc/,
lxc.apparmor.raw = deny mount -> /sys/,

# Cgroups common
lxc.cgroup.dir.monitor = lxc.monitor/proxy-26.ms
lxc.cgroup.dir.container = lxc/proxy-26.ms
lxc.cgroup.dir.container.inner = ns

# Hooks & bindings
lxc.hook.pre-start = /usr/local/sbin/lxc/hook_pre_start_is_ip_exists 192.168.8.216/23 192.168.9.11/24
lxc.hook.stop = /usr/local/sbin/lxc/hook_stop
lxc.hook.mount = /var/lib/lxc/proxy-26.ms/setup_routes

# cat /var/lib/lxc/proxy-26.ms/conf.d/memory.limit_in_bytes.conf 
lxc.cgroup.memory.limit_in_bytes = 5G

# cat /var/lib/lxc/proxy-26.ms/conf.d/memory.memsw.limit_in_bytes.conf 
lxc.cgroup.memory.memsw.limit_in_bytes = 5G

# cat /var/lib/lxc/proxy-26.ms/conf.d/networking.conf 
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 192.168.8.216/23
lxc.net.0.link = br8
lxc.net.0.name = eth0
lxc.net.0.type = veth
lxc.net.0.script.up = /usr/local/sbin/lxc/hook_net_up

lxc.net.1.flags = up
lxc.net.1.ipv4.address = 192.168.9.11/24
lxc.net.1.ipv4.gateway = 192.168.9.1
lxc.net.1.link = br37
lxc.net.1.name = eth1
lxc.net.1.type = veth
lxc.net.1.script.up = /usr/local/sbin/lxc/hook_net_up

###
Debian (10) 4.19.194-2
All packages from http://deb.debian.org/debian buster/main amd64 Packages
ii  liblxc1                              1:3.1.0+really3.0.3-8        amd64        Linux Containers userspace tools (library)
ii  lxc                                  1:3.1.0+really3.0.3-8        amd64        Linux Containers userspace tools
ii  lxc-exporter                         0.2.1                        amd64        Simple lxc eporter for Prometheus
ii  lxc-templates                        3.0.4-0+deb10u1              amd64        Linux Containers userspace tools (templates)
ii  lxcfs                                3.0.3-2                      amd64        FUSE based filesystem for LXC

stgraber · July 28, 2021, 4:28pm

@brauner can you comment on behavior with the above config and why the cgroups may not be deleted on shutdown?

nobody · July 28, 2021, 4:41pm

Ehehe, i am here to ask you about it
I dunno why. There is, on the same host, for example, i have proxy-25.ms with the same config, different only name and IPs. And there is no problem with him, and with others. I have only several (2 or maybe 3, dunno) CTs with this behavior. I have no idea what wrong with them, and what i did to have this behavior. They just stay and work many weeks, maybe months before i found it.

brauner · July 29, 2021, 4:17pm

@Blub knows this better.

nobody · August 3, 2021, 7:35am

Now i have one i can’t “fix”:

# rmdir /sys/fs/cgroup/devices/ns/proxy-98.fr
rmdir: failed to remove '/sys/fs/cgroup/devices/ns/proxy-98.fr': Device or resource busy

also i found, as i think, old but the same issues:

brauner · August 3, 2021, 1:02pm

That error means that there’s either still a process alive in that cgroup or that it has subcgroups.

This instructs LXC to create the following cgroup layout:

lxc.monitor/proxy-26.ms

which is used for the container’s supervising process aka the monitor.

For the container the layout is:

lxc/proxy-26.ms

This is where the cgroup limit itself will be applied and:

lxc/proxy-26.ms/ns

This is where the container itself. This setting ensures that the container can’t escape its cgroup limits.

I’m confused though. Your package installation output shows:

but that version of LXC doesn’t support the config options you’re using. And new versions of LXC supporting them delete the cgroups just fine.

nobody · August 3, 2021, 3:47pm

Hi there, Christian! Nice to see you again too!

That error means that there’s either still a process alive in that cgroup or that it has subcgroups.

Well, there is no any processes, or i can’t find it

# cat /sys/fs/cgroup/devices/ns/proxy-98.fr/tasks
# cat /sys/fs/cgroup/devices/ns/proxy-98.fr/system.slice/tasks
# ls -l /sys/fs/cgroup/devices/ns/proxy-98.fr/
total 0
-rw-r--r-- 1 root root 0 Aug  3 10:16 cgroup.clone_children
-rw-r--r-- 1 root root 0 Jul 23 12:33 cgroup.procs
--w------- 1 root root 0 Jul 23 12:33 devices.allow
--w------- 1 root root 0 Jul 23 12:33 devices.deny
-r--r--r-- 1 root root 0 Aug  3 10:16 devices.list
-rw-r--r-- 1 root root 0 Aug  3 10:16 notify_on_release
drwxr-xr-x 9 root root 0 Aug  3 10:16 system.slice
-rw-r--r-- 1 root root 0 Aug  3 10:16 tasks
# ls -l /sys/fs/cgroup/devices/ns/proxy-98.fr/system.slice
total 0
-rw-r--r-- 1 root root 0 Aug  3 16:28 cgroup.clone_children
-rw-r--r-- 1 root root 0 Aug  3 16:28 cgroup.procs
drwxr-xr-x 2 root root 0 Aug  3 16:28 dev-hugepages.mount
--w------- 1 root root 0 Aug  3 16:28 devices.allow
--w------- 1 root root 0 Aug  3 16:28 devices.deny
-r--r--r-- 1 root root 0 Aug  3 16:28 devices.list
drwxr-xr-x 2 root root 0 Aug  3 16:28 dev-mqueue.mount
drwxr-xr-x 2 root root 0 Aug  3 16:28 dev-ptmx.mount
drwxr-xr-x 2 root root 0 Aug  3 16:28 -.mount
-rw-r--r-- 1 root root 0 Aug  3 16:28 notify_on_release
drwxr-xr-x 2 root root 0 Aug  3 16:28 sys-devices-virtual-net.mount
drwxr-xr-x 2 root root 0 Aug  3 16:28 sys-fs-fuse-connections.mount
drwxr-xr-x 2 root root 0 Aug  3 16:28 systemd-update-utmp.service
-rw-r--r-- 1 root root 0 Aug  3 16:28 tasks

lxc.cgroup.dir.monitor = lxc.monitor/proxy-26.ms
lxc.cgroup.dir.container = lxc/proxy-26.ms
lxc.cgroup.dir.container.inner = ns

I think there is just one of them who works: lxc.cgroup.dir.container.inner cuz i can’t find lxc.monitor or just lxc in /sys/fs/cgroup/ . Other two just being ignored.

I can’t just update or downgrade my lxc, i just can use only what we have on debian 10. I believe it should work regardless of version, cuz it’s pretty critical.

nobody · August 4, 2021, 2:46pm

I found 1 “reason” of this behavior.

root@qdevice-2:/# reboot
# lxc-attach qdevice-2.ms
lxc-attach: qdevice-2.ms: attach.c: lxc_attach: 1187 No such file or directory - Failed to attach to mnt namespace of 24062
# lxc-ls -f
NAME            STATE   AUTOSTART GROUPS IPV4                       IPV6 UNPRIVILEGED
proxy-3.ms      RUNNING 1         -      192.168.8.7                -    false
proxy-5.ms      RUNNING 1         -      192.168.8.33               -    false
qdevice-2.ms    RUNNING 1         -      -                          -    false
# lxc-stop qdevice-2.ms
^C
# lxc-stop -k qdevice-2.ms
^C
# ps ax | grep qdevice-2.ms
11130 ?        Ss     0:00 [lxc monitor] /var/lib/lxc qdevice-2.ms
24995 pts/5    S+     0:00 grep --color=auto qdevice-2.ms

Such container can shutdown only with kill -9 after that, there will be /sys/fs/cgroup/ you can not remove.

Should i create new topic about that? How can i can debug it for you? May be with strace or somehow else?

nobody · August 5, 2021, 5:28pm

Another one just hunged up while stopping. Part of strace below

# strace -p 33017                                                                                                                                                                                                                             [301/331]
strace: Process 33017 attached                                                                                                                                                                                                                                                
epoll_wait(8, [{EPOLLIN|EPOLLHUP, {u32=2789014352, u64=94096932535120}}], 10, -1) = 1                                                                                                                                                                                         
recvmsg(10, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=0, uid=0, gid=0}}], msg_controllen=32, msg_flags=0}, 0) = 0                         
epoll_ctl(8, EPOLL_CTL_DEL, 10, NULL)   = 0                                                                                                                                                                                                                                   
close(10)                               = 0                                                                                                                                                                                                                                   
epoll_wait(8, [{EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 1                                                                                                                                                                                                  
accept(5, NULL, NULL)                   = 10                                                                                                                                                                                                                                  
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0                                                                                                                                                                                                                                   
setsockopt(10, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0                                                                                                                                                                                                                           
epoll_ctl(8, EPOLL_CTL_ADD, 10, {EPOLLIN, {u32=2789014160, u64=94096932534928}}) = 0                                                                                                                                                                                          
epoll_wait(8, [{EPOLLIN, {u32=2789014160, u64=94096932534928}}], 10, -1) = 1                                                                                                                                                                                                  
recvmsg(10, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=2961, uid=0, gid=0}}], msg_controllen=32, msg_flags=
0}, 0) = 16                                                                                                                                                                                                                                                                   
sendto(10, "\0\0\0\0\0\0\0\0\34\236\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16                                                                                                                                                                                             
epoll_wait(8, [{EPOLLIN|EPOLLHUP, {u32=2789014160, u64=94096932534928}}], 10, -1) = 1                                                                                                                                                                                         
recvmsg(10, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=0, uid=0, gid=0}}], msg_controllen=32, msg_flags=0}, 0) = 0                         
epoll_ctl(8, EPOLL_CTL_DEL, 10, NULL)   = 0                                                                                                                                                                                                                                   
close(10)                               = 0                                                                                                                                                                                                                                   
epoll_wait(8, [{EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 1                                                                                                                                                                                                  
accept(5, NULL, NULL)                   = 10                                                                                                                                                                                                                                  
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0                                                                                                                                                                                                                                   
setsockopt(10, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0                                                                                                                                                                                                                           
epoll_ctl(8, EPOLL_CTL_ADD, 10, {EPOLLIN, {u32=2789014352, u64=94096932535120}}) = 0                                                                                                                                                                                          
epoll_wait(8, [{EPOLLIN, {u32=2789014352, u64=94096932535120}}], 10, -1) = 1                                                                                                                                                                                                  
recvmsg(10, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=2961, uid=0, gid=0}}], msg_controllen=32, msg_flags=
0}, 0) = 16
sendto(10, "\0\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
epoll_wait(8, [{EPOLLIN|EPOLLHUP, {u32=2789014352, u64=94096932535120}}, {EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 2
recvmsg(10, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=0, uid=0, gid=0}}], msg_controllen=32, msg_flags=0}, 0) = 0
epoll_ctl(8, EPOLL_CTL_DEL, 10, NULL)   = 0
close(10)                               = 0
accept(5, NULL, NULL)                   = 10
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0
setsockopt(10, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
epoll_ctl(8, EPOLL_CTL_ADD, 10, {EPOLLIN, {u32=2789014160, u64=94096932534928}}) = 0
epoll_wait(8, [{EPOLLIN, {u32=2789014160, u64=94096932534928}}], 10, -1) = 1
recvmsg(10, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=2961, uid=0, gid=0}}], msg_controllen=32, msg_flags=
0}, 0) = 16
kill(40476, SIGKILL)                    = 0
openat(AT_FDCWD, "/sys/fs/cgroup/freezer//ns/proxy-26.ms/freezer.state", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 21
write(21, "THAWED", 6)                  = 6
close(21)                               = 0
epoll_wait(8, [{EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 1
accept(5, NULL, NULL)                   = 21
fcntl(21, F_SETFD, FD_CLOEXEC)          = 0
setsockopt(21, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
epoll_ctl(8, EPOLL_CTL_ADD, 21, {EPOLLIN, {u32=2789013776, u64=94096932534544}}) = 0
epoll_wait(8, [{EPOLLIN, {u32=2789013776, u64=94096932534544}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=3052, uid=0, gid=0}}], msg_controllen=32, msg_flags=
0}, 0) = 16
kill(40476, SIGKILL)                    = 0
openat(AT_FDCWD, "/sys/fs/cgroup/freezer//ns/proxy-26.ms/freezer.state", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 21
write(21, "THAWED", 6)                  = 6
close(21)                               = 0
epoll_wait(8, [{EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 1
accept(5, NULL, NULL)                   = 21
fcntl(21, F_SETFD, FD_CLOEXEC)          = 0
setsockopt(21, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
epoll_ctl(8, EPOLL_CTL_ADD, 21, {EPOLLIN, {u32=2789013776, u64=94096932534544}}) = 0
epoll_wait(8, [{EPOLLIN, {u32=2789013776, u64=94096932534544}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=3052, uid=0, gid=0}}], msg_controllen=32, msg_flags=
0}, 0) = 16
sendto(21, "\0\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
epoll_wait(8, [{EPOLLIN|EPOLLHUP, {u32=2789013776, u64=94096932534544}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=0, uid=0, gid=0}}], msg_controllen=32, msg_flags=0}, 0) = 0
epoll_ctl(8, EPOLL_CTL_DEL, 21, NULL)   = 0
close(21)                               = 0
epoll_wait(8, [{EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 1
accept(5, NULL, NULL)                   = 21
fcntl(21, F_SETFD, FD_CLOEXEC)          = 0
setsockopt(21, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
epoll_ctl(8, EPOLL_CTL_ADD, 21, {EPOLLIN, {u32=2789013968, u64=94096932534736}}) = 0
epoll_wait(8, [{EPOLLIN, {u32=2789013968, u64=94096932534736}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=3052, uid=0, gid=0}}], msg_controllen=32, msg_flags=
0}, 0) = 16
sendto(21, "\0\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
epoll_wait(8, [{EPOLLIN|EPOLLHUP, {u32=2789013968, u64=94096932534736}}, {EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 2
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=0, uid=0, gid=0}}], msg_controllen=32, msg_flags=0}, 0) = 0
epoll_ctl(8, EPOLL_CTL_DEL, 21, NULL)   = 0
close(21)                               = 0
accept(5, NULL, NULL)                   = 21
accept(5, NULL, NULL)                   = 21
fcntl(21, F_SETFD, FD_CLOEXEC)          = 0
setsockopt(21, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
epoll_ctl(8, EPOLL_CTL_ADD, 21, {EPOLLIN, {u32=2789013776, u64=94096932534544}}) = 0
epoll_wait(8, [{EPOLLIN, {u32=2789013776, u64=94096932534544}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=3052, uid=0, gid=0}}], msg_controllen=32, msg_flags=
0}, 0) = 16
sendto(21, "\0\0\0\0\0\0\0\0\2\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
epoll_wait(8, [{EPOLLIN|EPOLLHUP, {u32=2789013776, u64=94096932534544}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=0, uid=0, gid=0}}], msg_controllen=32, msg_flags=0}, 0) = 0
epoll_ctl(8, EPOLL_CTL_DEL, 21, NULL)   = 0
close(21)                               = 0
epoll_wait(8, [{EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 1
accept(5, NULL, NULL)                   = 21
fcntl(21, F_SETFD, FD_CLOEXEC)          = 0
setsockopt(21, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
epoll_ctl(8, EPOLL_CTL_ADD, 21, {EPOLLIN, {u32=2789013968, u64=94096932534736}}) = 0
epoll_wait(8, [{EPOLLIN, {u32=2789013968, u64=94096932534736}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\7\0\0\0\n\0\0\0V \250\361\"V\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=3052, uid=0, gid=0}}], msg_controllen=32, msg_flags
=0}, 0) = 16
recvfrom(21, "lxc.group\0", 10, 0, NULL, NULL) = 10
sendto(21, "\377\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
epoll_wait(8, [{EPOLLIN|EPOLLHUP, {u32=2789013968, u64=94096932534736}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=0, uid=0, gid=0}}], msg_controllen=32, msg_flags=0}, 0) = 0
epoll_ctl(8, EPOLL_CTL_DEL, 21, NULL)   = 0
close(21)                               = 0
epoll_wait(8, [{EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 1
accept(5, NULL, NULL)                   = 21
fcntl(21, F_SETFD, FD_CLOEXEC)          = 0
setsockopt(21, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
epoll_ctl(8, EPOLL_CTL_ADD, 21, {EPOLLIN, {u32=2789013776, u64=94096932534544}}) = 0
epoll_wait(8, [{EPOLLIN, {u32=2789013776, u64=94096932534544}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=3055, uid=0, gid=0}}], msg_controllen=32, msg_flags=
0}, 0) = 16
sendto(21, "\0\0\0\0\0\0\0\0\34\236\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
epoll_wait(8, [{EPOLLIN|EPOLLHUP, {u32=2789013776, u64=94096932534544}}], 10, -1) = 1
recvmsg(21, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="", iov_len=16}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=0, uid=0, gid=0}}], msg_controllen=32, msg_flags=0}, 0) = 0
epoll_ctl(8, EPOLL_CTL_DEL, 21, NULL)   = 0
close(21)                               = 0
epoll_wait(8, [{EPOLLIN, {u32=2789012816, u64=94096932533584}}], 10, -1) = 1
accept(5, NULL, NULL)                   = 21