Can't stop debian 8, 9 and 10 containers


(Roman V. Isaev) #1

It seems to happen on two different hosts: Ubuntu 18.04 (real machine) and CentOS 7 (remote KVM). Both running lxd 3.11 from snapd. If I create debian container:

 lxc launch images:debian/9 deb

and later try to shutdown it, nothing happens, lxc stop deb hangs. However, if I enter into second shell and issue second lxc stop deb command, then first lxc stop deb stops too and container is stopped. It seems, that debian 8, 9 and 10 images are affected, while alpine 3.9 and centos 7 aren’t. Did not try other images.

It seems that debian begins shutdown procedure, removes IP addresses and then something breaks. This is output of lxc list --format csv while lxc stop hanging, deb container had an ip address, but lxc stop removed ip, but did not finish shutdown procedure:

alp,RUNNING,172.16.172.31 (eth0),PERSISTENT,
cms,RUNNING,172.16.172.218 (eth0),PERSISTENT,
dd,RUNNING,172.16.172.47 (eth0),PERSISTENT,
deb,RUNNING,PERSISTENT,
static,RUNNING,172.16.172.57 (eth0),PERSISTENT,

Any ideas how to fix it?


(Bruce) #2

lxc stop --force [container-name] is the way forward here…


(Roman V. Isaev) #3

Forcing shutdown doesn’t look like good practice… something is wrong.


(Bruce) #4

Yeah, there are some debuging comments here…


(Roman V. Isaev) #5

Well, I see no errors in console after issuing first lxc stop deb:

[  OK  ] Stopped target Timers.
[  OK  ] Stopped Daily apt upgrade and clean activities.
[  OK  ] Removed slice system-getty.slice.
[  OK  ] Reached target Unmount All Filesystems.
[  OK  ] Stopped target Graphical Interface.
[  OK  ] Stopped target Multi-User System.
         Stopping Network Name Resolution...
         Stopping Login Service...
         Stopping D-Bus System Message Bus...
[  OK  ] Stopped target Login Prompts.
         Stopping Console Getty...
[  OK  ] Stopped Daily apt download activities.
[  OK  ] Stopped target System Time Synchronized.
[  OK  ] Stopped Daily Cleanup of Temporary Directories.
[  OK  ] Stopped Login Service.
[  OK  ] Stopped D-Bus System Message Bus.
[  OK  ] Stopped Network Name Resolution.
[  OK  ] Stopped Console Getty.
         Stopping Permit User Sessions...
[  OK  ] Stopped Permit User Sessions.
[  OK  ] Stopped target Remote File Systems.
[  OK  ] Stopped target Basic System.
[  OK  ] Stopped target Paths.
[  OK  ] Stopped target Slices.
[  OK  ] Removed slice User and Session Slice.
[  OK  ] Stopped target Sockets.
[  OK  ] Closed D-Bus System Message Bus Socket.
[  OK  ] Stopped target System Initialization.
[  OK  ] Stopped target Swap.
         Stopping Update UTMP about System Boot/Shutdown...
[  OK  ] Stopped target Encrypted Volumes.
[  OK  ] Stopped Forward Password Requests to Wall Directory Watch.
[  OK  ] Stopped Dispatch Password Requests to Console Directory Watch.
[  OK  ] Stopped target Network.
         Stopping Network Service...
         Stopping Raise network interfaces...
[  OK  ] Stopped Update UTMP about System Boot/Shutdown.
[  OK  ] Stopped Create Volatile Files and Directories.
[  OK  ] Stopped Network Service.

Here it hangs. If I issue lxc stop deb again in second shell, then this log continues:

[  OK  ] Stopped target Network.
         Stopping Network Service...
[  OK  ] Stopped Network Service.
[  OK  ] Reached target Shutdown.
Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Halting system.

And then it stops. Also, lxc stop --force deb kills container silently, no delays.

BTW, all faulty containers are bare containers from repository. I just ran lxc launch images:debian/9 deb and that’s all.


#6

It is likely the issue is related to the networking.
Because the first set of logs is interrupted at the Network Service and the second set continues at the Network Service.

Let’s verify this hypothesis.

$ lxc launch images:debian/9 deb
Creating deb
Starting deb
$ lxc shell deb
mesg: ttyname failed: Success
root@deb:~# ifdown eth0
Killed old client process
Internet Systems Consortium DHCP Client 4.3.5
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/00:16:3e:2f:45:17
Sending on   LPF/eth0/00:16:3e:2f:45:17
Sending on   Socket/fallback
DHCPRELEASE on eth0 to 10.100.100.1 port 67
root@deb:~# logout
$ lxc stop deb
$ lxc delete deb
$ 

Therefore, now the container shuts down cleanly.

What could be the issue? What commands does this Debian container image use to shutdown the networking? There is systemd in the container image, but networking is handled by ifupdown.


#7

when the container fails to stop is it still possible to do lxc exec bash ?
I had a similar problem and it was still possible and I found the problem in container’s syslog. It seems that the Debian 9 image don’t have rsyslog installed by default, but journalctl should give you the same information.