"lxc stop $CONTAINER" freezes


(Agostino) #1

Hi all,

I’m referring to this issue: https://github.com/lxc/lxd/issues/3687

Right now, I have the systemd fs mounted and if I install a fresh gentoo container I’m able to stop it.

While I’m doing some modification in the gentoo container (basically I recompiled the world), then I’m unable to stop it. The “lxc stop” command freezes; this is the verbose/debug:

ago@ctgate ~ $ lxc stop template2 --verbose --debug
DBUG[02-12|17:36:10] Connecting to a local LXD over a Unix socket 
DBUG[02-12|17:36:10] Sending request to LXD                   etag= method=GET url=http://unix.socket/1.0
DBUG[02-12|17:36:10] Got response struct from LXD 
DBUG[02-12|17:36:10] 
        {
                "config": {
                        "images.auto_update_interval": "0"
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\nMIIFQDCCAyigAwIBAgIQIEYS4oTN0U9CRyQj7d+KczANBgkqhkiG9w0BAQsFADA0\nMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QGN0\nZ2F0ZTAeFw0xODAyMDkxNTExNDFaFw0yODAyMDcxNTExNDFaMDQxHDAaBgNVBAoT\nE2xpbnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAY3RnYXRlMIICIjAN\nBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAxBkRxIGmWryDn8Dy1yf7U7r2tSxu\nFhDnwHErIgo1pDWypgtb9FNYuPRie1r3w/0Os1dNGICRVFQhb3zx/nul1gKYEhmN\nbHvzH6leSKofB2C77BUz7b/rgYxIMRTHP7Qw/CQSEwv0TFiS+IUn5iR6jvEc0TMe\nOrLPlWVI7sH94saK+ePA53jUTqTpMBRLTZxx6UyxaLnIJDYYB8g/Z4a5bAgTePLZ\nfGAfLY1eK0vFDbxrFjxyUvoF1laszsoS/XKOQ6M2Lyoec5v6qa1x6WobNu6CAlU3\n16QTPP2JJ3Qjc1VgtUqXd8wV7ooJ7+6ec78PXvh/axW2bnpWTqNUiu+rYrPnp7ue\nHmPJZASTX0+/T+7TxlLwG5GlR/4+J3PKya8fDxkRaD5H1Hqf4zj3V2Md5UzTuTue\nFp5Qy8wHAZzkfqkSRuYMLw/h7jcqvPTI3M/Aohaxg2JH4ObdfWyNhRv70tfbPKDS\nqL38T8w0bdSMLgTlOrFlqSSVB0RKU69mkrLLzZYYX5v4bTt2RyCP4GaqrZHgMY6s\nOqLbwU9NV3qVboCbV35knFehKk1gtXNU4LcxwemLP/P26ZYoTaXn4cFMPhV30ImD\nFWTop/2BFiV0/7r/zNqnLHeQpBGGa9wfASj1CbpNbwpFvFtXyaOMDtwWbENevB43\nEULq7sa4r83SY+UCAwEAAaNOMEwwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoG\nCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwFwYDVR0RBBAwDoIGY3RnYXRlhwTAqAAF\nMA0GCSqGSIb3DQEBCwUAA4ICAQAJT1cKWSwDejpFi8+JUFSvHywhzUUGXLUJF7IL\n/LdtS9gC6ombiW4cs4lm/cTVncrK8XVWmKjNbFN25eM8xI0w/ggAatHpGkHJAIPq\naM5SJYKzCyNAypNl9zoWnoRLLjy59CGMWKYCVInEwL1Z+SV5z/cCUjjTEEmvaIwT\nosrkZdOlEhxQ2TOjdzzl+5+xLvJSvplSq+pimXfw2pDFzfiGTh/BZzhZd/Kd6DA0\nBbY0ohmxYE61vFbq/I7aCBYm2q12d5ARPTnaIQWvdwFmvlo/rt4Doy/BdXijl9PI\n3Ji3nP1MHPfHD3ALC6TnWzg2aOlUDib/LB7Ljyh+L6pvqpd1xizvGlPSuIuxRLg8\nhYhVocq1jcJwGbpHOZ38TBmml2E9NVjxXqN6HTeaMpqXst6TXSIjMJxpyw43CEu0\nwd2PL7nz6kNj7xMagDVPgN+whMSu+DrZ7DZPHOK1pv8nXJy8Oz9ek3g/zLcKMD8P\nTX84Ti5sSCdEotTJP5k2kzB3K8mnxLx6dTZeSSYdb2ouXCz3epJin4DC6TxMIsax\nj1xNc92BzYykssLSQl2kvdKep/ttAni6ouFShpwK5M7iNY8ed86HfaeLdOU/OC1z\nIjK2Er1439klE5ZwbPL3IVvYccd2lM/TfDVHsGis4RNRtEKWx4csU8ljMvKIf18/\nmwcrdQ==\n-----END CERTIFICATE-----\n",
                        "certificate_fingerprint": "92d91a065b9bb83c62a719ba815756e778e657e8daf030f0d14e7f5aaf66d27d",
                        "driver": "lxc",
                        "driver_version": "2.1.1",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_version": "4.9.74-gentoo",
                        "server": "lxd",
                        "server_pid": 2623,
                        "server_version": "2.21",
                        "storage": "dir",
                        "storage_version": "1"
                }
        } 
DBUG[02-12|17:36:10] Connected to the websocket 
DBUG[02-12|17:36:10] Sending request to LXD                   etag= method=PUT url=http://unix.socket/1.0/containers/template2/state
DBUG[02-12|17:36:10] 
        {
                "action": "stop",
                "timeout": -1,
                "force": false,
                "stateful": false
        } 
DBUG[02-12|17:36:10] Got operation from LXD 
DBUG[02-12|17:36:10] 
        {
                "id": "80591ad9-da24-4ec1-ac21-722249b3766d",
                "class": "task",
                "created_at": "2018-02-12T17:36:10.857268356+01:00",
                "updated_at": "2018-02-12T17:36:10.857268356+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/template2"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": ""
        } 
DBUG[02-12|17:36:10] Sending request to LXD                   etag= method=GET url=http://unix.socket/1.0/operations/80591ad9-da24-4ec1-ac21-722249b3766d
DBUG[02-12|17:36:10] Got response struct from LXD 
DBUG[02-12|17:36:10] 
        {
                "id": "80591ad9-da24-4ec1-ac21-722249b3766d",
                "class": "task",
                "created_at": "2018-02-12T17:36:10.857268356+01:00",
                "updated_at": "2018-02-12T17:36:10.857268356+01:00",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/template2"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": ""
        }

In short this isn’t an LXD issue and this is the reason because I’m not continuing to write into the github ticket.
This is a container issue but I’d like to understand why it fails to stop


Can't stop container..stays in running state althoug IP address dissappears
(Stéphane Graber) #2

Does passing --force to lxc stop properly stopd the container?

If so, then the problem is that your init system in the container isn’t performing a clean shutdown when signaled by LXD. This could be a bug in the init system itself or in one of the init scripts.


(Agostino) #3

I found the issue.

The world recompilation modified (via dispatch-conf) the /etc/inittab file which contains the string “pf:12345:powerwait:/sbin/halt”

So without that string it failed to shutdown.


(Bruce) #4

I have this problem consistently with a Debian 9 host and Debian 9 containers… Are there any other suggestions or remedies other than lxc stop my-container --force?


(Stéphane Graber) #5

You may want to look at lxc console to see what it’s getting stuck on.


Can't stop debian 8, 9 and 10 containers
(John M Drescher) #6

I had the same problem on a several year old gentoo container today and this solved the issue for me thanks.