CENTOS 7 - Kernel 514 - 693 Cannot start any nodes after update

So I have ran some updates and rebooted my machine . After rebooting , I tried to start up the nodes again and I noticed they’re stuck in a stopped state…

Here are my logs that this spits out when in debug mode.

[root@dev ~]# lxd --debug
INFO[10-12|21:36:52] LXD 2.15 is starting in normal mode      path=/var/lib/lxd
WARN[10-12|21:36:52] AppArmor support has been disabled because of lack of kernel support
INFO[10-12|21:36:52] Kernel uid/gid map:
INFO[10-12|21:36:52]  - u 0 0 4294967295
INFO[10-12|21:36:52]  - g 0 0 4294967295
INFO[10-12|21:36:52] Configured LXD uid/gid map:
INFO[10-12|21:36:52]  - u 0 1000000 1000000000
INFO[10-12|21:36:52]  - g 0 1000000 1000000000
DBUG[10-12|21:36:52] Initializing and checking storage pool "default".
DBUG[10-12|21:36:52] Initializing a DIR driver.
DBUG[10-12|21:36:52] Checking DIR storage pool "default".
INFO[10-12|21:36:52] Connecting to a remote simplestreams server
INFO[10-12|21:36:52] Connecting to a remote simplestreams server
INFO[10-12|21:36:52] Expiring log files
INFO[10-12|21:36:52] Done expiring log files
INFO[10-12|21:36:52] Starting /dev/lxd handler
DBUG[10-12|21:36:52] Looking for existing certificates        cert=/var/lib/lxd/server.crt key=/var/lib/lxd/server.key
INFO[10-12|21:36:52] LXD isn't socket activated
INFO[10-12|21:36:52] REST API daemon:
INFO[10-12|21:36:52]  - binding Unix socket                   socket=/var/lib/lxd/unix.socket
INFO[10-12|21:36:52] Pruning expired images
INFO[10-12|21:36:52] Done pruning expired images
INFO[10-12|21:36:52] Updating images
DBUG[10-12|21:36:52] Processing image                         alias=16.04 fp=03c2fa6716b5f41684457ca5e1b7316df520715b7fea0378f9306d16fdc646ee protocol=simplestreams server=https://cloud-images.ubuntu.com/releases
INFO[10-12|21:36:52] Connecting to a remote simplestreams server
DBUG[10-12|21:36:52] Initializing a DIR driver.
INFO[10-12|21:36:52] Starting container                       action=start created=2017-10-02T14:56:47+0000 ephemeral=false name=centy stateful=false used=2017-10-09T19:31:55+0000
DBUG[10-12|21:36:52] handling                                 ip=@ method=GET url=/1.0
DBUG[10-12|21:36:52] Initializing a DIR driver.
DBUG[10-12|21:36:52]
        {
                "type": "sync",
                "status": "Success",
                "status_code": 200,
                "operation": "",
                "error_code": 0,
                "error": "",
                "metadata": {
                        "config": {},
                        "api_extensions": [
                                "storage_zfs_remove_snapshots",
                                "container_host_shutdown_timeout",
                                "container_syscall_filtering",
                                "auth_pki",
                                "container_last_used_at",
                                "etag",
                                "patch",
                                "usb_devices",
                                "https_allowed_credentials",
                                "image_compression_algorithm",
                                "directory_manipulation",
                                "container_cpu_time",
                                "storage_zfs_use_refquota",
                                "storage_lvm_mount_options",
                                "network",
                                "profile_usedby",
                                "container_push",
                                "container_exec_recording",
                                "certificate_update",
                                "container_exec_signal_handling",
                                "gpu_devices",
                                "container_image_properties",
                                "migration_progress",
                                "id_map",
                                "network_firewall_filtering",
                                "network_routes",
                                "storage",
                                "file_delete",
                                "file_append",
                                "network_dhcp_expiry",
                                "storage_lvm_vg_rename",
                                "storage_lvm_thinpool_rename",
                                "network_vlan",
                                "image_create_aliases",
                                "container_stateless_copy",
                                "container_only_migration",
                                "storage_zfs_clone_copy",
                                "unix_device_rename",
                                "storage_lvm_use_thinpool",
                                "storage_rsync_bwlimit",
                                "network_vxlan_interface",
                                "storage_btrfs_mount_options",
                                "entity_description",
                                "image_force_refresh",
                                "storage_lvm_lv_resizing",
                                "id_map_base",
                                "file_symlinks",
                                "container_push_target"
                        ],
                        "api_status": "stable",
                        "api_version": "1.0",
                        "auth": "trusted",
                        "public": false,
                        "environment": {
                                "addresses": [],
                                "architectures": [
                                        "x86_64",
                                        "i686"
                                ],
                                "certificate": "-----BEGIN CERTIFICATE-----\nMIIFdDCCA1ygAwIBAgIRAMqsodky16LPR4vMe8sZGkAwDQYJKoZIhvcNAQELBQAw\nQzEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEjMCEGA1UEAwwacm9vdEBs\nb2NhbGhvc3QubG9jYWxkb21haW4wHhcNMTcwNzAzMTgzNzAxWhcNMjcwNzAxMTgz\nNzAxWjBDMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMSMwIQYDVQQDDBpy\nb290QGxvY2FsaG9zdC5sb2NhbGRvbWFpbjCCAiIwDQYJKoZIhvcNAQEBBQADggIP\nADCCAgoCggIBAMQsJduyhlQt0GBLd65DCwY4mf5esPjkF6Eqg8KP+lQNP3FAl++i\n4sttzPxEQS2VHUnIuNufNGpfyc0FO6GP0+LON74JsixvMLeTGosk4SEFstqukXK/\nPBPyizgTzNzwk+/62MYr7EcyjtpgwoF8o/ctYOA5OAjCyF9y4c8I8PW78gxHM4bp\np9jbZ3Z8MqedG7ZQcuW9OA5JSSrzLOXiqCDO8jNMO+L1x4arveT4H/Uq8CwQnL3f\nRlctk0QIMgDf+sH5UD8viQpPVmwNKXhVV98tqSyYbMHKGewP+OMJUMmrQ7lsoCff\n/rtr3LQKArAQCqUvW/UPtMjQvLTuGWgmEJwRkGzKKOlvnSGh4a0BLSSDMXrO7un/\nXP0a14zMMV4sYtOuM+/T6zbk7t95MP9atXUp+HHn6TsbaGJ16ynoAaXk5BWKtKwM\nd8ZMvfKEEMB+l8QMeMZvXpm5JPvNs1HdxzWK0hqT3y7U7s6itdhhWki7lyA7yKQn\n7Pp8cR8lbaF5VhI9KVFCVXZD++M46PIUZHGmagCWgAfTb8DHScPdLBCVFFTcsTj9\npCW1tVEAu/GvbtVCQ+ANXgm3/1ksfdIzXrZb8paGmDw61s+RzeZSfOPv+hOz93pV\nLK/2Ok2Y6RFH/vPmMWObKtDdivjIKuQQoWSOt5tSBlTafWcgCu46KxrjAgMBAAGj\nYzBhMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMB\nAf8EAjAAMCwGA1UdEQQlMCOCFWxvY2FsaG9zdC5sb2NhbGRvbWFpbocECmQABYcE\nwKh6ATANBgkqhkiG9w0BAQsFAAOCAgEAcz2eQtTWu9yMrx89xU28y6kQbtw7FP4o\n7fxD+zchPBaEfalViaFdJ5bxljpNUt/+j/qNAumrRuHyx6T/g7RLfDWsnJw/QI7k\nz7aW/AqGiUa+PgWLjeg18df5loANfEYWi4ZBTe5VAh6V9BsT+2bWM7or6Ctz++fi\noFH4gNqCChrxsBP4vTvYqZI6XWmY0cfR9Csd7yg16XFy8AopHX4JIDLyb8dLmLlE\nUB+2vc8r4xqV92hVqIkrSIWqsyjuTH+NvnrhQnZK4+5dpTua4uJg2hCD6kj9GUE7\nZymKfpKOksE+IIcagWQYjw38Fs6pnaYuH+v36anzRuNSo2ajCCtFhAqB3r4r1PJ3\nI+oYIKeWbw5p8HtHCa8qMTq7I8F0vg+e92W4Sw32U9XplI22RssDJl9Crewv0zhE\nVzbiXFgdE0dedVeh9lBW7ZMQicbxJu4EiuAgKXte/SxK+a5ht7TRRyymsSimR8EC\nS7OFgpl4eppNmyUy8p4kRcJaRDQyj/wjemynGNuTe0jt6KM6ZSEHIJg2HlDO6lkI\nYR4s67KjsV7oTOxdOBkGXPVsfTsSAon00qvkgxO71Id+//+b5cHnFBKgVXpHpTJc\nwkhbmqR0/Hb5luWvJxbhbSA33x3zm0Y+E72dFxIpY/d0wC7oKDK3LzaJQtqec9gB\npnjnLp14TYU=\n-----END CERTIFICATE-----\n",
                                "certificate_fingerprint": "d07b69b84e69f017c325f7df6ecd46363c8c61da30b661875ad0ad0b4604c0f9",
                                "driver": "lxc",
                                "driver_version": "2.0.7",
                                "kernel": "Linux",
                                "kernel_architecture": "x86_64",
                                "kernel_version": "3.10.0-693.2.2.el7.x86_64",
                                "server": "lxd",
                                "server_pid": 29411,
                                "server_version": "2.15",
                                "storage": "dir",
                                "storage_version": "1"
                        }
                }
        }
DBUG[10-12|21:36:53] handling                                 ip=@ method=GET url=/internal/containers/14/onstart
DBUG[10-12|21:36:53] Initializing a DIR driver.
DBUG[10-12|21:36:53] Scheduler: container centy started: re-balancing
DBUG[10-12|21:36:53]
        {
                "type": "sync",
                "status": "Success",
                "status_code": 200,
                "operation": "",
                "error_code": 0,
                "error": "",
                "metadata": {}
        }
DBUG[10-12|21:36:53] handling                                 ip=@ method=GET url=/1.0
DBUG[10-12|21:36:53] Initializing a DIR driver.
DBUG[10-12|21:36:53]
        {
                "type": "sync",
                "status": "Success",
                "status_code": 200,
                "operation": "",
                "error_code": 0,
                "error": "",
                "metadata": {
                        "config": {},
                        "api_extensions": [
                                "storage_zfs_remove_snapshots",
                                "container_host_shutdown_timeout",
                                "container_syscall_filtering",
                                "auth_pki",
                                "container_last_used_at",
                                "etag",
                                "patch",
                                "usb_devices",
                                "https_allowed_credentials",
                                "image_compression_algorithm",
                                "directory_manipulation",
                                "container_cpu_time",
                                "storage_zfs_use_refquota",
                                "storage_lvm_mount_options",
                                "network",
                                "profile_usedby",
                                "container_push",
                                "container_exec_recording",
                                "certificate_update",
                                "container_exec_signal_handling",
                                "gpu_devices",
                                "container_image_properties",
                                "migration_progress",
                                "id_map",
                                "network_firewall_filtering",
                                "network_routes",
                                "storage",
                                "file_delete",
                                "file_append",
                                "network_dhcp_expiry",
                                "storage_lvm_vg_rename",
                                "storage_lvm_thinpool_rename",
                                "network_vlan",
                                "image_create_aliases",
                                "container_stateless_copy",
                                "container_only_migration",
                                "storage_zfs_clone_copy",
                                "unix_device_rename",
                                "storage_lvm_use_thinpool",
                                "storage_rsync_bwlimit",
                                "network_vxlan_interface",
                                "storage_btrfs_mount_options",
                                "entity_description",
                                "image_force_refresh",
                                "storage_lvm_lv_resizing",
                                "id_map_base",
                                "file_symlinks",
                                "container_push_target"
                        ],
                        "api_status": "stable",
                        "api_version": "1.0",
                        "auth": "trusted",
                        "public": false,
                        "environment": {
                                "addresses": [],
                                "architectures": [
                                        "x86_64",
                                        "i686"
                                ],
                                "certificate": "-----BEGIN CERTIFICATE-----\nMIIFdDCCA1ygAwIBAgIRAMqsodky16LPR4vMe8sZGkAwDQYJKoZIhvcNAQELBQAw\nQzEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEjMCEGA1UEAwwacm9vdEBs\nb2NhbGhvc3QubG9jYWxkb21haW4wHhcNMTcwNzAzMTgzNzAxWhcNMjcwNzAxMTgz\nNzAxWjBDMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMSMwIQYDVQQDDBpy\nb290QGxvY2FsaG9zdC5sb2NhbGRvbWFpbjCCAiIwDQYJKoZIhvcNAQEBBQADggIP\nADCCAgoCggIBAMQsJduyhlQt0GBLd65DCwY4mf5esPjkF6Eqg8KP+lQNP3FAl++i\n4sttzPxEQS2VHUnIuNufNGpfyc0FO6GP0+LON74JsixvMLeTGosk4SEFstqukXK/\nPBPyizgTzNzwk+/62MYr7EcyjtpgwoF8o/ctYOA5OAjCyF9y4c8I8PW78gxHM4bp\np9jbZ3Z8MqedG7ZQcuW9OA5JSSrzLOXiqCDO8jNMO+L1x4arveT4H/Uq8CwQnL3f\nRlctk0QIMgDf+sH5UD8viQpPVmwNKXhVV98tqSyYbMHKGewP+OMJUMmrQ7lsoCff\n/rtr3LQKArAQCqUvW/UPtMjQvLTuGWgmEJwRkGzKKOlvnSGh4a0BLSSDMXrO7un/\nXP0a14zMMV4sYtOuM+/T6zbk7t95MP9atXUp+HHn6TsbaGJ16ynoAaXk5BWKtKwM\nd8ZMvfKEEMB+l8QMeMZvXpm5JPvNs1HdxzWK0hqT3y7U7s6itdhhWki7lyA7yKQn\n7Pp8cR8lbaF5VhI9KVFCVXZD++M46PIUZHGmagCWgAfTb8DHScPdLBCVFFTcsTj9\npCW1tVEAu/GvbtVCQ+ANXgm3/1ksfdIzXrZb8paGmDw61s+RzeZSfOPv+hOz93pV\nLK/2Ok2Y6RFH/vPmMWObKtDdivjIKuQQoWSOt5tSBlTafWcgCu46KxrjAgMBAAGj\nYzBhMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMB\nAf8EAjAAMCwGA1UdEQQlMCOCFWxvY2FsaG9zdC5sb2NhbGRvbWFpbocECmQABYcE\nwKh6ATANBgkqhkiG9w0BAQsFAAOCAgEAcz2eQtTWu9yMrx89xU28y6kQbtw7FP4o\n7fxD+zchPBaEfalViaFdJ5bxljpNUt/+j/qNAumrRuHyx6T/g7RLfDWsnJw/QI7k\nz7aW/AqGiUa+PgWLjeg18df5loANfEYWi4ZBTe5VAh6V9BsT+2bWM7or6Ctz++fi\noFH4gNqCChrxsBP4vTvYqZI6XWmY0cfR9Csd7yg16XFy8AopHX4JIDLyb8dLmLlE\nUB+2vc8r4xqV92hVqIkrSIWqsyjuTH+NvnrhQnZK4+5dpTua4uJg2hCD6kj9GUE7\nZymKfpKOksE+IIcagWQYjw38Fs6pnaYuH+v36anzRuNSo2ajCCtFhAqB3r4r1PJ3\nI+oYIKeWbw5p8HtHCa8qMTq7I8F0vg+e92W4Sw32U9XplI22RssDJl9Crewv0zhE\nVzbiXFgdE0dedVeh9lBW7ZMQicbxJu4EiuAgKXte/SxK+a5ht7TRRyymsSimR8EC\nS7OFgpl4eppNmyUy8p4kRcJaRDQyj/wjemynGNuTe0jt6KM6ZSEHIJg2HlDO6lkI\nYR4s67KjsV7oTOxdOBkGXPVsfTsSAon00qvkgxO71Id+//+b5cHnFBKgVXpHpTJc\nwkhbmqR0/Hb5luWvJxbhbSA33x3zm0Y+E72dFxIpY/d0wC7oKDK3LzaJQtqec9gB\npnjnLp14TYU=\n-----END CERTIFICATE-----\n",
                                "certificate_fingerprint": "d07b69b84e69f017c325f7df6ecd46363c8c61da30b661875ad0ad0b4604c0f9",
                                "driver": "lxc",
                                "driver_version": "2.0.7",
                                "kernel": "Linux",
                                "kernel_architecture": "x86_64",
                                "kernel_version": "3.10.0-693.2.2.el7.x86_64",
                                "server": "lxd",
                                "server_pid": 29411,
                                "server_version": "2.15",
                                "storage": "dir",
                                "storage_version": "1"
                        }
                }
        }
DBUG[10-12|21:36:53] handling                                 ip=@ method=GET url="/internal/containers/14/onstop?target=stop"
EROR[10-12|21:36:53] stop hook failed                         container=centy err="Container is already running a start operation"
DBUG[10-12|21:36:53]
        {
                "error": "Container is already running a start operation",
                "error_code": 500,
                "type": "error"
        }
DBUG[10-12|21:36:53] Scheduler: network: vethTVH9GF has been added: updating network priorities
DBUG[10-12|21:36:54] Image already exists in the db           image=03c2fa6716b5f41684457ca5e1b7316df520715b7fea0378f9306d16fdc646ee
DBUG[10-12|21:36:54] Already up to date                       fp=03c2fa6716b5f41684457ca5e1b7316df520715b7fea0378f9306d16fdc646ee
DBUG[10-12|21:36:54] Processing image                         alias=centos/7 fp=cc972a0b0b8026427b7aa1f8c6de232d2cca588eb5a751ffb2886079043d130f protocol=simplestreams server=https://images.linuxcontainers.org
INFO[10-12|21:36:54] Connecting to a remote simplestreams server
INFO[10-12|21:36:56] Downloading image                        alias=centos/7 server=https://images.linuxcontainers.org
EROR[10-12|21:36:58] Failed starting container                action=start created=2017-10-02T14:56:47+0000 ephemeral=false name=centy stateful=false used=2017-10-09T19:31:55+0000
INFO[10-12|21:37:13] Image downloaded                         alias=centos/7 server=https://images.linuxcontainers.org
INFO[10-12|21:37:13] Done updating images

This Error sticks out near the bottom.

EROR[10-12|21:36:53] stop hook failed                         container=centy err="Container is already running a start operation"

Any ideas for how I can fix this?

I am running this on CENTOS 7 . I see an app armor warning at the top. So I am not sure if that is anything to worry about either.

Let me know if you want more information.

 lxd --version
2.15

thanks!

What does “lxc info centy --show-log” get you?

Thanks for the reply.

Log:

        lxc 20171013170003.234 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:309 - Going to wait for pid 5117.
        lxc 20171013170003.235 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:328 - Trying to sync with child process.
        lxc 20171013170003.235 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
        lxc 20171013170003.235 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
        lxc 20171013170003.235 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:366 - Using pipe file descriptor 5 for monitord.
        lxc 20171013170003.238 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:343 - Sucessfully synced with child process.
        lxc 20171013170003.238 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:312 - Finished waiting on pid 5117.
        lxc 20171013170003.239 INFO     lxc_container - lxccontainer.c:do_lxcapi_start:804 - Attempting to set proc title to [lxc monitor] /var/lib/lxd/containers centy
        lxc 20171013170003.239 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
        lxc 20171013170003.239 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver SELinux
        lxc 20171013170003.239 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
        lxc 20171013170003.239 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for reject_force_umount action 0.
        lxc 20171013170003.239 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
        lxc 20171013170003.240 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:309 - Going to wait for pid 5121.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for reject_force_umount action 0.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .[all].
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .kexec_load errno 38.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for kexec_load action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for kexec_load action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .open_by_handle_at errno 38.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for open_by_handle_at action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for open_by_handle_at action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .init_module errno 38.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for init_module action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for init_module action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .finit_module errno 38.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for finit_module action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for finit_module action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .delete_module errno 38.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for delete_module action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for delete_module action 327718.
        lxc 20171013170003.240 INFO     lxc_seccomp - seccomp.c:parse_config_v2:580 - Merging in the compat Seccomp ctx into the main one.
        lxc 20171013170003.240 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:328 - Trying to sync with child process.
        lxc 20171013170003.240 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/root/go/bin/lxd callhook /var/lib/lxd 14 start" for container "centy", config section "lxc".
        lxc 20171013170003.240 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 3.
        lxc 20171013170003.240 INFO     lxc_start - start.c:lxc_check_inherited:235 - Closed inherited fd: 6.
        lxc 20171013170003.240 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:366 - Using pipe file descriptor 5 for monitord.
        lxc 20171013170003.242 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:343 - Sucessfully synced with child process.
        lxc 20171013170003.243 DEBUG    lxc_monitor - monitor.c:lxc_monitord_spawn:312 - Finished waiting on pid 5121.
        lxc 20171013170003.243 INFO     lxc_monitor - monitor.c:lxc_monitor_sock_name:185 - Using monitor socket name "lxc/d78a9d7e97b4b375//var/lib/lxd/containers".
        lxc 20171013170003.270 DEBUG    lxc_start - start.c:setup_signal_fd:273 - Set SIGCHLD handler with file descriptor: 5.
        lxc 20171013170003.271 DEBUG    console - console.c:lxc_console_peer_default:468 - no console peer
        lxc 20171013170003.271 INFO     lxc_start - start.c:lxc_init:475 - Container "centy" is initialized.
        lxc 20171013170003.271 ERROR    lxc_start - start.c:must_drop_cap_sys_boot:641 - Failed to clone (0x30000011): Invalid argument (includes CLONE_NEWUSER).
        lxc 20171013170003.271 DEBUG    lxc_start - start.c:__lxc_start:1312 - Dropping CAP_SYS_BOOT capability.
        lxc 20171013170003.273 INFO     lxc_confile - confile.c:config_idmap:1531 - read uid map: type u nsid 0 hostid 1000000 range 1000000000
        lxc 20171013170003.273 INFO     lxc_confile - confile.c:config_idmap:1531 - read uid map: type g nsid 0 hostid 1000000 range 1000000000
        lxc 20171013170003.279 INFO     lxc_conf - conf.c:instantiate_veth:2647 - Retrieved mtu 1500 from lxdbr0
        lxc 20171013170003.280 INFO     lxc_conf - conf.c:instantiate_veth:2672 - Attached 'veth4VLO5C': to the bridge 'lxdbr0':
        lxc 20171013170003.280 DEBUG    lxc_conf - conf.c:instantiate_veth:2689 - instantiated veth 'veth4VLO5C/vethT9A17I', index is '51'
        lxc 20171013170003.280 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for centy
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-1" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-1: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-2" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-2: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-3" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-3: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-4" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-4: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-5" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-5: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-6" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-6: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-7" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-7: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-8" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-8: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-9" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-9: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-10" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-10: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-11" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-11: No such file or directory
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/centy-12" already existed.
        lxc 20171013170003.280 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/centy-12: No such file or directory
        lxc 20171013170003.281 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:filter_and_set_cpus:474 - No isolated cpus detected.
        lxc 20171013170003.281 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:handle_cpuset_hierarchy:644 - "cgroup.clone_children" was already set to "1".
        lxc 20171013170003.283 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x3c020000): Invalid argument.
        lxc 20171013170003.283 ERROR    lxc_start - start.c:lxc_spawn:1149 - Invalid argument - Failed to clone a new set of namespaces.
        lxc 20171013170003.307 INFO     lxc_conf - conf.c:lxc_delete_network:3015 - Removed interface "eth0" with index 51.
        lxc 20171013170003.314 WARN     lxc_conf - conf.c:lxc_delete_network:3038 - Failed to remove "veth4VLO5C" from host: Invalid argument.
        lxc 20171013170003.314 ERROR    lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "centy".
        lxc 20171013170003.314 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/root/go/bin/lxd callhook /var/lib/lxd 14 stop" for container "centy", config section "lxc".
        lxc 20171013170003.348 ERROR    lxc_conf - conf.c:run_buffer:405 - Script exited with status 1.
        lxc 20171013170003.348 ERROR    lxc_start - start.c:lxc_fini:546 - Failed to run lxc.hook.post-stop for container "centy".
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/systemd//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/cpuset//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/net_cls//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/blkio//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/devices//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/hugetlb//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/pids//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/freezer//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/cpu//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/memory//lxc/centy-13
        lxc 20171013170003.348 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x10000000): Invalid argument.
        lxc 20171013170003.348 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1270 - Error destroying /sys/fs/cgroup/perf_event//lxc/centy-13
        lxc 20171013170003.349 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - Command get_cgroup failed to receive response: Connection reset by peer.
        lxc 20171013170003.349 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - Command get_cgroup failed to receive response: Connection reset by peer.
        lxc 20171013170008.364 INFO     lxc_confile - confile.c:config_idmap:1531 - read uid map: type u nsid 0 hostid 1000000 range 1000000000
        lxc 20171013170008.364 INFO     lxc_confile - confile.c:config_idmap:1531 - read uid map: type g nsid 0 hostid 1000000 range 1000000000
        lxc 20171013170039.431 INFO     lxc_confile - confile.c:config_idmap:1531 - read uid map: type u nsid 0 hostid 1000000 range 1000000000
        lxc 20171013170039.431 INFO     lxc_confile - confile.c:config_idmap:1531 - read uid map: type g nsid 0 hostid 1000000 range 1000000000
        lxc 20171013170039.447 INFO     lxc_confile - confile.c:config_idmap:1531 - read uid map: type u nsid 0 hostid 1000000 range 1000000000
        lxc 20171013170039.447 INFO     lxc_confile - confile.c:config_idmap:1531 - read uid map: type g nsid 0 hostid 1000000 range 1000000000
        lxc 20171013170003.283 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x3c020000): Invalid argument.

That’s the problem. What kernel are you running on this system? (uname -a output)

It sounds like your kernel got upgraded and regressed user namespace support in the process.
Does starting a privileged container work?

Linux dev.com 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I haven’t tried. I can try that shortly.

Ok the instructions are for an Ubuntu system. I am on Centos I will need to research how to do this on Centos 7,.

lxc launch images:centos/7 blah -c security.privileged=true

It is hanging in the starting process…

lxc launch images:centos/7 blah -c security.privileged=true
Creating blah
Starting blah

Since my shell was hung, I jumpedin with another session.

lxc list

±------±--------±-----±-----±-----------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±------±--------±-----±-----±-----------±----------+
| blah | STOPPED | | | PERSISTENT | 0 |
±------±--------±-----±-----±-----------±----------+
| centy | STOPPED | | | PERSISTENT | 0 |
±------±--------±-----±-----±-----------±----------+

Also , this suggests that I might have Namespaces enabled.

sudo cat /boot/config-$(uname -a | awk ‘{print $3}’) |grep ‘^CONFIG_USER_NS’
CONFIG_USER_NS=y

Am I out of luck? I can try re installing the product again and see what I get.

I’m not sure why a privileged container would be hanging like that.
I’d recommend you upgrade to LXD 2.18 as LXD 2.15 hasn’t been supported for quite a while.

I’m still pretty suspicious of the Centos kernel as I know that the RHEL kernel was recently patched to effectively disable user namespaces regardless of the kernel configuration file. This was done as a temporary measure until some kernel bugs are fixed.

For now, I am rolling with the old Kernel…

Is there a way I can get some attention from the CentOs team on this kind of thing? I mean, I really like this project, I just want better collaboration for enterprise solutions.

My understanding is that CentOS 7 is using the same kernel tree as RHEL 7 so I’m not sure that something can really be done at the CentOS level.

Eric Biederman who’s the namespace maintainer in the kernel and a Red Hat employee is aware that the Red Hat kernel team effectively crippled the use of user namespaces pending some more backporting happening.

There’s little that the LXD team can do about this other than regularly pinging our contacts at Red Hat (which we’ve been doing already). I expect the best way to get some traction on this would be for a Red Hat customer to directly open a support ticket with Red Hat, asking for the user namespace to be made functional again.

@brauner were you the one who tracked down the patch that Red Hat added to disable user namespaces even if the kernel config has them enabled or was that Aleksa?

Would be good to get a link on that particular commit/patch so that people know what to link to when complaining to Red Hat people :slight_smile:

Ok, so this was fun to track down again. The problem is that by default RHEL7 and friends disallow a) creation of user namespaces and b) creation of mount namespaces in user namespaces. Both are problematic for unprivileged containers. To solve a) you have to boot your kernel with user_namespace.enable=1 on the kernel command line. Unless there’s a sysctl I’m unaware of.

Now, about b). I took a closer look at the RHEL7 kernel sources. (Getting at the kernel sources for RHEL7 is… interesting. I think the only solid way is to use the Centos 7 kernel sources. That’s what I did.) If you look at:

linux-3.10.0-693.2.2.el7/fs/namespace.c

which is where new mount namespaces are created/copied etc. you’ll see:

/* namespace.unpriv_enable = 1 */
static bool enable_unpriv_mnt_ns_creation;
module_param_named(unpriv_enable, enable_unpriv_mnt_ns_creation, bool, 0444);
MODULE_PARM_DESC(unpriv_enable, "Enable unprivileged creation of mount namespaces");

struct mnt_namespace *copy_mnt_ns(unsigned long flags, struct mnt_namespace *ns,
		struct user_namespace *user_ns, struct fs_struct *new_fs)
{
	struct mnt_namespace *new_ns;
	struct vfsmount *rootmnt = NULL, *pwdmnt = NULL;
	struct mount *p, *q;
	struct mount *old;
	struct mount *new;
	int copy_flags;

	BUG_ON(!ns);

	if (likely(!(flags & CLONE_NEWNS))) {
		get_mnt_ns(ns);
		return ns;
	}

	/* Unprivileged creation currently tech preview in RHEL7  */
	if (user_ns != &init_user_ns) {
		static int __read_mostly called_mark_tech_preview = 0;
		if (!enable_unpriv_mnt_ns_creation) {
			return ERR_PTR(-EPERM);
		}
		if (!called_mark_tech_preview &&
		    !xchg(&called_mark_tech_preview, 1))
			mark_tech_preview("unpriv mount namespace", NULL);
	}

	old = ns->root;

	new_ns = alloc_mnt_ns(user_ns);
	if (IS_ERR(new_ns))
		return new_ns;

	namespace_lock();
	/* First pass: copy the tree topology */
	copy_flags = CL_COPY_UNBINDABLE | CL_EXPIRE;
	if (user_ns != ns->user_ns)
		copy_flags |= CL_SHARED_TO_SLAVE | CL_UNPRIVILEGED;
	new = copy_tree(old, old->mnt.mnt_root, copy_flags);
	if (IS_ERR(new)) {
		namespace_unlock();
		free_mnt_ns(new_ns);
		return ERR_CAST(new);
	}
	new_ns->root = new;
	list_add_tail(&new_ns->list, &new->mnt_list);

	/*
	 * Second pass: switch the tsk->fs->* elements and mark new vfsmounts
	 * as belonging to new namespace.  We have already acquired a private
	 * fs_struct, so tsk->fs->lock is not needed.
	 */
	p = old;
	q = new;
	while (p) {
		q->mnt_ns = new_ns;
		if (new_fs) {
			if (&p->mnt == new_fs->root.mnt) {
				new_fs->root.mnt = mntget(&q->mnt);
				rootmnt = &p->mnt;
			}
			if (&p->mnt == new_fs->pwd.mnt) {
				new_fs->pwd.mnt = mntget(&q->mnt);
				pwdmnt = &p->mnt;
			}
		}
		p = next_mnt(p, old);
		q = next_mnt(q, new);
		if (!q)
			break;
		while (p->mnt.mnt_root != q->mnt.mnt_root)
			p = next_mnt(p, old);
	}
	namespace_unlock();

	if (rootmnt)
		mntput(rootmnt);
	if (pwdmnt)
		mntput(pwdmnt);

	return new_ns;
}

which means that RHEL7 is indead blocking the creation of mount namespaces in user namespaces. But it seems similar to the user namespace command line option there’s a command line option to enable creation of mount namespaces in user namespaces namespace.unpriv_enable = 1.
TL;DR, boot your kernel with user_namespace.enable=1 namespace.unpriv_enable = 1 and you should be good to go.

2 Likes

Nice work.

I take it this isn’t enough?

grubby --args=“user_namespace.enable=1” --update-kernel="$(grubby --default-kernel)"

I don’t think it is enough. In simple terms, most permission checks are performed against the user namespace something is currently running in. That means you need to be able to e.g. create new mount namespaces in user namespaces to e.g. be able to change the mount table of that user namespace. So you need namespace.unpriv_enable=1 on the kernel command line too. :slight_smile:

I’m testing User Namespaces with Docker on CentOS 7.4, My containers are now working with only namespace.unpriv_enable=1 on the commandline. Looks like User Namespaces might be enabled on CentOS 7.4 by default?

What worked for me was

echo 10000 > /proc/sys/user/max_user_namespaces

Which you can set in a file /etc/sysctl.d/99-foo.conf

See https://superuser.com/questions/1294215/is-it-safe-to-enable-user-namespaces-in-centos-7-4-and-how-to-do-it

1 Like