Can't start CTs after LXD 3.15 upgrade

I fail to see how this is caused by cgroups. They seem to be setup correctly. The eexist on last error is only happening because a prior container start failed and the cgroups did not get cleaned up properly.

Clearing out the cgroup files did not alleviate the problem. Also upon the first run of the container after boot, there were no such errors as the files didn’t exist.

I might be able to set up a test machine at some point, but I’m fighting fires left and right here right now. Thanks for your reply.

Then what’s the startup failure in that log? That’s the only thing that appears to cause an error and tear down.

Here’s the full trace I have from when I attempted to start a stock Ubuntu container after having purged the cgroup dirs that it was complaining about in the other logs:

[root@team-dev2 user]# lxc info --show-log wdelgenio-test1
Name: wdelgenio-test1
Location: team-dev2
Remote: unix://
Architecture: x86_64
Created: 2019/07/24 14:03 UTC
Status: Stopped
Type: persistent
Profiles: default
Log:
lxc wdelgenio-test1 20190724143740.776 TRACE    commands - commands.c:lxc_cmd:303 - Connection refused - Command "get_state" failed to connect command socket
lxc wdelgenio-test1 20190724143740.776 TRACE    start - start.c:lxc_init_handler:774 - Created anonymous pair {3,5} of unix sockets
lxc wdelgenio-test1 20190724143740.776 TRACE    commands - commands.c:lxc_cmd_init:1310 - Created abstract unix socket "/var/snap/lxd/common/lxd/containers/wdelgenio-test1/command"
lxc wdelgenio-test1 20190724143740.776 TRACE    start - start.c:lxc_init_handler:786 - Unix domain socket 7 for command server is ready
lxc wdelgenio-test1 20190724143740.777 INFO     lxccontainer - lxccontainer.c:do_lxcapi_start:993 - Set process title to [lxc monitor] /var/snap/lxd/common/lxd/containers wdelgenio-test1
lxc wdelgenio-test1 20190724143740.778 INFO     start - start.c:lxc_check_inherited:311 - Closed inherited fd 4
lxc wdelgenio-test1 20190724143740.783 TRACE    start - start.c:lxc_start:2145 - Doing lxc_start
lxc wdelgenio-test1 20190724143740.783 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver nop
lxc wdelgenio-test1 20190724143740.783 TRACE    start - start.c:lxc_init:805 - Initialized LSM
lxc wdelgenio-test1 20190724143740.783 TRACE    seccomp - seccomp.c:get_new_ctx:488 - Added arch 2 to main seccomp context
lxc wdelgenio-test1 20190724143740.783 TRACE    seccomp - seccomp.c:get_new_ctx:496 - Removed native arch from main seccomp context
lxc wdelgenio-test1 20190724143740.783 TRACE    seccomp - seccomp.c:get_new_ctx:488 - Added arch 3 to main seccomp context
lxc wdelgenio-test1 20190724143740.783 TRACE    seccomp - seccomp.c:get_new_ctx:496 - Removed native arch from main seccomp context
lxc wdelgenio-test1 20190724143740.783 TRACE    seccomp - seccomp.c:get_new_ctx:501 - Arch 4 already present in main seccomp context
lxc wdelgenio-test1 20190724143740.783 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc wdelgenio-test1 20190724143740.783 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc wdelgenio-test1 20190724143740.783 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 38"
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 38"
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 38"
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 38"
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 38"
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327718(errno)
lxc wdelgenio-test1 20190724143740.784 INFO     seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc wdelgenio-test1 20190724143740.784 TRACE    seccomp - seccomp.c:parse_config_v2:1018 - Merged first compat seccomp context into main context
lxc wdelgenio-test1 20190724143740.784 TRACE    seccomp - seccomp.c:parse_config_v2:1034 - Merged second compat seccomp context into main context
lxc wdelgenio-test1 20190724143740.784 TRACE    start - start.c:lxc_init:812 - Read seccomp policy
lxc wdelgenio-test1 20190724143740.784 TRACE    start - start.c:lxc_serve_state_clients:474 - Set container state to STARTING
lxc wdelgenio-test1 20190724143740.784 TRACE    start - start.c:lxc_serve_state_clients:477 - No state clients registered
lxc wdelgenio-test1 20190724143740.784 TRACE    start - start.c:lxc_init:820 - Set container state to "STARTING"
lxc wdelgenio-test1 20190724143740.784 TRACE    start - start.c:lxc_init:883 - Set environment variables
lxc wdelgenio-test1 20190724143740.784 INFO     conf - conf.c:run_script_argv:374 - Executing script "/proc/439929/exe callhook /var/snap/lxd/common/lxd 118 start" for container "wdelgenio-test1"
lxc wdelgenio-test1 20190724143740.784 TRACE    conf - conf.c:run_script_argv:421 - Set environment variable: LXC_HOOK_TYPE=pre-start
lxc wdelgenio-test1 20190724143740.784 TRACE    conf - conf.c:run_script_argv:429 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc wdelgenio-test1 20190724143740.846 TRACE    start - start.c:lxc_init:890 - Ran pre-start hooks
lxc wdelgenio-test1 20190724143740.846 TRACE    start - start.c:setup_signal_fd:356 - Created signal file descriptor 4
lxc wdelgenio-test1 20190724143740.846 TRACE    start - start.c:lxc_init:901 - Set up signal fd
lxc wdelgenio-test1 20190724143740.847 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:676 - No such device - The process does not have a controlling terminal
lxc wdelgenio-test1 20190724143740.847 DEBUG    terminal - terminal.c:lxc_terminal_create_log_file:848 - Using "/var/snap/lxd/common/lxd/logs/wdelgenio-test1/console.log" as terminal log file
lxc wdelgenio-test1 20190724143740.847 TRACE    terminal - terminal.c:lxc_terminal_create_ringbuf:829 - Allocated 131072 byte terminal ringbuffer
lxc wdelgenio-test1 20190724143740.847 TRACE    start - start.c:lxc_init:909 - Created console
lxc wdelgenio-test1 20190724143740.847 TRACE    terminal - terminal.c:lxc_terminal_map_ids:1192 - Chowned terminal "/dev/pts/0"
lxc wdelgenio-test1 20190724143740.847 TRACE    start - start.c:lxc_init:916 - Chowned console
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1018 - basecginfo is:
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1019 - 11:blkio:/
10:hugetlb:/
9:memory:/
8:cpuacct,cpu:/
7:net_prio,net_cls:/
6:pids:/
5:devices:/
4:cpuset:/
3:freezer:/
2:perf_event:/
1:name=systemd:/
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 0: blkio
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 1: hugetlb
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 2: memory
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 3: cpuacct
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 4: cpu
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 5: net_prio
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 6: net_cls
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 7: pids
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 8: devices
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 9: cpuset
lxc wdelgenio-test1 20190724143740.847 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 10: freezer
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1022 - kernel subsystem 11: perf_event
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1025 - named subsystem 0: name=systemd
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:2589 - Writable cgroup hierarchies:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:999 -   Hierarchies:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   0: base_cgroup: /
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/systemd
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: name=systemd
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   1: base_cgroup: /
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/perf_event
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: perf_event
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   2: base_cgroup: /
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/freezer
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: freezer
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   3: base_cgroup: /
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/cpuset
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: cpuset
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   4: base_cgroup: /
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/devices
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: devices
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   5: base_cgroup: /
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/pids
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: pids
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   6: base_cgroup: /
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/net_cls,net_prio
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: net_cls
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       1: net_prio
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   7: base_cgroup: /
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/cpu,cpuacct
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: cpu
lxc wdelgenio-test1 20190724143740.848 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       1: cpuacct
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   8: base_cgroup: /
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/memory
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: memory
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   9: base_cgroup: /
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/hugetlb
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: hugetlb
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1004 -   10: base_cgroup: /
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1005 -       mountpoint:  /sys/fs/cgroup/blkio
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1006 -       controllers:
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1008 -       0: blkio
lxc wdelgenio-test1 20190724143740.849 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:61 - Initialized cgroup driver cgfsng
lxc wdelgenio-test1 20190724143740.849 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:64 - Running with legacy cgroup layout
lxc wdelgenio-test1 20190724143740.849 TRACE    start - start.c:lxc_init:923 - Initialized cgroup driver
lxc wdelgenio-test1 20190724143740.849 TRACE    start - start.c:lxc_init:930 - Initialized LSM
lxc wdelgenio-test1 20190724143740.849 INFO     start - start.c:lxc_init:932 - Container "wdelgenio-test1" is initialized
lxc wdelgenio-test1 20190724143740.849 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_filter_and_set_cpus:502 - Removed isolated or offline cpus from cpuset
lxc wdelgenio-test1 20190724143740.849 TRACE    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:616 - "cgroup.clone_children" was already set to "1"
lxc wdelgenio-test1 20190724143740.864 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1405 - The monitor process uses "lxc.monitor/wdelgenio-test1" as cgroup
lxc wdelgenio-test1 20190724143740.864 DEBUG    storage - storage/storage.c:get_storage_by_name:232 - Detected rootfs type "dir"
lxc wdelgenio-test1 20190724143740.865 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_filter_and_set_cpus:502 - Removed isolated or offline cpus from cpuset
lxc wdelgenio-test1 20190724143740.871 DEBUG    lxccontainer - lxccontainer.c:wait_on_daemonized_start:861 - First child 440282 exited
lxc wdelgenio-test1 20190724143740.871 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:864 - No such file or directory - Failed to receive the container state
lxc 20190724143740.871 WARN     commands - commands.c:lxc_cmd_rsp_recv:135 - Connection reset by peer - Failed to receive response for command "get_state"

@brauner any idea what that No such file or directory refers to here?

Yeah, I’m looking at the liblxc code and trying to figure out where it fails and why we don’t have an error message.
@wdelgenio can you check that this is not segfaulting, please.

I saw no core dumps when I was working on this, so I don’t believe there were any segfaults. Can’t be sure, but I no longer have the CentOS environment to test.

Hm, so for the container. It uses macvlan. Does the interface eth0 exist?

Yeah, that’s not it.

Here’s the log from 3.0.4 before I did the echo 1000 > /proc/sys/user/max_user_namespaces and unshare -U and unshare -m

Log:

lxc working-gopher 20190724155154.320 ERROR    start - start.c:lxc_spawn:1737 - Invalid argument - Failed to clone a new set of namespaces
lxc working-gopher 20190724155154.322 ERROR    start - start.c:__lxc_start:2019 - Failed to spawn container "working-gopher"
lxc working-gopher 20190724155154.323 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:851 - Received container state "ABORTING" instead of "RUNNING"
lxc working-gopher 20190724155154.754 ERROR    conf - conf.c:userns_exec_1:4311 - Failed to clone process in new user namespace
lxc working-gopher 20190724155154.755 WARN     cgfsng - cgroups/cgfsng.c:cgfsng_payload_destroy:1108 - Failed to destroy cgroups
lxc 20190724155154.105 WARN     commands - commands.c:lxc_cmd_rsp_recv:135 - Connection reset by peer - Failed to receive response for command "get_state"

I did eventually get that working-gopher container to run on 3.0.4

I really can’t explain the error at the moment. The only thing that comes to mind is a segfault.

Hi everyone,
I recently encountered a very similar problem, the containers did not start with the same error that the TS @wdelgenio indicated. In my case, the cause of the problems was commonplace - time desynchronization on cluster members. One of the servers was 10 seconds lag due to an ntp configuration error. After correcting ntp settings the containers were created and launched without problems. Maybe this will be useful to someone :slight_smile:

Too many dependencies on all the nodes working exactly 100% correct for the cluster to work well. We have seen one bad server cause lots of issues for the whole cluster.

!!!