CentOS 6 container on Debian 10 won't start (CentOS 7 OK)

Hi, my LXC is running on Debian 10, installed with apt (3.1.0).

I can create unprivileged CentOS containers fine, but whereas CentOS 7 works fine, CentOS 6 won’t start.

root@lxc:~# lxc-create -n test -B btrfs -t download -- -d centos -r 6 -a amd64
Using image from local cache
Unpacking the rootfs

---
You just created a Centos 6 x86_64 (20191205_07:08) container.
root@lxc:~# lxc-start -n test -F -l DEBUG -o /tmp/centos6.log
root@lxc:~# cat /tmp/centos6.log
lxc-start test 20191205160529.104 INFO     confile - confile.c:set_config_idmaps:1605 - Read uid map: type u nsid 0 hostid 300000 range 65537
lxc-start test 20191205160529.104 INFO     confile - confile.c:set_config_idmaps:1605 - Read uid map: type g nsid 0 hostid 300000 range 65537
lxc-start test 20191205160529.104 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start test 20191205160529.104 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start test 20191205160529.105 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:714 - Using terminal "/dev/tty" as proxy
lxc-start test 20191205160529.105 DEBUG    terminal - terminal.c:lxc_terminal_signal_init:192 - Created signal fd 9
lxc-start test 20191205160529.105 DEBUG    terminal - terminal.c:lxc_terminal_winsz:90 - Set window size to 135 columns and 24 rows
lxc-start test 20191205160529.106 INFO     start - start.c:lxc_init:904 - Container "test" is initialized
lxc-start test 20191205160529.106 DEBUG    storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "btrfs"
lxc-start test 20191205160529.110 INFO     network - network.c:instantiate_veth:147 - Retrieved mtu 1500 from lxcbr0
lxc-start test 20191205160529.110 INFO     network - network.c:instantiate_veth:175 - Attached "vethCNM75K" to bridge "lxcbr0"
lxc-start test 20191205160529.111 DEBUG    network - network.c:instantiate_veth:201 - Instantiated veth "vethCNM75K/vethOYP4K7", index is "40"
lxc-start test 20191205160529.111 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:620 - "cgroup.clone_children" was already set to "1"
lxc-start test 20191205160529.111 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUSER
lxc-start test 20191205160529.112 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNS
lxc-start test 20191205160529.112 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWPID
lxc-start test 20191205160529.112 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUTS
lxc-start test 20191205160529.112 INFO     start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWIPC
lxc-start test 20191205160529.112 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved user namespace via fd 14
lxc-start test 20191205160529.112 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved mnt namespace via fd 15
lxc-start test 20191205160529.112 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved pid namespace via fd 16
lxc-start test 20191205160529.112 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved uts namespace via fd 17
lxc-start test 20191205160529.112 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved ipc namespace via fd 18
lxc-start test 20191205160529.112 DEBUG    conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start test 20191205160529.112 DEBUG    conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start test 20191205160529.112 DEBUG    conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
lxc-start test 20191205160529.131 INFO     start - start.c:do_start:1148 - Unshared CLONE_NEWNET
lxc-start test 20191205160529.132 DEBUG    conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start test 20191205160529.132 DEBUG    conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start test 20191205160529.132 DEBUG    conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
lxc-start test 20191205160529.142 DEBUG    start - start.c:lxc_spawn:1754 - Preserved net namespace via fd 10
lxc-start test 20191205160529.154 DEBUG    network - network.c:lxc_network_move_created_netdev_priv:2500 - Moved network device "vethOYP4K7"/"(null)" to network namespace of 6539
lxc-start test 20191205160529.154 NOTICE   utils - utils.c:lxc_switch_uid_gid:1378 - Switched to gid 0
lxc-start test 20191205160529.154 NOTICE   utils - utils.c:lxc_switch_uid_gid:1387 - Switched to uid 0
lxc-start test 20191205160529.154 NOTICE   utils - utils.c:lxc_setgroups:1400 - Dropped additional groups
lxc-start test 20191205160529.154 INFO     start - start.c:do_start:1254 - Unshared CLONE_NEWCGROUP
lxc-start test 20191205160529.154 DEBUG    storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "btrfs"
lxc-start test 20191205160529.154 DEBUG    conf - conf.c:lxc_mount_rootfs:1332 - Mounted rootfs "/var/lib/lxc/test/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
lxc-start test 20191205160529.154 INFO     conf - conf.c:setup_utsname:791 - Set hostname to "test"
lxc-start test 20191205160529.154 DEBUG    network - network.c:setup_hw_addr:2767 - Mac address "00:16:3e:8d:27:b7" on "eth0" has been setup
lxc-start test 20191205160529.154 DEBUG    network - network.c:lxc_setup_netdev_in_child_namespaces:3032 - Network device "eth0" has been setup
lxc-start test 20191205160529.155 INFO     network - network.c:lxc_setup_network_in_child_namespaces:3053 - network has been setup
lxc-start test 20191205160529.155 INFO     conf - conf.c:mount_autodev:1118 - Preparing "/dev"
lxc-start test 20191205160529.155 INFO     conf - conf.c:mount_autodev:1165 - Prepared "/dev"
lxc-start test 20191205160529.155 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "test", config section "lxc"
lxc-start test 20191205160529.198 INFO     conf - conf.c:lxc_fill_autodev:1209 - Populating "/dev"
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/full" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/full"
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/null" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/null"
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/random" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/random"
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/tty" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/tty"
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/urandom" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/urandom"
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:lxc_fill_autodev:1282 - Bind mounted host device node "/dev/zero" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/zero"
lxc-start test 20191205160529.198 INFO     conf - conf.c:lxc_fill_autodev:1286 - Populated "/dev"
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:mount_entry:2027 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:mount_entry:2048 - Flags for "/sys/fs/fuse/connections" were 4096, required extra flags are 0
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:mount_entry:2058 - Mountflags already were 4096, skipping remount
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:mount_entry:2102 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc-start test 20191205160529.198 DEBUG    conf - conf.c:mount_entry:2102 - Mounted "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/proc" with filesystem type "proc"
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:mount_entry:2102 - Mounted "sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/sys" with filesystem type "sysfs"
lxc-start test 20191205160529.199 INFO     conf - conf.c:mount_file_entries:2333 - Finished setting up mounts
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_setup_dev_console:1771 - Mounted pts device "/dev/pts/1" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/console"
lxc-start test 20191205160529.199 INFO     utils - utils.c:lxc_mount_proc_if_needed:1231 - I am 1, /proc/self points to "1"
lxc-start test 20191205160529.199 WARN     conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_setup_devpts:1653 - Mount new devpts instance with options "gid=5,newinstance,ptmxmode=0666,mode=0620,max=1024"
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_setup_devpts:1672 - Created dummy "/dev/ptmx" file as bind mount target
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_setup_devpts:1677 - Bind mounted "/dev/pts/ptmx" to "/dev/ptmx"
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/0" with master fd 11 and slave fd 14
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/1" with master fd 15 and slave fd 16
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/2" with master fd 17 and slave fd 18
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/3" with master fd 19 and slave fd 20
lxc-start test 20191205160529.199 INFO     conf - conf.c:lxc_allocate_ttys:1005 - Finished creating 4 tty devices
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_setup_ttys:940 - Bind mounted "/dev/pts/0" onto "/dev/tty1"
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_setup_ttys:940 - Bind mounted "/dev/pts/1" onto "/dev/tty2"
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_setup_ttys:940 - Bind mounted "/dev/pts/2" onto "/dev/tty3"
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:lxc_setup_ttys:940 - Bind mounted "/dev/pts/3" onto "/dev/tty4"
lxc-start test 20191205160529.199 INFO     conf - conf.c:lxc_setup_ttys:949 - Finished setting up 4 /dev/tty<N> device(s)
lxc-start test 20191205160529.199 INFO     conf - conf.c:setup_personality:1716 - Set personality to "0x0"
lxc-start test 20191205160529.199 DEBUG    conf - conf.c:setup_caps:2529 - Capabilities have been setup
lxc-start test 20191205160529.199 NOTICE   conf - conf.c:lxc_setup:3716 - The container "test" is set up
lxc-start test 20191205160529.199 INFO     apparmor - lsm/apparmor.c:apparmor_process_label_set:1084 - AppArmor profile unchanged
lxc-start test 20191205160529.200 DEBUG    start - start.c:lxc_spawn:1829 - Preserved cgroup namespace via fd 19
lxc-start test 20191205160529.200 NOTICE   start - start.c:start:2037 - Exec'ing "/sbin/init"
lxc-start test 20191205160529.200 NOTICE   start - start.c:post_start:2048 - Started "/sbin/init" with pid "6539"
lxc-start test 20191205160529.200 NOTICE   start - start.c:signal_handler:430 - Received 17 from pid 6540 instead of container init 6539
lxc-start test 20191205160529.242 DEBUG    start - start.c:signal_handler:447 - Container init process 6539 exited
lxc-start test 20191205160529.242 DEBUG    start - start.c:__lxc_start:1994 - Unknown exit status for container "test" init 11
lxc-start test 20191205160529.242 INFO     error - error.c:lxc_error_set_and_log:54 - Child <6539> ended on signal (11)
lxc-start test 20191205160529.242 WARN     network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 40
lxc-start test 20191205160529.242 DEBUG    network - network.c:lxc_delete_network:3180 - Deleted network devices
lxc-start test 20191205160529.242 DEBUG    conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start test 20191205160529.242 DEBUG    conf - conf.c:idmaptool_on_path_and_privileged:2860 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start test 20191205160529.242 DEBUG    conf - conf.c:lxc_map_ids:2952 - Functional newuidmap and newgidmap binary found
lxc-start test 20191205160529.256 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "test", config section "lxc"
root@lxc:~# cat /var/lib/lxc/test/config
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: -d centos -r 6 -a amd64
# Template script checksum (SHA-1): 273c51343604eb85f7e294c8da0a5eb769d648f3
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)


# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = x86_64

# Container specific configuration
lxc.apparmor.profile = unconfined
lxc.apparmor.allow_nesting = 1
lxc.idmap = u 0 300000 65537
lxc.idmap = g 0 300000 65537
lxc.rootfs.path = btrfs:/var/lib/lxc/test/rootfs
lxc.uts.name = test

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:8d:27:b7

I’m still reasonably new to LXC, so it could be a simple fix, but I’m stuck. Thanks for any pointers!

I believe if you add vsyscall=emulate to your kernel boot command line this should fix it.

See also CentOS 6 images instantly stop (Failed to open dir "/sys/fs/cgroup/blkio//lxc.payload/container-name")

That fixed it, thank you! I should have asked sooner rather than spending hours trying to solve it myself.