LXC containers suddenly stopped working on Ubuntu 18.04

Hi,

Some of our LXC containers suddenly started causing kernel panic and after stopping them we have been unable to start them again. Does not give any obvious errors.

Host machine: Ubuntu 18.04.5 LTS
LXC version: 3.0.3
LXC client: Ubuntu 16.04

Debug log:

lxc-start container.name 20210802120735.820 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start container.name 20210802120735.820 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start container.name 20210802120735.820 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:714 - Using terminal "/dev/tty" as proxy
lxc-start container.name 20210802120735.821 DEBUG    terminal - terminal.c:lxc_terminal_signal_init:192 - Created signal fd 9
lxc-start container.name 20210802120735.821 DEBUG    terminal - terminal.c:lxc_terminal_winsz:90 - Set window size to 237 columns and 63 rows
lxc-start container.name 20210802120735.821 INFO     start - start.c:lxc_init:897 - Container "container.name" is initialized
lxc-start container.name 20210802120735.823 INFO     network - network.c:instantiate_veth:147 - Retrieved mtu 1500 from br0
lxc-start container.name 20210802120735.823 INFO     network - network.c:instantiate_veth:175 - Attached "vethCY7394" to bridge "br0"
lxc-start container.name 20210802120735.823 DEBUG    network - network.c:instantiate_veth:201 - Instantiated veth "vethCY7394/veth8XGMRB", index is "66"
lxc-start container.name 20210802120735.826 INFO     network - network.c:instantiate_veth:147 - Retrieved mtu 1500 from br1
lxc-start container.name 20210802120735.826 INFO     network - network.c:instantiate_veth:175 - Attached "vethE86H86" to bridge "br1"
lxc-start container.name 20210802120735.826 DEBUG    network - network.c:instantiate_veth:201 - Instantiated veth "vethE86H86/vethL83695", index is "68"
lxc-start container.name 20210802120735.827 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:620 - "cgroup.clone_children" was already set to "1"
lxc-start container.name 20210802120735.828 INFO     start - start.c:lxc_spawn:1688 - Cloned CLONE_NEWNS
lxc-start container.name 20210802120735.828 INFO     start - start.c:lxc_spawn:1688 - Cloned CLONE_NEWPID
lxc-start container.name 20210802120735.828 INFO     start - start.c:lxc_spawn:1688 - Cloned CLONE_NEWUTS
lxc-start container.name 20210802120735.828 INFO     start - start.c:lxc_spawn:1688 - Cloned CLONE_NEWIPC
lxc-start container.name 20210802120735.828 INFO     start - start.c:lxc_spawn:1688 - Cloned CLONE_NEWNET
lxc-start container.name 20210802120735.828 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved mnt namespace via fd 15
lxc-start container.name 20210802120735.828 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved pid namespace via fd 16
lxc-start container.name 20210802120735.828 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved uts namespace via fd 17
lxc-start container.name 20210802120735.828 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved ipc namespace via fd 18
lxc-start container.name 20210802120735.828 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved net namespace via fd 19
lxc-start container.name 20210802120735.828 INFO     cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2237 - Limits for the legacy cgroup hierarchies have been setup
lxc-start container.name 20210802120735.829 DEBUG    start - start.c:lxc_spawn:1742 - Preserved net namespace via fd 10
lxc-start container.name 20210802120735.864 DEBUG    network - network.c:lxc_network_move_created_netdev_priv:2500 - Moved network device "veth8XGMRB"/"(null)" to network namespace of 5422
lxc-start container.name 20210802120735.903 DEBUG    network - network.c:lxc_network_move_created_netdev_priv:2500 - Moved network device "vethL83695"/"(null)" to network namespace of 5422
lxc-start container.name 20210802120735.904 INFO     start - start.c:do_start:1242 - Unshared CLONE_NEWCGROUP
lxc-start container.name 20210802120735.905 DEBUG    storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "dir"
lxc-start container.name 20210802120735.905 DEBUG    conf - conf.c:lxc_mount_rootfs:1332 - Mounted rootfs "/home/lxc-containers/container.name/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc" with options "(null)"
lxc-start container.name 20210802120735.905 INFO     conf - conf.c:setup_utsname:791 - Set hostname to "container.name"
lxc-start container.name 20210802120735.928 DEBUG    network - network.c:setup_hw_addr:2767 - Mac address "00:16:9c:26:f6:1b" on "eth0" has been setup
lxc-start container.name 20210802120735.928 DEBUG    network - network.c:lxc_setup_netdev_in_child_namespaces:3032 - Network device "eth0" has been setup
lxc-start container.name 20210802120735.947 DEBUG    network - network.c:setup_hw_addr:2767 - Mac address "00:16:0a:48:16:02" on "eth1" has been setup
lxc-start container.name 20210802120735.948 DEBUG    network - network.c:lxc_setup_netdev_in_child_namespaces:3032 - Network device "eth1" has been setup
lxc-start container.name 20210802120735.948 INFO     network - network.c:lxc_setup_network_in_child_namespaces:3053 - network has been setup
lxc-start container.name 20210802120735.948 INFO     conf - conf.c:mount_autodev:1118 - Preparing "/dev"
lxc-start container.name 20210802120735.948 INFO     conf - conf.c:mount_autodev:1165 - Prepared "/dev"
lxc-start container.name 20210802120735.948 INFO     conf - conf.c:mount_file_entries:2333 - Finished setting up mounts
lxc-start container.name 20210802120735.948 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "container.name", config section "lxc"
lxc-start container.name 20210802120735.975 INFO     conf - conf.c:lxc_fill_autodev:1209 - Populating "/dev"
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:lxc_fill_autodev:1224 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/dev/full"
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:lxc_fill_autodev:1224 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/dev/null"
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:lxc_fill_autodev:1224 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/dev/random"
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:lxc_fill_autodev:1224 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/dev/tty"
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:lxc_fill_autodev:1224 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/dev/urandom"
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:lxc_fill_autodev:1224 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/dev/zero"
lxc-start container.name 20210802120735.975 INFO     conf - conf.c:lxc_fill_autodev:1286 - Populated "/dev"
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:mount_entry:2027 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/sys/fs/fuse/connections" to respect bind or remount options
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:mount_entry:2048 - Flags for "/sys/fs/fuse/connections" were 4096, required extra flags are 0
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:mount_entry:2058 - Mountflags already were 4096, skipping remount
lxc-start container.name 20210802120735.975 DEBUG    conf - conf.c:mount_entry:2102 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/sys/fs/fuse/connections" with filesystem type "none"
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2027 - Remounting "/sys/kernel/debug" on "/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/debug" to respect bind or remount options
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2048 - Flags for "/sys/kernel/debug" were 4096, required extra flags are 0
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2058 - Mountflags already were 4096, skipping remount
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2102 - Mounted "/sys/kernel/debug" on "/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/debug" with filesystem type "none"
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2027 - Remounting "/sys/kernel/security" on "/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/security" to respect bind or remount options
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2048 - Flags for "/sys/kernel/security" were 4110, required extra flags are 14
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2102 - Mounted "/sys/kernel/security" on "/usr/lib/x86_64-linux-gnu/lxc/sys/kernel/security" with filesystem type "none"
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2027 - Remounting "/sys/fs/pstore" on "/usr/lib/x86_64-linux-gnu/lxc/sys/fs/pstore" to respect bind or remount options
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2048 - Flags for "/sys/fs/pstore" were 4110, required extra flags are 14
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2102 - Mounted "/sys/fs/pstore" on "/usr/lib/x86_64-linux-gnu/lxc/sys/fs/pstore" with filesystem type "none"
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:mount_entry:2102 - Mounted "mqueue" on "/usr/lib/x86_64-linux-gnu/lxc/dev/mqueue" with filesystem type "mqueue"
lxc-start container.name 20210802120735.976 INFO     conf - conf.c:mount_file_entries:2333 - Finished setting up mounts
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:lxc_setup_ttydir_console:1796 - Created directory for console and tty devices at "/usr/lib/x86_64-linux-gnu/lxc/dev/lxc"
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:lxc_setup_ttydir_console:1841 - Mounted "/dev/pts/2" onto "/usr/lib/x86_64-linux-gnu/lxc/dev/lxc/console"
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:lxc_setup_ttydir_console:1849 - Mounted "/dev/pts/2" onto "/usr/lib/x86_64-linux-gnu/lxc/dev/lxc/console"
lxc-start container.name 20210802120735.976 DEBUG    conf - conf.c:lxc_setup_ttydir_console:1851 - Console has been setup under "/usr/lib/x86_64-linux-gnu/lxc/dev/lxc/console" and mounted to "/usr/lib/x86_64-linux-gnu/lxc/dev/console"
lxc-start container.name 20210802120735.976 INFO     utils - utils.c:lxc_mount_proc_if_needed:1231 - I am 1, /proc/self points to "1"
lxc-start container.name 20210802120735.999 WARN     conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance
lxc-start container.name 20210802120735.999 DEBUG    conf - conf.c:lxc_setup_devpts:1653 - Mount new devpts instance with options "gid=5,newinstance,ptmxmode=0666,mode=0620,max=1024"
lxc-start container.name 20210802120735.999 DEBUG    conf - conf.c:lxc_setup_devpts:1672 - Created dummy "/dev/ptmx" file as bind mount target
lxc-start container.name 20210802120735.999 DEBUG    conf - conf.c:lxc_setup_devpts:1677 - Bind mounted "/dev/pts/ptmx" to "/dev/ptmx"
lxc-start container.name 20210802120736.198 DEBUG    conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/0" with master fd 11 and slave fd 14
lxc-start container.name 20210802120736.285 DEBUG    conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/1" with master fd 15 and slave fd 16
lxc-start container.name 20210802120736.363 DEBUG    conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/2" with master fd 17 and slave fd 18
lxc-start container.name 20210802120736.438 DEBUG    conf - conf.c:lxc_allocate_ttys:989 - Created tty "/dev/pts/3" with master fd 19 and slave fd 20
lxc-start container.name 20210802120736.449 INFO     conf - conf.c:lxc_allocate_ttys:1005 - Finished creating 4 tty devices
lxc-start container.name 20210802120736.498 DEBUG    conf - conf.c:lxc_setup_ttys:913 - Bind mounted "/dev/pts/0" onto "/dev/lxc/tty1"
lxc-start container.name 20210802120736.528 DEBUG    conf - conf.c:lxc_setup_ttys:913 - Bind mounted "/dev/pts/1" onto "/dev/lxc/tty2"
lxc-start container.name 20210802120736.556 DEBUG    conf - conf.c:lxc_setup_ttys:913 - Bind mounted "/dev/pts/2" onto "/dev/lxc/tty3"
lxc-start container.name 20210802120736.591 DEBUG    conf - conf.c:lxc_setup_ttys:913 - Bind mounted "/dev/pts/3" onto "/dev/lxc/tty4"
lxc-start container.name 20210802120736.604 INFO     conf - conf.c:lxc_setup_ttys:949 - Finished setting up 4 /dev/tty<N> device(s)
lxc-start container.name 20210802120736.641 DEBUG    conf - conf.c:setup_caps:2503 - Dropped mac_admin (33) capability
lxc-start container.name 20210802120736.651 DEBUG    conf - conf.c:setup_caps:2503 - Dropped mac_override (32) capability
lxc-start container.name 20210802120736.659 DEBUG    conf - conf.c:setup_caps:2503 - Dropped sys_time (25) capability
lxc-start container.name 20210802120736.667 DEBUG    conf - conf.c:setup_caps:2503 - Dropped sys_module (16) capability
lxc-start container.name 20210802120736.675 DEBUG    conf - conf.c:setup_caps:2503 - Dropped sys_rawio (17) capability
lxc-start container.name 20210802120736.681 DEBUG    conf - conf.c:setup_caps:2506 - Capabilities have been setup
lxc-start container.name 20210802120736.687 NOTICE   conf - conf.c:lxc_setup:3692 - The container "container.name" is set up
lxc-start container.name 20210802120736.774 INFO     lsm - lsm/lsm.c:lsm_process_label_set_at:178 - Set AppArmor label to "lxc-container-default-cgns"
lxc-start container.name 20210802120736.793 INFO     apparmor - lsm/apparmor.c:apparmor_process_label_set:249 - Changed apparmor profile to lxc-container-default-cgns
lxc-start container.name 20210802120736.123 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.deny" set to "a"
lxc-start container.name 20210802120736.127 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c *:* m"
lxc-start container.name 20210802120736.130 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "b *:* m"
lxc-start container.name 20210802120736.132 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 1:3 rwm"
lxc-start container.name 20210802120736.136 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 1:5 rwm"
lxc-start container.name 20210802120736.138 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 1:7 rwm"
lxc-start container.name 20210802120736.140 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 5:0 rwm"
lxc-start container.name 20210802120736.142 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 5:1 rwm"
lxc-start container.name 20210802120736.145 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 5:2 rwm"
lxc-start container.name 20210802120736.147 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 1:8 rwm"
lxc-start container.name 20210802120736.149 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 1:9 rwm"
lxc-start container.name 20210802120736.151 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 136:* rwm"
lxc-start container.name 20210802120736.153 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 10:229 rwm"
lxc-start container.name 20210802120736.156 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 254:0 rm"
lxc-start container.name 20210802120736.158 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 10:200 rwm"
lxc-start container.name 20210802120736.160 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 10:228 rwm"
lxc-start container.name 20210802120736.162 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2232 - Set controller "devices.allow" set to "c 10:232 rwm"
lxc-start container.name 20210802120736.163 INFO     cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2237 - Limits for the legacy cgroup hierarchies have been setup
lxc-start container.name 20210802120736.166 DEBUG    start - start.c:lxc_spawn:1817 - Preserved cgroup namespace via fd 20
lxc-start container.name 20210802120736.187 NOTICE   utils - utils.c:lxc_setgroups:1400 - Dropped additional groups
lxc-start container.name 20210802120736.189 NOTICE   start - start.c:start:2025 - Exec'ing "/sbin/init"
lxc-start container.name 20210802120736.237 NOTICE   start - start.c:post_start:2036 - Started "/sbin/init" with pid "5422"
lxc-start container.name 20210802120736.117 DEBUG    start - start.c:signal_handler:447 - Container init process 5422 exited
lxc-start container.name 20210802120736.117 DEBUG    start - start.c:__lxc_start:1982 - Unknown exit status for container "container.name" init 11
lxc-start container.name 20210802120736.117 INFO     error - error.c:lxc_error_set_and_log:54 - Child <5422> ended on signal (11)
lxc-start container.name 20210802120736.117 WARN     network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 66
lxc-start container.name 20210802120736.117 WARN     network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth1" with index 68
lxc-start container.name 20210802120736.117 DEBUG    network - network.c:lxc_delete_network:3180 - Deleted network devices
lxc-start container.name 20210802120736.118 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "container.name", config section "lxc"

The container config:

lxc.include = /usr/share/lxc/config/ubuntu.common.conf
#lxc.include = /home/lxc-containers/lxc_shared_config.conf
lxc.rootfs.path = dir:/home/lxc-containers/container.name/rootfs
lxc.mount.fstab = /home/lxc-containers/container.name/fstab
lxc.uts.name = container.name
lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.1.type = veth
lxc.net.1.flags = up
lxc.net.1.link = br1
lxc.net.0.hwaddr = 00:16:9c:26:f6:1b
lxc.net.1.hwaddr = 00:16:0a:48:16:02
#lxc.apparmor.profile = lxc-container-default-cgns

Any help would be appreciated.

That log suggests /sbin/init is called in the container and then pretty much immediately fails and exits.

There’s nothing obviously wrong (no error) at the LXC level, so it may be filesystem corruption?

Thank you for replying so quickly.

After investigating every possible problem we think that the issue is either Systemd has broken or there is a kernel mismatch between the container and the host.

We have had to recreate our containers and move the data over.

Containers don’t run kernels, that’s what makes them containers, so that can’t be the issue :slight_smile:

The systemd binary or a library it uses getting corrupted is the most likely explanation for such an immediate failure.