Failed to fork off sandboxing environment for executing generators: Protocol error

Hi, Today I’ve been trying to run the not-yet-released Ubuntu 24.04 in LXC container on my Ubuntu 22.04. However, I’m having problems with running newer versions of Ubuntu or Debian. I’m able to start LXC container running Ubuntu 22.04 or Debian Bullseye, but I get the same errors when trying to run Ubuntu 24.04 or current Debian Testing (Trixie). For example, when I try Ubuntu 24.04, I’m able to create the container no problem:

$ sudo lxc-create --name mycontainer --template download -- --dist ubuntu --release noble --arch amd64
Using image from local cache
Unpacking the rootfs

---
You just created an Ubuntu noble amd64 (20240326_07:42) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

But I can’t start it:

$ sudo lxc-start -F --name=mycontainer  
systemd 255.2-3ubuntu2 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Ubuntu Noble Numbat (development branch)!

Initializing machine ID from random generator.
Failed to fork off sandboxing environment for executing generators: Protocol error
[!!!!!!] Failed to start up manager.
Exiting PID 1...

Since older versions of Debian/Ubuntu start fine inside LXC, I thought it’s maybe because of the change from cgroups1 to cgroups2? But since my host operating system is Ubuntu 22.04 that shouldn’t be a problem:

$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

The checkconfig for LXC shows the following on my system:

sudo lxc-checkconfig 
LXC version 5.0.0
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-6.5.0-26-generic
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled

Cgroup v1 mount points: 


Cgroup v2 mount points: 
/sys/fs/cgroup

Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup ns_cgroup: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Does anyone by any chance have any hint of an idea what might I be doing wrong? :slight_smile:

Same issue with debian trixie (debian testing).

Also hits redhat: Loading... (that bug is from 5 April 2024, so these are all recent.)

This bug is relevant: Failed to fork off sandboxing environment · Issue #29621 · systemd/systemd · GitHub which isolates the failure as /tmp being a symlink to /var/tmp However, it does not explain the LXC failure for me, because my container/rootfs/tmp really is a directory and not a symlink.

Still the same problem here with the most recent Ubuntu noble image (20240425_07:42). When I add lxc.init.cmd = /sbin/init --log-level=debug --default_standard_output=journal+console to my lxc config, I get the following output from systemd:

$ lxc-start -F -n dimtest-ubu24-1
systemd 255.4-1ubuntu8 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
Detected virtualization lxc.
CPUID func 1 0
CPUID result 50657 8000800 fffa3223 f8bfbff
CPUID is hypervisor: yes
CPUID func 0 0
CPUID result d 756e6547 6c65746e 49656e69
CPUID sig 'GenuineIntel'
CPUID func 80000000 0
CPUID result 80000008 756e6547 6c65746e 49656e69
CPUID func 21 0
CPUID result 2ff a88 a88 0
CPUID sig 'n/a'
Detected architecture x86-64.
Detected initialized system, this is not the first boot.
Kernel version 6.5.0-28-generic, our baseline is 4.15
Mounting cgroup to /sys/fs/cgroup/misc of type cgroup with options misc.
Mounting cgroup (cgroup) on /sys/fs/cgroup/misc (MS_NOSUID|MS_NODEV|MS_NOEXEC "misc")...
Mounting cgroup to /sys/fs/cgroup/perf_event of type cgroup with options perf_event.
Mounting cgroup (cgroup) on /sys/fs/cgroup/perf_event (MS_NOSUID|MS_NODEV|MS_NOEXEC "perf_event")...
Mounting cgroup to /sys/fs/cgroup/net_cls,net_prio of type cgroup with options net_cls,net_prio.
Mounting cgroup (cgroup) on /sys/fs/cgroup/net_cls,net_prio (MS_NOSUID|MS_NODEV|MS_NOEXEC "net_cls,net_prio")...
Mounting cgroup to /sys/fs/cgroup/devices of type cgroup with options devices.
Mounting cgroup (cgroup) on /sys/fs/cgroup/devices (MS_NOSUID|MS_NODEV|MS_NOEXEC "devices")...
Mounting cgroup to /sys/fs/cgroup/blkio of type cgroup with options blkio.
Mounting cgroup (cgroup) on /sys/fs/cgroup/blkio (MS_NOSUID|MS_NODEV|MS_NOEXEC "blkio")...
Mounting cgroup to /sys/fs/cgroup/cpu,cpuacct of type cgroup with options cpu,cpuacct.
Mounting cgroup (cgroup) on /sys/fs/cgroup/cpu,cpuacct (MS_NOSUID|MS_NODEV|MS_NOEXEC "cpu,cpuacct")...
Mounting cgroup to /sys/fs/cgroup/cpuset of type cgroup with options cpuset.
Mounting cgroup (cgroup) on /sys/fs/cgroup/cpuset (MS_NOSUID|MS_NODEV|MS_NOEXEC "cpuset")...
Mounting cgroup to /sys/fs/cgroup/memory of type cgroup with options memory.
Mounting cgroup (cgroup) on /sys/fs/cgroup/memory (MS_NOSUID|MS_NODEV|MS_NOEXEC "memory")...
Mounting cgroup to /sys/fs/cgroup/hugetlb of type cgroup with options hugetlb.
Mounting cgroup (cgroup) on /sys/fs/cgroup/hugetlb (MS_NOSUID|MS_NODEV|MS_NOEXEC "hugetlb")...
Mounting cgroup to /sys/fs/cgroup/freezer of type cgroup with options freezer.
Mounting cgroup (cgroup) on /sys/fs/cgroup/freezer (MS_NOSUID|MS_NODEV|MS_NOEXEC "freezer")...
Mounting cgroup to /sys/fs/cgroup/rdma of type cgroup with options rdma.
Mounting cgroup (cgroup) on /sys/fs/cgroup/rdma (MS_NOSUID|MS_NODEV|MS_NOEXEC "rdma")...
Mounting cgroup to /sys/fs/cgroup/pids of type cgroup with options pids.
Mounting cgroup (cgroup) on /sys/fs/cgroup/pids (MS_NOSUID|MS_NODEV|MS_NOEXEC "pids")...
No credentials passed from initrd.
Acquired 0 regular credentials, 0 untrusted credentials.

Welcome to Ubuntu 24.04 LTS!

Hostname was already set to <dimtest-ubu24-1>.
127.0.0.1 has already been added to loopback interface
::1 has already been added to loopback interface
Successfully brought loopback interface up
Setting '/proc/sys/net/unix/max_dgram_qlen' to '512'
Setting '/proc/sys/fs/file-max' to '9223372036854775807'
RLIMIT_MEMLOCK is already as high or higher than we need it, not bumping.
Found cgroup on /sys/fs/cgroup/systemd, legacy hierarchy
Using cgroup controller name=systemd. File system hierarchy is at /sys/fs/cgroup/systemd.
bpf-firewall: Not running with unified cgroup hierarchy, BPF firewalling is not supported.
Not running with unified cgroups, BPF device control is not supported.
Controller 'cpu' supported: no
Controller 'cpuacct' supported: no
Controller 'cpuset' supported: no
Controller 'io' supported: no
Controller 'blkio' supported: no
Controller 'memory' supported: yes
Controller 'devices' supported: no
Controller 'pids' supported: no
Controller 'bpf-firewall' supported: no
Controller 'bpf-devices' supported: no
Controller 'bpf-foreign' supported: no
Controller 'bpf-socket-bind' supported: no
Controller 'bpf-restrict-network-interfaces' supported: no
Set up TFD_TIMER_CANCEL_ON_SET timerfd.
Enabling (yes) showing of status (command line).
Successfully forked off '(sd-gens)' as PID 22.
PR_SET_MM_ARG_START failed: Operation not permitted
Failed to remount root directory as MS_SLAVE: Permission denied
(sd-gens) failed with exit status 1.
Failed to fork off sandboxing environment for executing generators: Protocol error
[!!!!!!] Failed to start up manager.
Exiting PID 1...

So the actual errors seem to be:

  • PR_SET_MM_ARG_START failed: Operation not permitted
  • Failed to remount root directory as MS_SLAVE: Permission denied

It looks a lot like Cannot start new distributions with systemd 255 · Issue #4402 · lxc/lxc · GitHub, so may another AppArmor issue?

And on the host machine, I see in syslog:

Apr 26 09:15:17 host02 kernel: [ 1585.079511] audit: type=1400 audit(1714122917.743:236): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=5824 comm="(sd-gens)" flags="rw, rslave"

So it looks quite clear what is going wrong: AppArmor denies the remount. No clue as to why …