Something i want to explore when i have containers ready

there is something i want to explore when i have containers ready to go. the idea is to run qemu in user-mode, where it does the emulation of other architectures on a per-process basis, including support for threads (mapped to host threads). the full idea is to create a container with the image of another architecture and run init in the container under qemu user-mode. each child forked and/or execed from there is running under qemu user-mode. thus, the entire container can be running in isolated emulation. if that works well, the many needs to run virtual machines for emulated architectures to do things like compiling, building, linking, etc., can operate more like a virtual machines but with the performance boost of not having two operating system layers.

has anyone here done anything like this?

Some templates (lxc-ubuntu and lxc-debian I believe) may still have code to do this.

The user experience was pretty terrible though as qemu, at least back then was extremely fragile when dealing with threads and due to how binfmt works, anything using ptrace() would break, same was true for anything using the netlink API.

The result was that to get a container to boot properly, a large chunk of packages had to be installed using the native architecture of the system, leading to a weird container that was part native arch, part foreign. It was good enough for some users, but is not something we’d ever really recommend.

That was quite a few years ago, so maybe qemu-user-static has now fixed those issues and all you need is the right qemu-user-static binary to be dumped into the container’s rootfs, but I somehow doubt it.

i’m thinking more along the lines of a hack to qemu user mode to have enter the container after it is running.