Systemd-binfmt.service is masked

Hello, good day, I am trying to restart the systemd-binfmt.service service but it shows the following error:

Failed to restart systemd-binfmt.service: Unit systemd-binfmt.service is masked.

I have tried with 2 incus images with debian/12 and with ubuntu/22.04 and I have the same error, I have tried all the options on the internet to unmask the service but I have not had success, I have also tried with the option -c security.nesting=true and I still cannot solve the problem, could someone help me, thank you, greetings

System info:
Linux ubuntu-server-incus 6.5.0-1027 Ubuntu 22.04 aarch64

config show:

config:
  image.architecture: arm64
  image.description: Debian bullseye arm64 (20240906_05:24)
  image.os: Debian
  image.release: bullseye
  image.serial: "20240906_05:24"
  image.type: squashfs
  image.variant: default
  security.nesting: "true"
  volatile.base_image: 799d582bbf3cdec060f34673dc18a1a2751ecfcb51091d1a44cdb655370f1dfc
  volatile.cloud-init.instance-id: a785cf64-cb9e-400b-9f04-e2944cbcc613
  volatile.eth0.host_name: veth07967c57
  volatile.eth0.hwaddr: 00:16:3e:0b:82:54
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 7e6cd3ce-05ee-404d-8c2f-257f16fdf226
  volatile.uuid.generation: 7e6cd3ce-05ee-404d-8c2f-257f16fdf226
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

Welcome!

Have a look at this discussion, Binfmt_misc: permission denied in unprivileged container

It says that the mounting of binfmt_misc inside unprivileged containers is a new kernel feature in Linux 6.7 and newer. Therefore, your version of the Linux kernel in Ubuntu 22.04 needs to be updated.

You can use the Hardware Enablement (HWE) in Ubuntu 22.04 to update the Linux kernel, and by doing so you will get Linux 6.8, Ubuntu kernel lifecycle and enablement stack | Ubuntu

Hello Simos and thank you for responding.

I have made the suggested recommendations but I still have the problem, the service is still masked and that state does not change, what could be happening?

Indeed, there’s an issue here, and currently it appears to be a systemd issue. systemd is complaining that the service is masked, specifically masked-runtime. That means that the mask is supposed to go away upon a restart. But it does not.

There’s also a mention on zzz-lxc-service.conf

$ incus launch images:ubuntu/24.04/cloud binfmt
Launching binfmt
$ incus shell binfmt
root@binfmt:~# systemctl status systemd-binfmt.service 
Warning: The unit file, source configuration file or drop-ins of systemd-binfmt.service changed on disk. Run 'systemctl daemon-reload' to reload units.
â—‹ systemd-binfmt.service
     Loaded: masked (Reason: Unit systemd-binfmt.service is masked.)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead)
root@binfmt:~# systemctl daemon-reload
root@binfmt:~# systemctl status systemd-binfmt.service 
Warning: The unit file, source configuration file or drop-ins of systemd-binfmt.service changed on disk. Run 'systemctl daemon-reload' to reload units.
â—‹ systemd-binfmt.service
     Loaded: masked (Reason: Unit systemd-binfmt.service is masked.)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead)
root@binfmt:~# systemctl list-unit-files | grep binfmt
proc-sys-fs-binfmt_misc.automount            static          -
proc-sys-fs-binfmt_misc.mount                disabled        disabled
systemd-binfmt.service                       masked-runtime  enabled
root@binfmt:~# systemctl enable proc-sys-fs-binfmt_misc.mount
Created symlink /etc/systemd/system/sysinit.target.wants/proc-sys-fs-binfmt_misc.mount → /usr/lib/systemd/system/proc-sys-fs-binfmt_misc.mount.
root@binfmt:~# systemctl status systemd-binfmt.service 
Warning: The unit file, source configuration file or drop-ins of systemd-binfmt.service changed on disk. Run 'systemctl daemon-reload' to reload units.
â—‹ systemd-binfmt.service
     Loaded: masked (Reason: Unit systemd-binfmt.service is masked.)
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: inactive (dead)
root@binfmt:~# systemctl daemon-reload
root@binfmt:~# systemctl start systemd-binfmt.service 
Failed to start systemd-binfmt.service: Unit systemd-binfmt.service is masked.
root@binfmt:~# systemctl list-unit-files | grep binfmt
proc-sys-fs-binfmt_misc.automount            static          -
proc-sys-fs-binfmt_misc.mount                enabled         disabled
systemd-binfmt.service                       masked-runtime  enabled
root@binfmt:~#

The systemd service for binfmt does not want to start up, and it requires some additional searching to figure out how to clear this hurdle. It’s not essential for this systemd service to start, because this can be setup even manually.

Here is me using binfmt successfully in a container in an Incus container. Note that the issue was with a missing mount.

$ incus launch images:ubuntu/24.04/cloud binfmt
Launching binfmt
$ incus shell binfmt
root@binfmt:~# cat > helloworld.py
print("Hello, world!")
root@binfmt:~# chmod +x helloworld.py 
root@binfmt:~# ./helloworld.py 
./helloworld.py: line 1: syntax error near unexpected token `"Hello, world!"'
./helloworld.py: line 1: `print("Hello, world!")'
root@binfmt:~# echo ":python3:E::py::/usr/bin/python3:" | sudo tee /proc/sys/fs/binfmt_misc/register
tee: /proc/sys/fs/binfmt_misc/register: No such file or directory
:python3:E::py::/usr/bin/python3:
root@binfmt:~# mount | grep binfmt
default/containers/binfmt on / type zfs (rw,relatime,idmapped,xattr,posixacl,casesensitive)
root@binfmt:~# systemctl start proc-sys-fs-binfmt_misc.mount
root@binfmt:~# mount | grep binfmt
default/containers/binfmt on / type zfs (rw,relatime,idmapped,xattr,posixacl,casesensitive)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
root@binfmt:~# echo ":python3:E::py::/usr/bin/python3:" | sudo tee /proc/sys/fs/binfmt_misc/register
:python3:E::py::/usr/bin/python3:
root@binfmt:~# cat helloworld.py 
print("Hello, world!")
root@binfmt:~# ./helloworld.py 
Hello, world!
root@binfmt:~# 

How can we do better?

  1. First you need to systemctl enable proc-sys-fs-binfmt_misc.mount so that the mounting occurs automatically.
  2. My understanding is that systemd-binfmt.service will read /etc/binfmt.d/ and register automatically for you your extensions. This does not work, so you have to perform it yourself for now.

Wow, everything is a little cumbersome just to make a service work, don’t you think? I really appreciate your attention and experience in these issues. I’m going to do some tests and try to use other less complicated solutions. Thank you again and greetings. Greetings

Thankfully, it’s not that complex. There’s an easy workaround that you can try straight away and do your work for now.

The part that I am missing is how to debug the masked-runtime state. Most likely it’s something easy that I am missing because I cannot get systemd to give me the details. Once that issue is resolved, then everything works easily.

Perhaps @Greelan can help out here with that masked-runtime systemd state for the binfmt_misc service.

1 Like

What did I miss?

You managed to use the binfmt_misc service earlier.

When I try to replicate now, with Incus 6.5.0, on both Debian and Ubuntu images, I get a message that the service systemd-binfmt.service is masked (masked-runtime). Did you ever get that message?

No, still running fine here on Incus 6.0.0 and Ubuntu 24.04 (host and container)

Also fine after switching to Incus 6.5

1 Like

uses system aarch64 ?

Here is me trying binfmt_misc on the host to run a Python program. Normally, a shell script requires #!/bin/sh or something similar in order for the shell to run it properly. In this case though, the Linux kernel recognizes the .py extension and automatically runs python3 on the file.

$ cat helloworld.py 
print('Hello, world!')
$ chmod +x helloworld.py 
$ ./helloworld.py 
./helloworld.py: line 1: syntax error near unexpected token `'Hello, world!''
./helloworld.py: line 1: `print('Hello, world!')'
$ echo ":python3:E::py::/usr/bin/python3:" | sudo tee /proc/sys/fs/binfmt_misc/register
:python3:E::py::/usr/bin/python3:
$ ./helloworld.py 
Hello, world!
$ 

Now, this is me, without rebooting, running this in a container. It works as well, because the host was configured earlier.

$ incus launch images:ubuntu/24.04/cloud binfmt
Launching binfmt
$ incus shell binfmt
root@binfmt:~# cat > helloworld.py
print("Hello, world!")
root@binfmt:~# chmod +x helloworld.py 
root@binfmt:~# ./helloworld.py 
Hello, world!
root@binfmt:~# logout
$

Let’s disable python3 on the host.

$ ./helloworld.py 
Hello, world!
$ echo -1 | sudo tee /proc/sys/fs/binfmt_misc/python3
-1
$ ./helloworld.py 
./helloworld.py: line 1: syntax error near unexpected token `'Hello, world!''
./helloworld.py: line 1: `print('Hello, world!')'
$ 

Let’s start over. What we see now, is that even when we install the package binfmt-support, the runtime does not perform the mount. Perhaps the mount is done be some other service? Does binfmt-support only enables any configuration that is found in /etc/binfmt.d/? Which package performs the mount?

$ incus launch images:ubuntu/24.04/cloud binfmt
Launching binfmt
$ incus shell binfmt
root@binfmt:~# apt install -y binfmt-support
...
Setting up binfmt-support (2.2.2-7) ...
Created symlink /etc/systemd/system/multi-user.target.wants/binfmt-support.servi
ce → /usr/lib/systemd/system/binfmt-support.service.
Processing triggers for libc-bin (2.39-0ubuntu8.3) ...
root@binfmt:~# systemctl status binfmt-support
â—Ź binfmt-support.service - Enable support for additional executable binary form>
     Loaded: loaded (/usr/lib/systemd/system/binfmt-support.service; enabled; p>
    Drop-In: /run/systemd/system/service.d
             └─zzz-lxc-service.conf
     Active: active (exited) since Fri 2024-09-13 13:54:55 UTC; 20s ago
       Docs: man:update-binfmts(8)
    Process: 760 ExecStart=/usr/sbin/update-binfmts --enable (code=exited, stat>
   Main PID: 760 (code=exited, status=0/SUCCESS)
        CPU: 12ms

Sep 13 13:54:55 binfmt systemd[1]: Starting binfmt-support.service - Enable sup>
Sep 13 13:54:55 binfmt systemd[1]: Finished binfmt-support.service - Enable sup>
root@binfmt:~# echo ":python3:E::py::/usr/bin/python3:" | sudo tee /proc/sys/fs/binfmt_misc/register
tee: /proc/sys/fs/binfmt_misc/register: No such file or directory
:python3:E::py::/usr/bin/python3:
root@binfmt:~# mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
root@binfmt:~# echo ":python3:E::py::/usr/bin/python3:" | sudo tee /proc/sys/fs/binfmt_misc/register
:python3:E::py::/usr/bin/python3:
root@binfmt:~# cat > helloworld.py
print("Hello, world!")
root@binfmt:~# chmod +x helloworld.py 
root@binfmt:~# ./helloworld.py 
Hello, world!
root@binfmt:~# logout
$ 

Yes, that’s right. That’s why I needed it, so I could run box64/box86 emulation in the container

I found this issue on a new Debian 12 container. After much work trying to troubleshoot what was causing it to be masked, I could only come up with the workaround of:

cp /lib/systemd/system/systemd-binfmt.service /etc/systemd/system/systemd-binfmt.service
systemctl daemon-reload
systemctl start systemd-binfmt.service

Weird

1 Like

I discarded this topic since I started it, instead I use the following emulator that works wonderfully, greetings