Docker issues after 6.15 Upgrade

rdub · August 13, 2025, 6:17pm

Can anyone provide assistance with the following issue:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "mqueue" to rootfs at "/dev/mqueue": change mount propagation through procfd: resolving path inside rootfs failed: lstat /var/lib/docker/overlay2/8c209f91d6403d5cb3b079487722a4087de4739c743dcaaf85c82c7a73665022/merged//dev/mqueue: permission denied: unknown

Performed update of Incus Server to 6.15 and now all docker containers running inside LXC instance are giving the same /dev/mqueue: permission denied: unknown error.

After upgrade if containers are restarted they WILL NOT REBOOT AGAIN AS EXPECTED !!!

Is there a working group or issue created for the problem being caused by the increased security functions in 6.15?

Is there a way to rollback to 6.14 until this is addressed?

stgraber · August 13, 2025, 8:17pm

Hmm, there was no change to the LXC driver between 6.14 and 6.15, so this failure is almost certainly unrelated to the update from 6.14 to 6.15.

rdub · August 13, 2025, 8:19pm

What logs would you need in order to help with identifying the root cause?

stgraber · August 13, 2025, 8:19pm

We also have daily tests validating that Docker generally works correctly, so any change on our end that would break all containers would generally be picked up way before release time.

Chances are something else happened here to trigger this situation.

Can you show the full incus config show --expanded NAME for an affected container as well as what OS is running inside the Incus container and what version of Docker is installed in there, also what storage driver is used on the Incus side and what kernel version is running?

rdub · August 13, 2025, 8:21pm

 incus config show --expanded lxcon
architecture: x86_64
config:
  boot.autostart: "true"
  boot.autostart.delay: "2"
  boot.autostart.priority: "60"
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (20250620_07:42)
  image.os: Ubuntu
  image.release: noble
  image.requirements.cgroup: v2
  image.serial: "20250620_07:42"
  image.type: squashfs
  image.variant: default
  limits.memory: 8%
  limits.memory.enforce: soft
  security.nesting: "true"
  volatile.base_image: 1d94fe0153f33d8035e816a702ce92cc1f3a1f4dee999276579993d2212ffbfa
  volatile.cloud-init.instance-id: a8fb5a99-684b-4c21-b744-08702e1a4ba0
  volatile.eth0.hwaddr: 00:16:3e:78:8d:26
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: 77fc0d90-461e-4a65-8176-1576d1dfee7e
  volatile.uuid.generation: 77fc0d90-461e-4a65-8176-1576d1dfee7e
devices:
  eth0:
    nictype: bridged
    parent: bridge0
    type: nic
  root:
    path: /
    pool: b-0n1
    type: disk
ephemeral: false
profiles:
- default
- bridgeprofile
stateful: false
description: ""

rdub · August 13, 2025, 8:22pm

The container starts as expected upon server reboot but the docker instances running inside of the container are failing with the initially attached error from this post.

stgraber · August 13, 2025, 8:23pm

Still need Docker version in container, storage driver used on the Incus front and kernel version.

rdub · August 13, 2025, 8:27pm

cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
rdub@lxcon:~$
rdub@lxcon:~$
rdub@lxcon:~$
rdub@lxcon:~$
rdub@lxcon:~$ docker version
Client: Docker Engine - Community
 Version:           28.2.2
 API version:       1.50
 Go version:        go1.24.3
 Git commit:        e6534b4
 Built:             Fri May 30 12:07:27 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.2.2
  API version:      1.50 (minimum version 1.24)
  Go version:       go1.24.3
  Git commit:       45873be
  Built:            Fri May 30 12:07:27 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
rdub@lxcon:~$
rdub@lxcon:~$
rdub@lxcon:~$
rdub@lxcon:~$
rdub@lxcon:~$ docker info | grep "Storage Driver"
 Storage Driver: overlay2

stgraber · August 13, 2025, 8:29pm

Still no kernel version, can you show uname -a

rdub · August 13, 2025, 8:29pm

uname -r
6.8.0-71-generic

rdub · August 13, 2025, 8:29pm

Linux inc01 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

rdub · August 13, 2025, 8:31pm

Docker compose version inside LXC

docker compose version
Docker Compose version v2.36.2

stgraber · August 13, 2025, 8:33pm

Still missing the Incus storage driver in the info I requested above.

Can you show incus storage list

rdub · August 13, 2025, 8:33pm

incus storage list
+-------+--------+------------------+---------+---------+
| NAME  | DRIVER |   DESCRIPTION    | USED BY |  STATE  |
+-------+--------+------------------+---------+---------+
| b-0n1 | btrfs  | nvme0n1          | 28      | CREATED |
+-------+--------+------------------+---------+---------+
| b-1n1 | btrfs  | nvme1n1          | 21      | CREATED |
+-------+--------+------------------+---------+---------+
| baby  | dir    | 10TB Nas W2 Baby | 9       | CREATED |
+-------+--------+------------------+---------+---------+
| hm    | dir    | 3tb SSD          | 6       | CREATED |
+-------+--------+------------------+---------+---------+

rdub · August 13, 2025, 8:42pm

Any additional information needed?

stgraber · August 13, 2025, 8:47pm

No, got it reproduced here, looks like a regression in AppArmor’s handling of abi4.0 versus normal rules, at least for the version that’s in Ubuntu 24.04…

I’m testing to see if we can get something where we allow the 4.0 feature we need without changing the profile abi so we can get the best of both worlds since getting something fixed in AppArmor and pushed to the stable distros would be rather slow.

stgraber · August 13, 2025, 8:49pm

stgraber · August 13, 2025, 8:51pm

apt install incus=1:6.15-ubuntu24.04-202508091853 incus-base=1:6.15-ubuntu24.04-202508091853 incus-client=1:6.15-ubuntu24.04-202508091853

Can be used as a temporary workaround to roll back to an Incus 6.15 build that didn’t have the updated AppArmor profile.

rdub · August 13, 2025, 8:53pm

Ok applying and rebooting

rdub · August 13, 2025, 8:56pm

That worked to get the containers restarted. Seems a manual restart of the dockers is needed as well.