Containers don´t start after lxd-to-incus

Incus
,
1

Hi!

I migrated, according to the lxd-to-incus script, sucessfully to incus. When executing incus list I see all my container and VMs. But none of them will start. I don’t get any error messages when executing incus start u1. Also --show-log doesn’t show anything. The logs below are from /var/log/incus/u1.

$ cat u1/console.log 
Failed to drop FS_IMMUTABLE_FL from /sys/firmware/efi/efivars/LoaderRandomSeed-xxx, ignoring: Permission denied
Failed to reduce access mode of /sys/firmware/efi/efivars/LoaderRandomSeed-xxx, ignoring: Permission denied
Failed to drop FS_IMMUTABLE_FL from /sys/firmware/efi/efivars/LoaderSystemToken-xxx, ignoring: Permission denied
Failed to reduce access mode of /sys/firmware/efi/efivars/LoaderSystemToken-xxx, ignoring: Permission denied
Failed to read LoaderRandomSeed EFI variable, ignoring: Permission denied

Welcome to Debian GNU/Linux 12 (bookworm)!

Failed to allocate notification socket: Permission denied
[!!!!!!] Failed to start up manager.
Exiting PID 1...

$ cat u1/lxc.log
lxc 20240224220931.810 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20240224220931.810 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"

I found this post containers don’t start after migration and checked my /opt. I can cd into /opt/incus/lib/lxc/rootfs as a normal user.

Any help would be appreciated!
Thank you

2

Hmm, that’s an unusual error, can you share:

  • What distribution this is running on
  • Output of uname -a
  • Output of dmesg | tail -n 50 right after attempting to start a container
3

Sure can do.

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
$ uname -a
Linux vh 5.15.126-1-pve #1 SMP PVE 5.15.126-1 (2023-10-03T17:24Z) x86_64 GNU/Linux
$ dmesg
[35195.902029] vmbr0: port 2(veth132f931b) entered blocking state
[35195.902036] vmbr0: port 2(veth132f931b) entered disabled state
[35195.902075] device veth132f931b entered promiscuous mode
[35195.953412] audit: type=1400 audit(1708849699.241:61): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus-cmk_</var/lib/incus>" pid=2034074 comm="apparmor_parser"
[35196.100061] physeqJ3Ov: renamed from vethee4b904f
[35196.120683] eth0: renamed from physeqJ3Ov
[35196.156748] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[35196.156852] vmbr0: port 2(veth132f931b) entered blocking state
[35196.156866] vmbr0: port 2(veth132f931b) entered forwarding state
[35196.384204] audit: type=1400 audit(1708849699.673:62): apparmor="DENIED" operation="create" info="failed type and protocol match" error=-13 profile="incus-cmk_</var/lib/incus>" pid=2034212 comm="systemd" family="unix" sock_type="dgram" protocol=0 requested_mask="create" denied_mask="create" addr=none
[35196.444232] physeqJ3Ov: renamed from eth0
[35196.480163] vmbr0: port 2(veth132f931b) entered disabled state
[35196.489884] vethee4b904f: renamed from physeqJ3Ov
[35196.550168] device veth132f931b left promiscuous mode
[35196.550191] vmbr0: port 2(veth132f931b) entered disabled state
[35197.353330] audit: type=1400 audit(1708849700.641:63): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-cmk_</var/lib/incus>" pid=2034389 comm="apparmor_parser"

The installation was made with a Proxmox installation media and still uses the kernel from the project. Proxmox supports Debian with a zfs root, which Debian doesn`t. Directly after installation everything Proxmox related was removed, except the kernel plus necessary packages, and instead lxd via snap installed. It was running fine with lxd for multiple years.

4

Firewall rules? I run Debian 12 and I use UFW. Here’s how I run my incus firewall on the OS:

sudo ufw allow in on incusbr0
sudo ufw route allow in on incusbr0
sudo ufw route allow out on incusbr0

If you’re still using lxdbr0, this is probably not a solution, but it’s easy to check (should still have the equivalent for lxdbr0 though)?

GL

Andrew

5

ufw is inactive, iptables and nft show no configured rules. I used vmbr0 and vmbr1 in lxd. I assume lxd-to-incus converted this to incus configuration.

6

I removed incus completely, also all remaining zfs volumes remaining from lxd-to-incus conversion. Then executed incus admin init and started with a blank incus installation. Still the same problem:

[...]
Failed to allocate notification socket: Permission denied
[!!!!!!] Failed to start up manager.

Any idea what else I can try?

7

The root cause is apparmor and lxc. The following messages appeared in dmesg:

audit: apparmor="DENIED" operation="create" info="failed type and protocol match" error=-13 profile="incus-third_</var/lib/incus>" pid=123 comm="udevadm" family="unix" sock_type="seqpacket" protocol=0 requested_mask="create" denied_mask="create" addr=none
audit: apparmor="DENIED" operation="create" info="failed type and protocol match" error=-13 profile="incus-third_</var/lib/incus>" pid=456 comm="systemd" family="unix" sock_type="dgram" protocol=0 requested_mask="create" denied_mask="create" addr=none

I found this LXD/lxc issue: https://github.com/lxc/lxc/issues/4150 - and applying the fix, the container is now running:

$ incus config set first raw.lxc "lxc.apparmor.profile=unconfined"
$ incus start first
$ incus list
+-------+---------+-----------------------+---------------------------------------------+-----------+-----------+
| NAME  |  STATE  |    IPV4          |                    IPV6          |   TYPE          | SNAPSHOTS |
+-------+---------+-----------------------+---------------------------------------------+-----------+-----------+
| first | RUNNING |  1.2.3.4 (eth0) |  xxxxxxxxxxxxxxxxxxx | CONTAINER |            0         |
+-------+---------+-----------------------+---------------------------------------------+-----------+-----------+

Since this was working with LXD can anyone answer what I’m missing in incus and how I can fix it?

Thank you!