I just had a Ubuntu 24.04 LTS machine hang after boot with a kernel crash. I just happened to reboot after a kernel update from 6.14 to 6.17. The machine only crashed after Incus started autoboot containers.
My workaround is continuing to use Kernel 6.14 for now.
My research:
It looks like using Podman inside of a container triggers a bug in AppArmor which leads to a null-pointer dereference. It has already been fixed upstream, but not back-ported to Ubuntu 24.04 LTS. Kernel NULL Pointer Dereference while starting rootless container with podman + crun (#568) · Issues · AppArmor / apparmor · GitLab
It seems the Proxmox-people already stumbled over the same issue 2 Months ago
BUG: kernel NULL pointer dereference | Proxmox Support Forum
Logs from my machine:
Feb 09 23:33:02 [...] kernel: BUG: kernel NULL pointer dereference, address: 0000000000000018
Feb 09 23:33:02 [...] kernel: #PF: supervisor read access in kernel mode
Feb 09 23:33:02 [...] kernel: #PF: error_code(0x0000) - not-present page
Feb 09 23:33:02 [...] kernel: PGD 0 P4D 0
Feb 09 23:33:02 [...] kernel: Oops: Oops: 0000 [#1] SMP NOPTI
Feb 09 23:33:02 [...] kernel: CPU: 4 UID: 1000000 PID: 32760 Comm: crun Tainted: P O 6.17.0-14-generic #14~24.04.1-Ubuntu PREEMPT(voluntary)
Feb 09 23:33:02 [...] kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
Feb 09 23:33:02 [...] kernel: Hardware name: [...]
Feb 09 23:33:02 [...] kernel: RIP: 0010:aa_file_perm+0xb9/0x3b0
Feb 09 23:33:02 [...] kernel: Code: ff 45 31 c0 45 31 c9 e9 b0 53 68 ff 49 8b 47 20 49 8b 4f 18 0f b7 00 66 25 00 f0 66 3d 00 c0 75 18 41 f7 c4 46 00 10 00 75 0f <48> 8b 41 18 66 83 78 10 01 0f 84 44 01 00 00 f7 d2 44 21 e2 89 55
Feb 09 23:33:02 [...] kernel: RSP: 0018:ffffd1e2501ab770 EFLAGS: 00010246
Feb 09 23:33:02 [...] kernel: RAX: 000000000000c000 RBX: ffff8cf837c94580 RCX: 0000000000000000
Feb 09 23:33:02 [...] kernel: RDX: 0000000000000000 RSI: ffff8cf806744cc0 RDI: ffffffffa97c6288
Feb 09 23:33:02 [...] kernel: RBP: ffffd1e2501ab7c8 R08: 0000000000000000 R09: 0000000000000000
Feb 09 23:33:02 [...] kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Feb 09 23:33:02 [...] kernel: R13: ffff8cf83088bd58 R14: ffff8cf837c94580 R15: ffff8cf87a782240
Feb 09 23:33:02 [...] kernel: FS: 0000703037ddf840(0000) GS:ffff8d0733c63000(0000) knlGS:0000000000000000
Feb 09 23:33:02 [...] kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 09 23:33:02 [...] kernel: CR2: 0000000000000018 CR3: 0000000106a0d000 CR4: 0000000000f50ef0
Feb 09 23:33:02 [...] kernel: PKRU: 55555554
Feb 09 23:33:02 [...] kernel: Call Trace:
Feb 09 23:33:02 [...] kernel: <TASK>
Feb 09 23:33:02 [...] kernel: common_file_perm+0x6c/0x1a0
Feb 09 23:33:02 [...] kernel: apparmor_file_receive+0x56/0x70
Feb 09 23:33:02 [...] kernel: security_file_receive+0x31/0x50
Feb 09 23:33:02 [...] kernel: receive_fd+0x1d/0xf0
Feb 09 23:33:02 [...] kernel: scm_detach_fds+0xbf/0x200
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? simple_copy_to_iter+0x3a/0x60
Feb 09 23:33:02 [...] kernel: __scm_recv_common.isra.0+0x68/0x180
Feb 09 23:33:02 [...] kernel: scm_recv_unix+0x32/0x140
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? unix_destroy_fpl+0x49/0xb0
Feb 09 23:33:02 [...] kernel: __unix_dgram_recvmsg+0x2b6/0x450
Feb 09 23:33:02 [...] kernel: unix_seqpacket_recvmsg+0x43/0x70
Feb 09 23:33:02 [...] kernel: sock_recvmsg+0xe1/0xf0
Feb 09 23:33:02 [...] kernel: ____sys_recvmsg+0xa2/0x230
Feb 09 23:33:02 [...] kernel: ? consume_skb+0x52/0xf0
Feb 09 23:33:02 [...] kernel: ___sys_recvmsg+0x90/0xf0
Feb 09 23:33:02 [...] kernel: __sys_recvmsg+0x89/0x100
Feb 09 23:33:02 [...] kernel: __x64_sys_recvmsg+0x1d/0x30
Feb 09 23:33:02 [...] kernel: x64_sys_call+0x697/0x2680
Feb 09 23:33:02 [...] kernel: do_syscall_64+0x80/0xa30
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? __sys_recvmsg+0x89/0x100
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? arch_exit_to_user_mode_prepare.isra.0+0xd/0xe0
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? do_syscall_64+0xb6/0xa30
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? count_memcg_events+0xf0/0x1e0
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? handle_mm_fault+0x237/0x370
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? do_user_addr_fault+0x1d2/0x8d0
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? arch_exit_to_user_mode_prepare.isra.0+0xd/0x100
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? irqentry_exit_to_user_mode+0x2d/0x1d0
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? irqentry_exit+0x43/0x50
Feb 09 23:33:02 [...] kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Feb 09 23:33:02 [...] kernel: ? exc_page_fault+0x90/0x1b0
Feb 09 23:33:02 [...] kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Feb 09 23:33:02 [...] kernel: RIP: 0033:0x703037f79ba6
Feb 09 23:33:02 [...] kernel: Code: 00 00 48 8b 15 53 12 17 00 64 89 02 48 c7 c2 ff ff ff ff 48 8b 5d f8 c9 48 89 d0 c3 0f 1f 84 00 00 00 00 00 48 8b 45 10 0f 05 <48> 63 d0 3d 00 f0 ff ff 77 10 48 8b 5d f8 48 89 d0 c9 c3 0f 1f 80
Feb 09 23:33:02 [...] kernel: RSP: 002b:00007ffde40d1c00 EFLAGS: 00000202 ORIG_RAX: 000000000000002f
Feb 09 23:33:02 [...] kernel: RAX: ffffffffffffffda RBX: 0000703037ddf840 RCX: 0000703037f79ba6
Feb 09 23:33:02 [...] kernel: RDX: 0000000000000000 RSI: 00007ffde40d1c40 RDI: 000000000000000a
Feb 09 23:33:02 [...] kernel: RBP: 00007ffde40d1c10 R08: 0000000000000000 R09: 0000000000000000
Feb 09 23:33:02 [...] kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffde40d1c40
Feb 09 23:33:02 [...] kernel: R13: 00007ffde40d2240 R14: 0000000000000005 R15: 0000000000000009
Feb 09 23:33:02 [...] kernel: </TASK>
Feb 09 23:33:02 [...] kernel: Modules linked in: xt_addrtype xt_nat xt_mark xt_conntrack xt_tcpudp xt_comment xt_MASQUERADE nft_compat vhost_net tap veth br_netfilter nft_nat nft_masq nft_chain_nat bridge nf_tables vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 psample cfg80211 nvme_fabrics binfmt_misc nls_iso8859_1 amdgpu amd_atl intel_rapl_msr snd_hda_codec_alc662 intel_rapl_common snd_hda_codec_realtek_lib snd_hda_codec_generic snd_hda_codec_atihdmi snd_hda_codec_hdmi snd_hda_intel edac_mce_amd amdxcp drm_panel_backlight_quirks snd_hda_codec gpu_sched drm_buddy drm_ttm_helper snd_hda_core kvm_amd ttm snd_intel_dspcfg snd_intel_sdw_acpi drm_exec drm_suballoc_helper snd_hwdep drm_display_helper kvm snd_pcm irqbypass snd_timer cec gigabyte_wmi ccp rc_core i2c_piix4 rapl wmi_bmof snd i2c_algo_bit soundcore i2c_smbus k10temp gpio_amdpt input_leds joydev mac_hid sch_fq_codel efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4
Feb 09 23:33:02 [...] kernel: dm_crypt zfs(PO) 8021q garp spl(O) mrp stp llc hid_generic usbhid nvme uas hid usb_storage nvme_core r8169 polyval_clmulni ghash_clmulni_intel ahci nvme_keyring realtek libahci nvme_auth video wmi aesni_intel
Feb 09 23:33:02 [...] kernel: CR2: 0000000000000018
Feb 09 23:33:02 [...] kernel: ---[ end trace 0000000000000000 ]---
Feb 09 23:33:02 [...] kernel: RIP: 0010:aa_file_perm+0xb9/0x3b0
Feb 09 23:33:02 [...] kernel: Code: ff 45 31 c0 45 31 c9 e9 b0 53 68 ff 49 8b 47 20 49 8b 4f 18 0f b7 00 66 25 00 f0 66 3d 00 c0 75 18 41 f7 c4 46 00 10 00 75 0f <48> 8b 41 18 66 83 78 10 01 0f 84 44 01 00 00 f7 d2 44 21 e2 89 55
Feb 09 23:33:02 [...] kernel: RSP: 0018:ffffd1e2501ab770 EFLAGS: 00010246
Feb 09 23:33:02 [...] kernel: RAX: 000000000000c000 RBX: ffff8cf837c94580 RCX: 0000000000000000
Feb 09 23:33:02 [...] kernel: RDX: 0000000000000000 RSI: ffff8cf806744cc0 RDI: ffffffffa97c6288
Feb 09 23:33:02 [...] kernel: RBP: ffffd1e2501ab7c8 R08: 0000000000000000 R09: 0000000000000000
Feb 09 23:33:02 [...] kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Feb 09 23:33:02 [...] kernel: R13: ffff8cf83088bd58 R14: ffff8cf837c94580 R15: ffff8cf87a782240
Feb 09 23:33:02 [...] kernel: FS: 0000703037ddf840(0000) GS:ffff8d0733c63000(0000) knlGS:0000000000000000
Feb 09 23:33:02 [...] kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 09 23:33:02 [...] kernel: CR2: 0000000000000018 CR3: 0000000106a0d000 CR4: 0000000000f50ef0
Feb 09 23:33:02 [...] kernel: PKRU: 55555554
Feb 09 23:33:02 [...] kernel: note: crun[32760] exited with irqs disabled