Just seems like QEMU ontop of io_uring doesn’t like being on anything other than real physical disks.
We’ve tried to account for that by detecting use of loop backed storage pools and disk images, but we cannot detect stacked storage layers/VM layers.
That is possible. I don’t have a physical server available right now to test it without LUKS, because they are in use without VM’s 
I’ll leave this with @stgraber on how we proceed, but given that io_uring or QEMU + io_uring support is rather haphazard at the moment I suspect we’ll need one or more settings to control io_uring at various levels, with my preference being to defaulting to off until such time that it matures.
1 Like
There is no workaround for this? I can’t start production VM’s right now.
There is no manual option for disabling it currently I’m afraid.
Using a loop backed storage pool will always disable it.
Have created issue to track:
1 Like
I realise that its not much use for the problem right now (as you can’t switch the latest/stable channel to the LTS channels), but if you need a more convervative release channel for LXD for your production workloads then 4.0/stable may suit you better as the feature release latest/stable tends to move quite fast and gets all of the new features.
Please see
Yeah, I know. These kind of bugs are not very common. So I accept it for now. Hopefully it will be possible to fix this today. Otherwise I have to migrate. Thanks Thomas!
I tried the Focal HWE kernel and that seems to work with LVM and ZFS atop LVM.
sudo apt install --install-recommends linux-generic-hwe-20.04
Then reboot.
It works in Hyper-V with ZFS. I will test it now on my bare-metal server with LUKS.
1 Like
It does not accept my LUKS password anymore after the upgrade to the HWE enabled kernel and I dont know why. I will reinstall the machine and put everything back. Unfortunately don’t have a header backup.
Booting back to the non-HWE kernel doesn’t work?
Yep, reinstalled the node with 20.04 HWE kernel and now copying the VM’s back to test.
Error after the HWE kernel installation: “No key available with this passphrase.” on the ZFS partitions. Unfortunately it also doesn’t work with the old kernel and my 2 passphrases. So be careful with the kernel upgrade in combination with LUKS.
I did not expect this because the OS installation itself isn’t encrypted. Anyway… We have backups for that. Everything is up & running and the VMs are working with the HWE kernel.
@tomp @stgraber
Hello gentlemen,
today I got several prod servers hung after reboot, came to conclusion that 4.24 snap lxd package is the cause. Downgrading to 4.23 from rescue helped even though it was not straightforward. Then I came to this issue. I see the fix is merged into the master. I am holding my breath and trying not touch/reboot other hundreds of the servers… could you please advise when new snap package will come with the fix? I want to decide if I should downgrade massively or just wait…
Thanks in advance.
It’s building right now, I expect we’ll start an accelerated roll-out in the next couple of hours.
1 Like
io_uring works with HWE and LUKS in 20.04. I couldn’t find any problems.
I will use HWE from now on (until 22.04 is available)
1 Like
Today I saw that my / partition was full. 
with this spam in syslog / kern.log
Mar 26 13:29:48 server kernel: [ 217.602554] <TASK>
Mar 26 13:29:48 server kernel: [ 217.602554] kvm_arch_vcpu_ioctl_run+0xe6/0x5d0 [kvm]
Mar 26 13:29:48 server kernel: [ 217.602555] vcpu_enter_guest+0x354/0x11e0 [kvm]
Mar 26 13:29:48 server kernel: [ 217.602589] kvm_vcpu_ioctl+0x247/0x5f0 [kvm]
Mar 26 13:29:48 server kernel: [ 217.602594] ? kvm_skip_emulated_instruction+0x1f/0x40 [kvm]
Mar 26 13:29:48 server kernel: [ 217.602616] ? __fget_light+0xce/0xf0
Mar 26 13:29:48 server kernel: [ 217.602619] __x64_sys_ioctl+0x91/0xc0
Mar 26 13:29:48 server kernel: [ 217.602622] do_syscall_64+0x61/0xb0
Mar 26 13:29:48 server kernel: [ 217.602624] ? do_syscall_64+0x6e/0xb0
Mar 26 13:29:48 server kernel: [ 217.602625] ? do_syscall_64+0x6e/0xb0
Mar 26 13:29:48 server kernel: [ 217.602627] ? asm_sysvec_irq_work+0xa/0x20
Mar 26 13:29:48 server kernel: [ 217.602630] entry_SYSCALL_64_after_hwframe+0x44/0xae
Mar 26 13:29:48 server kernel: [ 217.602633] RIP: 0033:0x7ff6420a83db
Mar 26 13:29:48 server kernel: [ 217.602634] Code: 0f 1e fa 48 8b 05 b5 7a 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 7a 0d 00 f7 d8 64 89 01 48
Mar 26 13:29:48 server kernel: [ 217.602636] RSP: 002b:00007ff637071248 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Mar 26 13:29:48 server kernel: [ 217.602638] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007ff6420a83db
Mar 26 13:29:48 server kernel: [ 217.602640] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000019
Mar 26 13:29:48 server kernel: [ 217.602641] RBP: 000055e0cf5e19e0 R08: 000055e0cd790e68 R09: 000055e0cd447370
Mar 26 13:29:48 server kernel: [ 217.602642] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
Mar 26 13:29:48 server kernel: [ 217.602643] R13: 000055e0cdddefe0 R14: 0000000000000000 R15: 0000000000000000
Mar 26 13:29:48 server kernel: [ 217.602645] </TASK>
Mar 26 13:29:48 server kernel: [ 217.602646] ---[ end trace 836ff300d5b60eae ]---
Mar 26 13:29:02 server kernel: [ 171.484052] Call Trace:
Mar 26 13:29:02 server kernel: [ 171.484051] __x64_sys_ioctl+0x91/0xc0
Mar 26 13:29:02 server kernel: [ 171.484053] <TASK>
Mar 26 13:29:02 server kernel: [ 171.484057] do_syscall_64+0x61/0xb0
Mar 26 13:29:02 server kernel: [ 171.484061] ? syscall_exit_to_user_mode+0x27/0x50
Mar 26 13:29:02 server kernel: [ 171.484065] ? do_syscall_64+0x6e/0xb0
Mar 26 13:29:02 server kernel: [ 171.484068] ? syscall_exit_to_user_mode+0x27/0x50
Mar 26 13:29:02 server kernel: [ 171.484071] ? do_syscall_64+0x6e/0xb0
Mar 26 13:29:02 server kernel: [ 171.484074] ? do_syscall_64+0x6e/0xb0
Mar 26 13:29:02 server kernel: [ 171.484076] ? do_syscall_64+0x6e/0xb0
Mar 26 13:29:02 server kernel: [ 171.484079] ? asm_sysvec_call_function+0xa/0x20
Mar 26 13:29:02 server kernel: [ 171.484055] vcpu_enter_guest+0x354/0x11e0 [kvm]
Mar 26 13:29:02 server kernel: [ 171.484084] entry_SYSCALL_64_after_hwframe+0x44/0xae
Mar 26 13:29:02 server kernel: [ 171.484089] RIP: 0033:0x7ff6420a83db
Mar 26 13:29:02 server kernel: [ 171.484092] Code: 0f 1e fa 48 8b 05 b5 7a 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 85 7a 0d 00 f7 d8 64 89 01 48
Mar 26 13:29:02 server kernel: [ 171.484095] RSP: 002b:00007ff637071248 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Mar 26 13:29:02 server kernel: [ 171.484099] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007ff6420a83db
Mar 26 13:29:02 server kernel: [ 171.484101] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000019
Mar 26 13:29:02 server kernel: [ 171.484104] RBP: 000055e0cf5e19e0 R08: 000055e0cd790e68 R09: 000000003b9aca00
Mar 26 13:29:02 server kernel: [ 171.484106] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
Mar 26 13:29:02 server kernel: [ 171.484108] R13: 000055e0cdddefe0 R14: 0000000000000004 R15: 0000000000000000
Mar 26 13:29:02 server kernel: [ 171.484112] </TASK>
Mar 26 13:29:02 server kernel: [ 171.484113] ---[ end trace 836ff300d5b227e9 ]---
Mar 26 13:29:02 server kernel: [ 171.484111] ? kvm_skip_emulated_instruction+0x1f/0x40 [kvm]
Mar 26 13:29:02 server kernel: [ 171.484144] ------------[ cut here ]------------
Have reinstalled the server without HWE kernel. Let’s see if it continues to run well.
-30 is safe, -38 may be good, -37 and -39 are definitely broken.
1 Like