Testing out LXD 3.19 candidate

I cannot find a config drive in the output of mount on the host. The only config-related is configfs on /sys/kernel/config type configfs (rw,relatime) which I think is unrelated?

The config.iso file does not exist (at /var/snap/lxd/common/lxd/virtual-machines/vm3/) when I get this error.

Going to try VM in VM? good luck :slight_smile:

Ha indeed :slight_smile:

@simos that’s because it goes away, we only mount that path when trying to start the container.

@simos but if you remove the config drive, so you can start the VM, then the iso file should still be around once its mounted.

I believe we already have a function to determine the backing fs of a given path and have magic numbers for all architectures too, so should be pretty straightforward to special case files on zfs and avoid aio in that case.

I just tested on 4.15 kernel, have re-created the problem. Quick test of using aio=threads didn’t help.

I need to spend a bit of time getting my graphics drivers and networking working in 4.15 so I can work in that environment for a bit (in a resolution >800x600 :))

I have got it starting using:

cache = "writeback"
aio = "threads"

However that seemingly suffers from the kernel crash we saw on ZFS loopback files that froze my whole system.

There’s a useful table in the qemu manual:

 cache=cache
               cache is "none", "writeback", "unsafe", "directsync" or "writethrough" and controls how the host cache is
               used to access block data. This is a shortcut that sets the cache.direct and cache.no-flush options (as in
               -blockdev), and additionally cache.writeback, which provides a default for the write-cache option of block
               guest devices (as in -device). The modes correspond to the following settings:

                                    │ cache.writeback   cache.direct   cache.no-flush
                       ─────────────┼─────────────────────────────────────────────────
                       writeback    │ on                off            off
                       none         │ on                on             off
                       writethrough │ off               off            off
                       directsync   │ off               on             off
                       unsafe       │ on                off            on

               The default mode is cache=writeback.

We have it set to “cache=none” right now, and “aio=native”.

However we need to disable cache direct, which also requires setting “aio=threads”.

I will try writethrough, which seems to turn the most stuff off.

I’m getting this crash:

[  363.385907] INFO: task jbd2/sda2-8:275 blocked for more than 120 seconds.
[  363.385914]       Tainted: P           OE    4.15.0-74-generic #84-Ubuntu
[  363.385916] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  363.385919] jbd2/sda2-8     D    0   275      2 0x80000000
[  363.385924] Call Trace:
[  363.385934]  __schedule+0x24e/0x880
[  363.385938]  schedule+0x2c/0x80
[  363.385940]  io_schedule+0x16/0x40
[  363.385947]  wait_on_page_bit_common+0xd8/0x160
[  363.385952]  ? page_cache_tree_insert+0xe0/0xe0
[  363.385957]  __filemap_fdatawait_range+0xfa/0x160
[  363.385963]  ? submit_bio+0x73/0x140
[  363.385966]  ? submit_bio+0x73/0x140
[  363.385971]  ? guard_bio_eod+0x2c/0xf0
[  363.385976]  filemap_fdatawait_range_keep_errors+0x12/0x40
[  363.385981]  jbd2_journal_commit_transaction+0x6f6/0x1730
[  363.385985]  ? __switch_to_asm+0x41/0x70
[  363.385989]  ? __switch_to_asm+0x41/0x70
[  363.385996]  kjournald2+0xc8/0x270
[  363.385999]  ? kjournald2+0xc8/0x270
[  363.386005]  ? wait_woken+0x80/0x80
[  363.386009]  kthread+0x121/0x140
[  363.386012]  ? commit_timeout+0x20/0x20
[  363.386015]  ? kthread_create_worker_on_cpu+0x70/0x70
[  363.386019]  ret_from_fork+0x35/0x40

So using kernel 4.15.0-74-generic on a ZFS loop file, the only setting that works is:

cache = "unsafe"
aio = "threads"

Note: This isn’t just for the config drive iso, this is for all drives. Whilst if the cache.direct mode is enabled, it fails to start complaining about O_DIRECT, if you remove the config drive, then it still starts to boot and then qemu uses 100% CPU and the entire I/O system halts and I have to turn my PC off as the kernel crashes.

This is even with the sync=disabled in the ZFS LXD driver too.

@stgraber should I use these options when detecting ZFS backend?

Is your zfs pool backed by a loop file or a real device?

If you remove the config drive does the vm start up ok on your zfs pool?

Thanks
Tom

I tried both cases, ZFS back by loop file and ZFS on a real device. They had the same outcome, relating to O_DIRECT missing from ZFS (v0.7.5-1ubuntu16.6, i.e. Ubuntu 18.04.3).

I edited the vm profile not to include the config drive, and I still get the same error.

$ lxc launch ubuntu:18.04 vm4 --vm --profile default --profile vm
Creating vm4
Starting vm4                                
Error: Failed to run: qemu-system-x86_64 -S -name vm4 -uuid b34ce8e1-5cf8-4374-58f1-a7647d9e1357 -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-reboot -no-user-config -readconfig /var/snap/lxd/common/lxd/logs/vm4/qemu.conf -pidfile /var/snap/lxd/common/lxd/logs/vm4/qemu.pid -D /var/snap/lxd/common/lxd/logs/vm4/qemu.log -chroot /var/snap/lxd/common/lxd/virtual-machines/vm4 -runas lxd: 
Try `lxc info --show-log local:vm4` for more info

$ lxc info --show-log local:vm4
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/vm4/qemu.conf:150: file system may not support O_DIRECT
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/vm4/qemu.conf:150: Could not open '/var/snap/lxd/common/lxd/virtual-machines/vm4/config.iso': Invalid argument

Thanks @simos

Can you double check with lxc config show v1 --expanded that the config disk isn’t being used, as the qemu config gets regenerated each time a VM is started, and if you’re still getting that error (specifically about the config.iso file), then the config disk is still being attached somehow.

I’m working on a fix for detecting ZFS in various guises and then switching to async I/O.

There is a typo on my behalf above. Although I created a vmnoconfig profile, I used the vm profile when creating vm4 (shell history). Therefore, let’s try again.

$ lxc delete vm4
$ lxc launch ubuntu:18.04 vm4 --vm --profile default --profile vmnoconfig
Creating vm4
Starting vm4
$ lxc list vm4
+------+---------+------+------+-----------------+-----------+
| NAME |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+------+---------+------+------+-----------------+-----------+
| vm4  | RUNNING |      |      | VIRTUAL-MACHINE | 0         |
+------+---------+------+------+-----------------+-----------+
$ 

Therefore, the virtual machine manages to start.

I am running Ubuntu 18.04.3 (v0.7.5-1ubuntu16.6) with ZFS on a real drive, on one computer.
The other computer with ZFS on a loop file is running on Ubuntu 19.10 (which already has a newer ZFS with O_DIRECT).

That’s interesting, on the 18.04.3 box, can you give the output of lxc info v1 --show-log when starting the VM without the config drive. I want to see if you still get the message about lack of direct I/O support or not. Thanks

I get no messages when I launch a LXD virtual machine that has no config drive.

$ lxc info vm4 --show-log
Name: vm4
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/01/22 10:23 UTC
Status: Running
Type: virtual-machine
Profiles: default, vmnoconfig
Pid: 5758
Resources:
  Processes: -1
  Disk usage:
    root: 2.22MB

Log:


Cool, so it seems that using direct I/O mode on ubuntu 18.04 with kernel 4.15 doesn’t cause issues as long as the disk is backed by a real device, and is not an image.

This means my patch should work fine, as it only detects root disks on loop file backed ZFS storage pools and disks attached pointing directly to files on ZFS filesystems.

That particular fix is in candidate

If you run snap info lxd, you can see that LXD 3.19 has been released to the stable channel.
Here is how to switch back from candidate to stable. When I run snap refresh, I pressed Enter a few times in order to capture some of the snapd messages.

$ snap switch lxd --channel stable
"lxd" switched to the "stable" channel

$ snap refresh
Download snap "lxd" (13073) from channel "stable"                              
Mount snap "lxd" (13073)                                                       
Stop snap "lxd" services                                                       
Setup snap "lxd" (13073) security profiles                                     
Start snap "lxd" (13073) services                                              
Consider re-refresh of "lxd"                                                   
lxd 3.19 from Canonical✓ refreshed
$ 

I am marking this post as Solution so that others can view the instructions to switch back to stable.

I updated to the newer lxd (currently, candidate is same as stable) and created a LXD VM.
It worked fine with ZFS (on its own partition).

I was surprised that it was so fast to launch a LXD VM. Specifically, it was less than 3s to perform the launch.

$ time lxc launch ubuntu:18.04 vm5 --vm --profile default --profile vm
Creating vm5
Starting vm5

real	0m2,832s
user	0m0,073s
sys	0m0,051s

Then, it took less than 10s to get an IP address (i.e. for systemd to complete and have systemd_networkd request an IP address).

$ lxc list vm5
+------+---------+---------------------+------+-----------------+-----------+
| NAME |  STATE  |        IPV4         | IPV6 |      TYPE       | SNAPSHOTS |
+------+---------+---------------------+------+-----------------+-----------+
| vm5  | RUNNING | 10.10.10.100 (eth0) |      | VIRTUAL-MACHINE | 0         |
+------+---------+---------------------+------+-----------------+-----------+
1 Like