Slow ZFS performance on LXD instances

LXD
82

i will delete the pool. destrou the lxd storage pool and recreate it

83

@shimmy,

I recreate the pool with ashift=13. and i split 1 nvme in 2 partitions. 1 for slog 1 for 2larc.

system is 20 times faster now. i will reimport all data and see if it stays this way. the setup is now as follow:

root@esx:~# zpool status
pool: default
state: ONLINE
config:

    NAME                                                STATE     READ WRITE CKSUM
    default                                             ONLINE       0     0     0
      mirror-0                                          ONLINE       0     0     0
        ata-Samsung_SSD_870_QVO_1TB_S5RRNF0R642530P     ONLINE       0     0     0
        ata-Samsung_SSD_870_QVO_1TB_S5RRNF0R644421A     ONLINE       0     0     0
      mirror-1                                          ONLINE       0     0     0
        ata-Samsung_SSD_870_QVO_1TB_S5RRNF0R642539W     ONLINE       0     0     0
        ata-Samsung_SSD_870_QVO_1TB_S5RRNF0R642467A     ONLINE       0     0     0
    logs
      nvme-Samsung_SSD_980_500GB_S64DNF0R517822Y-part2  ONLINE       0     0     0
    cache
      nvme-Samsung_SSD_980_500GB_S64DNF0R517822Y-part1  ONLINE       0     0     0

errors: No known data errors

pool: downloads
state: ONLINE
config:

    NAME                                           STATE     READ WRITE CKSUM
    downloads                                      ONLINE       0     0     0
      ata-Samsung_SSD_870_QVO_1TB_S5RRNF0R642514M  ONLINE       0     0     0

errors: No known data errors

pool: usb_backup
state: ONLINE
config:

    NAME                                           STATE     READ WRITE CKSUM
    usb_backup                                     ONLINE       0     0     0
      ata-Samsung_SSD_870_QVO_2TB_S5RPNF0R603981X  ONLINE       0     0     0

errors: No known data errors

84

Congrats! You got those brag’n rights that I was talking about in the process, too. With those disks, in that configuration, you should have a rocketship. Your only perceivable bottleneck is going to be your physical nics and lxd network configuration. If you’re dissatisfied with your throughput; then you’ll have to look at your options. With your use case I’d look into a macvlan before using a bridge because it’s faster.

85

Hi Shimmy,

Ho can i easily migrate to vlan? i want to keep everything in the same netwerk plus i have one machine that uses 2 nics(bridge) firewall.

Also if my system crashes and i reinstall the host is it possible to add the excisting zpool(from my 4 other disks) to the host?

86

It’s not too hard at all; it’s just a learning curve. Just take time to read the docs, tutorials, and reflect on how best to use the capabilities for your use case. Like anything linux, you have to experiment a little, but with focus and determination you get a customized deployment that meets your exact specifications. In a couple of days you’ve gone from being comfortable with Btrfs to being fairly confident with ZFS and some of it’s intricacies; for instance building a complicated zpool from scratch (several times).

To figure out what networking solution is best; you’ll have to click on links and go down those rabbit holes a bit, but pay attention to the information about macvlans and bridges with fans. The guy responsible for LXD, @stgraber, has video on youtube for discussing different ways on how to access containers from outside networks; such as the LAN or Internet:

https://www.youtube.com/watch?v=TmGvbXfwJEA&t=477s

Here’s the top of the rabbit hole I mentioned earlier:

https://linuxcontainers.org/lxd/docs/latest/howto/network_create/

87

Hi Thank you for all your help!

i will read into the topic. One question about zfs. when i reinstall my host os can i then reimport my zfs pool with containers to the host os?

zfs import i believe? am i correct?

88

By the sounds of it; you are looking to reinstall your host OS? If you are asking about adding/removing a whole zpool and being able to view/use the contents as normal after the reinstall. Then zfs can do that. But while LXD can do things like this with “storage volumes” I’m not confident that you can do this with a whole pool. In LXD volumes are a subset of pools and there is not an “attach/detach” feature for pools.

89

Hi @shimmy ,
I was hoping you could help me some more regarding macvlan. I still us bridged interfaces to connect everything together. My nics are 1gb nics. If i am going to use macvlan is the trueput faster then? From container to container or vm?

I now have my host and here i create 2 bridges (wan and lan). Then i have multiple containers and vm machines. One of these is a firewall where all traffic runs trough. (wan and lan)

kind regards,

Chris

90

Hypothetically, it can be faster, but I never liked the macvlan implementations used by LXD; I’ve switched to running VAs instead of VEs. Less system overhead and easier network handling (automatic). So, theoretically yes with VEs, but from holistic system perspective VAs with bridges offers “seat of pants” equal or better performance IMO. With all that speed on board, you’d be best served by investing into a 10GBe NIC. Of course you’re upgrading the rest of your network L3 on down to help realize those speeds. But blazing fast is blazing fast.

Also, I apologize for the late reply. Thank-you for your patience.

91

Hi @shimmy,

No problem. Hope you are doing wel with your healt. I think you mean with vas (virtual machines) but what do you mean with ves?

Ok so now i have everything bridged network wise.

Kind regards,

Chris

92

VM = Virtual Machines (Qemu/KVM)
VE = Virtual Environments (LXC)
VA = Virtual Application (podman)