Pool creation with ashift=9 for older SAS drives?

IncusOS
1

Working with refurbished hardware, I noticed that IncusOS hardcodes the 2026-sensible default of asize=12 on pool creation. However, I’m working with a stack of older SAS drives (yeah :expressionless_face:) that have native 512B sector sizes. Would it be of wider utility to have an option to specify (or override) asize=9, or should I go the hackish route where I create a pool with the right value and then import it using import-storage-pool? I wouldn’t be asking if there were an end in sight of current price levels.

2

Yeah, feels like something we can add to the create-pool logic easily enough.
We’d probably phrase it as alignment or something similar, then set the correct ashift behind the scenes.

Feel free to file a request at Sign in to GitHub · GitHub

3

The main downside of an ashift=9 zpool is that down the road if you add a 4KiB native drive you’ll get massive write amplification as ZFS’s use of 512 byte blocks will force the new drive to rewrite its 4KiB sectors multiple times.

Conversely, ashift=12 on a 512 byte native drive does waste a little bit of space when not performing writes that are multiples of 4KiB, but that overhead amortizes towards zero pretty quickly for large writes. Probably the only workloads to benefit from an exact alignment match with the smaller sector size would be lots of small writes, or possibly some raidz configurations based on comments I’ve seen online.

Supporting different alignments wouldn’t be hard to do, but also feels like it could be an easy gotcha that we’ll want to properly warn about.

4

I concur: ashift=12 is The Right Thing™ even with 512 byte sector drives. You lose a few percent in overall storage capacity, but the downsides of ashift=9 are pretty major (I’ve been bitten in the past, but it was so long ago I’d have to dig for the details).

5

thank you for the replies everyone, much appreciated.

Yeah, the use-case is a couple of raidz2 sets. I have a stack of spares, so replacement with 4K drives is quite unlikely. I’ll do some more research in benchmarks. If the difference is negligible, all the better.

The counterarguments make a lot more sense to me in the general case, so I understand if this will remain unsupported.

6

The ability to set a custom alignment value when creating a storage pool has been added in this PR

7

Wow, that was fast. Also, the ability to configure this as part of a seed is really neat! I really look forward to trying this out on a test system. Thank you for picking this up.

Is a feature like this eligible to get backported for a future LTS point release, or are those strictly fixes only?

8

This is all implemented in the IncusOS management logic, so it should be included in the next release. There’s currently no “LTS” branch of IncusOS; the Incus application does have a LTS variant, but that’s separate from the underlying IncusOS system image.

9

Ah, I had not realised versioning was decoupled like that. I’ll try to give the new functionality a spin in the week of Sept 21.

10

I just had a go, but I’m getting stuck before I can confirm things work. I’m probably holding it wrong. So to verify, is this what storage.yaml as part of a seed configuration should look like to configure a local pool as a raid1 over two devices using the new alignment feature?

- devices:
      - /dev/disk/by-partlabel/local-data
      - /dev/disk/by-id/scsi-35000c500XXXXXXXX
    name: local
    type: zfs-raid1
    alignment: 512

I’m asking, because after a successful installation, I get the following error on boot:

ERROR refusing to create new zpool with devices of different sizes unless AllowMixedDevSizes is true

I picked up /dev/disk/by-partlabel/local-data from the relevant documentation. Is AllowMixedDevSizes always needed for local because IncusOS will be mirroring a partition to a disk? If so, should that be the default?

FWIW: The disks are equally sized; I have manually created a zfs-raid1 pool over two devices on these same disks in the past post-installation.

I noticed these things too:

  1. reading the seed documentation, it was unclear to me what the difference is between StoragePools in incus.yaml and pools in storage.yaml. I picked storage.yaml at random.
  2. The webbased image builder allows one to specify a network seed, but not a storage seed. I read back the tar from the second partition, modified it and used dd to overwrite the partition with the modified tar. That gave a PE integrity check error. Using the flasher-tool did work, which I guess relates to an updated corresponding checksum in the first partition?