I’ve experimented with LXC on an Antsle server. Now I’d like to run a bunch of tests on in-house hardware. But have many noobie questions that I’ll spin out one at a time:
Question: My systems each have a single hard drive that I can dedicate to the project. What file system should I choose?
Thanks,
Lloyd
Have a look at incus/doc/storage.md at main · lxc/incus · GitHub
There is a table with the available storage backends (i.e. filesystems) and there is also a section for the recommended storage backends. Spoiler: use ZFS, or btrfs if ZFS is not available.
In your case, are you able to repartition the disk? It would be greatly beneficial to create a separate partition to put the storage. Because LXD has the feature to be able to create the ZFS (or btrfs) storage on a loop file (of repartitioning is a no no).
When you run sudo lxd init for the first time, you will be asked for the following. We accepted the defaults (new storage pool, to be named default, select ZFS, new ZFS pool, and finally use an existing block device). You specify the partition device (here, it was /dev/sde5). This partition should be empty. Do not create a ZFS filesystem in there because LXD will do this for you.
...
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (btrfs, dir, lvm, zfs) [default=zfs]:
Create a new ZFS pool? (yes/no) [default=yes]:
Would you like to use an existing block device? (yes/no) [default=no]: yes
Path to the existing block device: /dev/sde5
...
Many thanks Simos for your thoughtful reply. This is the step that has kept me from moving on.
I wasn’t clear in my post. My test systems have two drives— one ssd for OS and a second drive that I can dedicate to LXD. I was hesitant to select ZFS since from what I’ve read it requires two to four drives. And, since I have no experience with btrfs or LVM, I thought I had better ask wiser heads.
Incidentally, the OS drives could conceivably be partioned to provide additional space.
Thanks again,
Lloyd
You can dedicate the full second disk to ZFS, or a partition.
ZFS does not require multiple disks but works better if you have them. With ZFS you can get RAID features if you have the disks. But it is still OK if it is a single disk. Many people use ZFS on a single disk.
Thank you!
I have many more questions re scaling and networking, but will post them on separate threads.
Lloyd
Be careful of ZFS and LXD in production. I have been using LXD since version 1. And at least half the problems I have had is with ZFS or I should say how LXD recovers from any problems. I am presently trying to resolve a similar problem. Because if you have to reinstall LXD you lose your Zfs Pool.
For me the I am unsure if I am go with ZFS in Future. I am tired of issues with it. There is also fragmentation issues with SSD with ZFS
That’s one of the ideas that made me pick Btrfs in fact. I wanted to use also LXD on Raspberry PI. I wanted only to be familiar with one file system, I did not want to have to think: it’s a raspberry pi, so I have to remember the btrfs commands, it’s not i have to remember zfs commands. I use btrfs everywhere and so I reduce my ‘cognitive load’. Bonus point; I am not tempted to answer questions about zfs on this forum 
If not ZFS, what do you recommend?
Thanks,
Lloyd
Thinking plain file. My biggest concern is recovery. I am going to try in a few minutes actually and see If can recover zfs pool into old ZFS pool into new lxd install.
There is nothing wrongs with Zfs, it is how LXD puts its containers in it I don’t like.
may be it is my lack of knowledge, but it makes recovery very difficult. And it is hard to get support for ZFS here.
Please share your experience. I look forward to it.
Best wishes,
Lloyd
If your zpool is functional and you see all your containers in zfs list -t all, then recovery by mounting them into their usual spot with zfs mount followed by importing them back into the database with lxd import works fine.
So are you saying I should do a LXD init and attach to cluster and then do the import.
I would rather fixed the LXd In have installed now that is stuck and wont run after I rebooted.
Any ideas?