I mean, if I create multiple containers from the same image, the image size is about 2G, then the containers would occupy 2*X G storage, where X is the number of the container? Or only the differences between the container and the image would occupy additional storage?
Hi!
See the section Feature comparison at https://lxd.readthedocs.io/en/latest/storage/
This means that depending on the storage backend, you may get optimized storage. Specifically, if you use the dir storage backend, you do not get any storage optimization. With any of the rest you do get storage optimization.
Oh, thx. That what “Optimized image storage” mean. I use btrfs as the storage backend. According to the document, btrfs supports more features, but you recommends ZFS over btrfs a little bit?
If your host runs Ubuntu, then ZFS is a safe bet. All Ubuntu Linux kernels come with ZFS built-in, so ZFS is always available. Other distributions may offer ZFS as a DKMS module.
For other distributions it may be more suitable to select btrfs.
For any distribution you can choose LVM, Ceph, etc.
Regarding the future of ZFS on Ubuntu, see
@simos I cannot still understand when btrfs is part of Linux kernel, feature parity with zfs (except may be minor things here and there), then why to promote zfs unfairly over btrfs.
So far been using btrfs for over 5 years with lxc and later lxd didn’t have any issue. So I don’t know, why there is so much promotion and talking of zfs not btrfs. I am ok if both are equally treated because both are CoW.
My main issue with btrfs is its broken quota system. It’s effectively impossible to truly limit the size of a container with it, which disqualifies it immediately for any environment where an untrusted process is allowed to write data to disk.
Last time I checked btrfs team resolved many of the issues with quota and qgroup. But I am not sure if its at same level as zfs. Performance wise there is not much difference between zfs and btrfs. Indeed xfs and ext4 are better in performance, but as they are not CoW its not a good use-case for containers.
So you think even in 2019 btrfs hasn’t come over the performance issues of quota and qgroups?
Performance issue is still mentioned in btrfs man pages.
I think this is grossly unfair.
In my first reply above, I show the comparison table of the different storage backends, without mentioning any specific storage backend.
In my second and third replies, I suggest that ZFS on an Ubuntu host is a safe bet. Because the kernel module is built-in into the provided Ubuntu Linux kernel, while elsewhere it is often provided as a DKMS module. Also, ZFS on Ubuntu is supported by Canonical and there are more good things coming up in newer versions of Ubuntu.
You write that you have been using btrfs for over 5 years with lxc and later lxd didn’t have any issue.
This is empirical experience and does not carry great weight. Software can fail in spectacular ways and you need more comprehensive testing to figure out how well it works.
See the following comparison post (June 2019) between ZFS, btrfs and mdadm, which goes into depth about data integrity issues,
https://www.unixsheikh.com/articles/battle-testing-data-integrity-verification-with-zfs-btrfs-and-mdadm-dm-integrity.html
@simos Apologize I didn’t mean it that way to criticize you. My evidence is anecdotal so don’t think it can represent regression tests.
I am not sure whats the status of ZFS on Linux vs btrfs. But in spite of deficiency in btrfs I prefer it over ZFS given Oracle’s sword will always be on neck. They might lose in court, but its a big expense to even fight with them in court. Given their record with Java and what they have done with it. I prefer not to touch something which is patent encumbered or appear to be one.
So I prefer btrfs over ZFS. The only issue with btrfs has been in RAID56 (its considered unstable). So I just contend with 10 or 1 for redundancy and do not use RAID 5, probably that’s the reason I did not hit edge cases.
Unless there is a case precedence or being ruled that ZoL (ZFS on Linux) is not patent encumbered with oracle, I will use it, until then I will just rely on btrfs or xfs.
Regarding any licensing issues, Canonical made a statement at https://ubuntu.com/blog/zfs-licensing-and-linux
That statement is adequate for me. If any drama ever arises in the future, then Canonical will first deal with it.
A blog post by Petros Koutoupis on ZFS and Ubuntu,
https://koutoupis.com/2019/08/26/ubuntu-and-zfs-on-linux-and-how-to-get-it-right/
It talks about the integration of ZFS into Ubuntu and the benefits that it can reap from all this.
Still, other major distributions do not include ZFS in their Linux kernel. ZFS is an addon to them through DKMS or FUSE. DKMS support may be adequate for them, however ZFS has to be recompiled everytime those other kernels are updated. That’s a requirement when you use DKMS modules.
XFS has now support for copy on write. That’s very probably why RedHat will no longer include btrfs. I dont’ think that COW is supported for XFS in LXD.
Latest update from Linus Torvalds recommending not to use ZFS with linux. It’s one more reason not to promote ZFS on lxd and change the examples to use btrfs or other filesystem which is not license encumbered, as it might fail spectacularly given kernel developers don’t care if its broken, as its not a kernel module.
https://www.realworldtech.com/forum/?threadid=189711&curpostid=189841