Image generation fails when publishing

eneko · February 9, 2018, 10:44am

Before hand I would like to remark I am no expert on lxd or zfs.
I am getting the following error

error: failed to copy file content: write /var/lib/lxd/images/lxd_build_268164345/lxd_build_tar_225871251: no space left on device

So I have done my homework and found something alike here: https://github.com/lxc/lxd/issues/2135

I thought my case was the same but now I think is just similar.
My server’s filesystem set up was 2x480 SSD and 2x2TB disks.
The SSDs have the root filesystem in /dev/md1 raid0, swap and the rest was intended for lxd containers so I created a zfs mirror partition called vmpool.
The extra 2TB were formated as ext4 in raid but now I formated it to a zfs mirror too, called it vmpool2.

A tried to follow the steps described in github until I realized it wasn’t my case.

zfs umount" of anything under /varlib/lxd/images
"mv /var/lib/lxd/images /var/lib/lxd/images.pre
“zfs create vmpool/lxd/images -o mountpoint=/var/lib/lxd/images”
“mv /var/lib/lxd/images.pre/* /var/lib/lxd/images/” (I copy it instead)
“rmdir /var/lib/lxd/images.pre” (haven’t done this yet)
“zfs mount” of anything under /var/lib/lxd/images

This last step is confusing for me as it says they are mounted under vmpool and not vmpool2.
I am guessing my images were already in vmpool and that’s where I don’t have enough space. How can I move them to vmpool2 safely?

This is some outputs of my current situtation:

df -h
Filesystem                                                                      Size  Used Avail Use% Mounted on
udev                                                                             32G     0   32G   0% /dev
tmpfs                                                                           6.3G  661M  5.7G  11% /run
/dev/md1                                                                         87G   65G   18G  79% /
tmpfs                                                                            32G     0   32G   0% /dev/shm
tmpfs                                                                           5.0M     0  5.0M   0% /run/lock
tmpfs                                                                            32G     0   32G   0% /sys/fs/cgroup
vmpool/containers/bird01                                                        207G   69G  139G  33% /var/lib/lxd/containers/bird01.zfs
vmpool/containers/proxy01                                                       140G  912M  139G   1% /var/lib/lxd/containers/proxy01.zfs
vmpool/containers/ubuntu01                                                      140G  747M  139G   1% /var/lib/lxd/containers/ubuntu01.zfs
vmpool/containers/web01                                                         140G  767M  139G   1% /var/lib/lxd/containers/web01.zfs
vmpool/containers/web02                                                         140G  756M  139G   1% /var/lib/lxd/containers/web02.zfs
vmpool/containers/wp01                                                          141G  1.9G  139G   2% /var/lib/lxd/containers/wp01.zfs
vmpool/containers/proxy01bk                                                     140G  792M  139G   1% /var/lib/lxd/containers/proxy01bk.zfs
tmpfs                                                                           100K     0  100K   0% /var/lib/lxd/devlxd
vmpool/containers/bird01bk                                                      207G   68G  139G  33% /var/lib/lxd/containers/bird01bk.zfs
vmpool/containers/wp01bk                                                        141G  1.9G  139G   2% /var/lib/lxd/containers/wp01bk.zfs
vmpool2/images                                                                  1.8T   25G  1.8T   2% /var/lib/lxd/images
vmpool/images/0bc1af456e0b380b5572bbd1715959b598224fd986cd1a9872f2fa923ecb00ad  140G  756M  139G   1% /var/lib/lxd/images/0bc1af456e0b380b5572bbd1715959b598224fd986cd1a9872f2fa923ecb00ad.zfs
vmpool/images/0f150b45150f2649f75f317637937d2df4abef4d29c93cb4ca862c22be394fe9  187G   49G  139G  26% /var/lib/lxd/images/0f150b45150f2649f75f317637937d2df4abef4d29c93cb4ca862c22be394fe9.zfs
vmpool/images/1cd84e6da1c68612cbcb233eb8eeaed5ebd5af1850d1e9fde658ba913dac5296  147G  7.6G  139G   6% /var/lib/lxd/images/1cd84e6da1c68612cbcb233eb8eeaed5ebd5af1850d1e9fde658ba913dac5296.zfs
vmpool/images/55b0df04a2d657f2f4e7e2bdb867f4c477cbdaa59122801d736cf9e03899dbe0  186G   47G  139G  26% /var/lib/lxd/images/55b0df04a2d657f2f4e7e2bdb867f4c477cbdaa59122801d736cf9e03899dbe0.zfs
vmpool/images/567cc6fc2ff10f9f13664c28297d56562ba9740c583323cc8ac40b679f1d70bf  140G  767M  139G   1% /var/lib/lxd/images/567cc6fc2ff10f9f13664c28297d56562ba9740c583323cc8ac40b679f1d70bf.zfs
vmpool/images/8220e89e33e6f62b56cb451cfed61574074416a66a6e7c61ff574d95572e6661  140G  650M  139G   1% /var/lib/lxd/images/8220e89e33e6f62b56cb451cfed61574074416a66a6e7c61ff574d95572e6661.zfs
vmpool/images/8540281cfdefe904332ceef5591799244c017aa7e8f27085f2ac72dee6426073  140G  788M  139G   1% /var/lib/lxd/images/8540281cfdefe904332ceef5591799244c017aa7e8f27085f2ac72dee6426073.zfs
vmpool/images/89d10330148b51d04ef03b7704854b0d266d16672640749b262e00097a1123b4  140G  760M  139G   1% /var/lib/lxd/images/89d10330148b51d04ef03b7704854b0d266d16672640749b262e00097a1123b4.zfs
vmpool/images/8a3b6cc4b594724c69aaffa6fe2e847fce9533cd59377d8707f31f04e4aaaf22  140G  647M  139G   1% /var/lib/lxd/images/8a3b6cc4b594724c69aaffa6fe2e847fce9533cd59377d8707f31f04e4aaaf22.zfs
vmpool/images/9c89741fbb3565e34aa95dbef6c4e1f79059ac619d4802fb16323fe3fa561406  140G  1.5G  139G   2% /var/lib/lxd/images/9c89741fbb3565e34aa95dbef6c4e1f79059ac619d4802fb16323fe3fa561406.zfs
vmpool/images/f092e40aa4d39dde09aa41b63d635b05ae1c443754bf47b1987eae7bea0fcb59  141G  1.8G  139G   2% /var/lib/lxd/images/f092e40aa4d39dde09aa41b63d635b05ae1c443754bf47b1987eae7bea0fcb59.zfs
tmpfs                                                                           6.3G     0  6.3G   0% /run/user/1000

and

zfs list
NAME                                                                                     USED  AVAIL  REFER  MOUNTPOINT
vmpool                                                                                   208G   139G    96K  none
vmpool/containers                                                                       97.2G   139G    96K  none
vmpool/containers/bird01                                                                93.1G   139G  68.1G  /var/lib/lxd/containers/bird01.zfs
vmpool/containers/bird01bk                                                               392K   139G  67.7G  /var/lib/lxd/containers/bird01bk.zfs
vmpool/containers/proxy01                                                               1.08G   139G   911M  /var/lib/lxd/containers/proxy01.zfs
vmpool/containers/proxy01bk                                                             5.74M   139G   791M  /var/lib/lxd/containers/proxy01bk.zfs
vmpool/containers/ubuntu01                                                               183M   139G   747M  /var/lib/lxd/containers/ubuntu01.zfs
vmpool/containers/web01                                                                 9.07M   139G   767M  /var/lib/lxd/containers/web01.zfs
vmpool/containers/web02                                                                 9.07M   139G   756M  /var/lib/lxd/containers/web02.zfs
vmpool/containers/wp01                                                                  2.80G   139G  1.84G  /var/lib/lxd/containers/wp01.zfs
vmpool/containers/wp01bk                                                                 392K   139G  1.81G  /var/lib/lxd/containers/wp01bk.zfs
vmpool/deleted                                                                           704M   139G    96K  none
vmpool/deleted/containers                                                                 96K   139G    96K  none
vmpool/deleted/images                                                                    704M   139G    96K  none
vmpool/deleted/images/d23ee1f4fd284aeaba6adeb67cccf7b871e96178d637fec96320aab7cc9634b1   704M   139G   704M  none
vmpool/images                                                                            110G   139G    96K  none
vmpool/images/0bc1af456e0b380b5572bbd1715959b598224fd986cd1a9872f2fa923ecb00ad           755M   139G   755M  /var/lib/lxd/images/0bc1af456e0b380b5572bbd1715959b598224fd986cd1a9872f2fa923ecb00ad.zfs
vmpool/images/0f150b45150f2649f75f317637937d2df4abef4d29c93cb4ca862c22be394fe9          48.4G   139G  48.4G  /var/lib/lxd/images/0f150b45150f2649f75f317637937d2df4abef4d29c93cb4ca862c22be394fe9.zfs
vmpool/images/1cd84e6da1c68612cbcb233eb8eeaed5ebd5af1850d1e9fde658ba913dac5296          7.55G   139G  7.55G  /var/lib/lxd/images/1cd84e6da1c68612cbcb233eb8eeaed5ebd5af1850d1e9fde658ba913dac5296.zfs
vmpool/images/55b0df04a2d657f2f4e7e2bdb867f4c477cbdaa59122801d736cf9e03899dbe0          46.9G   139G  46.9G  /var/lib/lxd/images/55b0df04a2d657f2f4e7e2bdb867f4c477cbdaa59122801d736cf9e03899dbe0.zfs
vmpool/images/567cc6fc2ff10f9f13664c28297d56562ba9740c583323cc8ac40b679f1d70bf           767M   139G   767M  /var/lib/lxd/images/567cc6fc2ff10f9f13664c28297d56562ba9740c583323cc8ac40b679f1d70bf.zfs
vmpool/images/8220e89e33e6f62b56cb451cfed61574074416a66a6e7c61ff574d95572e6661           649M   139G   649M  /var/lib/lxd/images/8220e89e33e6f62b56cb451cfed61574074416a66a6e7c61ff574d95572e6661.zfs
vmpool/images/8540281cfdefe904332ceef5591799244c017aa7e8f27085f2ac72dee6426073           788M   139G   788M  /var/lib/lxd/images/8540281cfdefe904332ceef5591799244c017aa7e8f27085f2ac72dee6426073.zfs
vmpool/images/89d10330148b51d04ef03b7704854b0d266d16672640749b262e00097a1123b4           759M   139G   759M  /var/lib/lxd/images/89d10330148b51d04ef03b7704854b0d266d16672640749b262e00097a1123b4.zfs
vmpool/images/8a3b6cc4b594724c69aaffa6fe2e847fce9533cd59377d8707f31f04e4aaaf22           646M   139G   646M  /var/lib/lxd/images/8a3b6cc4b594724c69aaffa6fe2e847fce9533cd59377d8707f31f04e4aaaf22.zfs
vmpool/images/9c89741fbb3565e34aa95dbef6c4e1f79059ac619d4802fb16323fe3fa561406          1.45G   139G  1.45G  /var/lib/lxd/images/9c89741fbb3565e34aa95dbef6c4e1f79059ac619d4802fb16323fe3fa561406.zfs
vmpool/images/f092e40aa4d39dde09aa41b63d635b05ae1c443754bf47b1987eae7bea0fcb59          1.79G   139G  1.79G  /var/lib/lxd/images/f092e40aa4d39dde09aa41b63d635b05ae1c443754bf47b1987eae7bea0fcb59.zfs
vmpool/snapshots                                                                         384K   139G    96K  none
vmpool/snapshots/bird01                                                                   96K   139G    96K  none
vmpool/snapshots/proxy01                                                                  96K   139G    96K  none
vmpool/snapshots/wp01                                                                     96K   139G    96K  none
vmpool2                                                                                 24.3G  1.73T    96K  none
vmpool2/images                                                                          24.3G  1.73T  24.3G  /var/lib/lxd/images

simos · February 12, 2018, 12:09pm

The no space left on device means that the partition got full.

eneko · February 12, 2018, 1:16pm

I understand it gets full when creating an image, that is why I am trying to move images to another zfs pool. Unfortunately I see images themselves are also zfs filesystems. Am I right?
So I created a new zfs pool called vmpool2 and mounted as /var/lib/lxd/images
I moved all images there but when I remount images individually they are still pointing to vmpool

vmpool2/images                                                                  1.8T   25G  1.8T   2% /var/lib/lxd/images
vmpool/images/0bc1af456e0b380b5572bbd1715959b598224fd986cd1a9872f2fa923ecb00ad  140G  756M  139G   1% /var/lib/lxd/images/0bc1af456e0b380b5572bbd1715959b598224fd986cd1a9872f2fa923ecb00ad.zfs

How can I change this?

simos · February 12, 2018, 2:04pm

Okay, got confused by the title “Image generation fails when publishing”.

At https://github.com/lxc/lxd/issues/2135 it says that when generating a container image (Through lxc publish), it uses temporary space from your distribution (the root filesystem, “/”). Therefore, you may have lots of space for ZFS, but the root filesystem happens to have too little.
From your case, it looks like /dev/md1 (the “/” root filesystem) has 18GB free space, which should be enough.

When you publish an image, it creates a tarball and then take the hash of that tarball. It is possible that this image happens to have lots of files and the tarball files up the space.

Can you post the output of

lxc image list

It looks like two of your images are about 50GB, and if are trying to generate another of those, then it would fill the 18GB free space.

eneko · February 13, 2018, 7:47am

Thanks for your reply. By creating a new zfs pool (vmpool2) and mounting it as /var/lib/lxd/images the space problem is solved.
What I discovered is that a temporary dir is created inside this folder called lxd_build_xxxxxx during publishing. In my case it was generating a 65GB file and then a compressed file of about 10GB.

What is confusing yet for me is that once the image is published it is mounted as vmpool/images/whateveridentifier
Does this mean it is using space in both pools? vmpool and vmpool2?