LXD export and root file system size

LXD
1

If I have relatively small / filesystem, but large /opt partition and I want to export a container to /opt, and said container is larger than free space on / then lxc export fails:

Error: Create backup: Backup storage: Failed to run: tar -cf /var/snap/lxd/common/lxd/backups/main/backup0 --xattrs -C /var/snap/lxd/common/lxd/backups/lxd_backup_583972121 --transform s,^./,backup/, .: tar: /var/snap/lxd/common/lxd/backups/main/backup0:   Wrote only 2048 of 10240 bytes
tar: Error is not recoverable: exiting now

Then the machine is left in broken state: disk is full, lxc list is unresponsive, etc. I have to go to /var/snap/lxd/common/lxd/backups and erase everything there, then I have to reboot the machine, snap restart lxd does not help.

I tried to avoid this problem symlinking backup directory, making /opt/backups and pointing /var/snap/lxd/common/lxd/backups to /opt/backups. Does not work:

Error: Create backup: Backup storage: stat /var/snap/lxd/common/lxd/backups: no such file or directory

So, it seems that I can’t avoid dumping container to root filesystem first. Is it possible to change this behavior to export containers even if root filesystem is smaller than largest container? Leaving a lot of rarely used space in root fs does not make sense when SSDs are involved
 what about an option --backup-tmpdir or something like that?

2

The symlink won’t work because of the way the snap environment works.
A bind-mount would work though.

So make sure that /var/snap/lxd/common/lxd/backups is a directory, then use mount --bind /opt/backups /var/snap/lxd/common/lxd/backups.

We have an open issue upstream to make it configurable where images and backups are stored to avoid such issues.

3

Yep, thanks, this works. What about free space check before export?

4

We can’t look for free space ahead of time as it’s impossible to tell how big a backup will be ahead of time. Some containers will be using 2GB on ZFS but will take 6GB or more in backup. Some containers on dir backend can shrink significantly, 


What we need though is to ensure that LXD cleans things up on failure to generate the backup so that you don’t have to go cleaning things up and reboot at least. @monstermunchkin

5

I sent a fix for the error codepath cleanup, it’s been merged upstream.

6

Hey @stgraber,

Could you supply the link to the issue here?

Cheers

7
1 Like
8

Hi,

Is it allredeay possible to use a temp directory to export containers? Without using root space? How can i manage this?

9

it’s possible if you use LXD 3.17, on most distros it means using snap LXD. I have never used the feature myself but by clicking the link you should see that adding a pool on a dedicated partition and creating a custom volume (lxc storage create <driver/type zfs, btrfs,
> source=<path/to/partition> then lxc storage volume ), then using the storage.backup key to the config (lxc config set storage.backup=pool/volume should do the trick.

10

hmm

i installed lxd true snap.

The key storage.backup is not recognized?

11

a it needed to be lxc config set storage.backups_volume=pool/volume

12

hmm

followed the steps and now nothing works anymore

lxc list errors out?

Error: Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused

how can we resolve this?

13

If I understand correctly, you had LXD 3.0 installed from system packages and switched to snap, in this case you should uninstall the lxc system packages; to test this try to run
/snap/bin/lxc list
if it works, that’s the problem.

14

Hi no i always installed true snap.

I just created a new pool for my exports.

After reboot lxd is not starting anymore

15

maybe you installed your pool on a partition that is not mounted when LXD starts ? if you restart lxd manually by running snap restart lxd, does it work better ?

16

No the daemon fails to start??

17

Also the partition itself is mounted.

I think lxd does not mount it?

18

Does looking at
sudo tail -n 100 /var/snap/lxd/common/lxd/logs/lxd.log
just after the daemon fails to start bring some light to the problem ?

19

lvl=info msg=“LXD 3.20 is starting in normal mode” path=/var/snap/lxd/common/lxd t=2020-02-05T17:21:36+0100 lvl=eror msg=“Failed to start the daemon: Failed to chmod dir /var/snap/lxd/common/lxd/backups: chmod /var/snap/lxd/common/lxd/backups: no such file or directory” t=2020-02-05T17:21:36+0100 lvl=info msg=“Starting shutdown sequence”

20

The backup folder is linked to the volume i created but it can not be found

backups -> /var/snap/lxd/common/lxd/storage-pools/exports/custom/exports_volume