"lxd export" fails with archive/tar: write too long

1

Hello,

I can not perform lxd export on some containers:

lxc export ipa1 ipa1.tar.gz
Error: Create backup: Backup create: Error adding "/var/snap/lxd/common/lxd/storage-pools/system/containers/ipa1/rootfs/var/log/krb5kdc.log" as "backup/container/rootfs/var/log/krb5kdc.log" to tarball: Failed to copy file content "/var/snap/lxd/common/lxd/storage-pools/system/containers/ipa1/rootfs/var/log/krb5kdc.log": archive/tar: write too long

This container’s storage backend is zfs:

zfs list syspool/lxd/containers/ipa1
NAME                             USED  AVAIL     REFER  MOUNTPOINT
syspool/lxd/containers/ipa1  5.67G  17.2G     5.75G  /var/snap/lxd/common/lxd/storage-pools/system/containers/ipa1

I have plenty of space available in /var/snap/lxd:

df -h /var/snap/lxd/
Filesystem      Size  Used Avail Use% Mounted on
/dev/md127       96G   11G   81G  12% /

Thanks for your help

2

What version of LXD are you using?

3

Does this error still occur if the container is stopped when you export it?

4

@tomp I am using LXD 3.23 (rev 13987).

It tried to export a stopped copy of the container (as I cannot stop it at the moment) and it worked!

Thanks for your help

EDIT: I tried to export a stopped container on which I had the same issue and it also worked:

lxc export hive-s2 hive-s2.tar.gz
Error: Create backup: Backup create: Error adding “/var/snap/lxd/common/lxd/storage-pools/system/containers/hive-s2/rootfs/var/log/ambari-metrics-monitor/ambari-metrics-monitor.out” as “backup/container/rootfs/var/log/ambari-metrics-monitor/ambari-metrics-monitor.out” to tarball: Failed to copy file content “/var/snap/lxd/common/lxd/storage-pools/system/containers/hive-s2/rootfs/var/log/ambari-metrics-monitor/ambari-metrics-monitor.out”: archive/tar: write too long
lxc stop hive-s2
lxc export hive-s2 hive-s2.tar.gz
Backup exported successfully!

It looks like the archiving process is having issues with large files being written.

5

So we’re going to have to do two things here:

  • When possible, make a snapshot and export that
  • Make our exporter as tolerant as tar was in the past (as in, write partial, potentially corrupted files, instead of completely failing)
6

Hi thanks for the update.

We have recently switched to using Go’s internal tar writer rather than the external tar command for creating the export tarballs.

It appears that Go is stricter about detecting a file growing during the write to the tarball whereas the tar command just silently cuts the end of the file off by the looks of it. Not great for a backup.

This has had the end result of failing an export if the tar file would be inconsistent.

Even before this switch, stopping the container before running an export is the best way to get a consistent export file.

However as @stgraber says above we are going to use snapshots where practical to get a consistent export and where not we will emulate tar’s behaviour of silently chopping the end off any files that grow whilst being exported.

7

This PR restores the old behaviour:

8

Thanks @tomp