I suspect you need to set the ‘volume.size’ setting temporarily on the storage pool config to allow the image volume created to accommodate the image unpack.
Can you try that and see if it helps.
So from what I understand the container that they are trying to restore into has a 10GB size (the 100GB is not being set), but the backup is much larger than the 10GB.
OK makes sense. This is because lxd first creates a so called ‘image volume’ from the image being used, and from there creates the container volume as a cheap snapshot of the image volume.
Because ceph is a block oriented driver, all volumes must have a fixed size and cannot be unlimited. As such the initial image volume is created as volume.size or 10gb if not set.
I am also wondering why you created an image publish export rather than an instance backup export (‘lxc export …’) which would have then allowed you to import the backup file without creating an intermediate image volume.
Makes sense as the core fault, yeah. Is there no way to determine this default a bit more dynamically or at least allow the profile specification to override the default, to prevent this scenario?
As to why I didn’t do the backup export, I did the backup export/import first. That also failed in the same way. I was trying this method out of desperation to get something to work
OK let’s go back to your original problem, and please in all cases provide full command and error examples, along with the debug log output you did earlier.
Are you referring to the original failure in importing the container? I’ll have to re-export it right quick. One moment
Yes without using the image publish export.
Okay, export in progress using --instance-only and --compression none
Same result 
Log output:
t=2021-02-15T15:08:35-0500 lvl=dbug msg="Reading backup file info"
t=2021-02-15T15:08:35-0500 lvl=dbug msg="Backup file info loaded" backend=zfs name=target optimized=false pool=ceph_pool project=default snapshots=[] type=container
t=2021-02-15T15:08:35-0500 lvl=dbug msg="New task Operation: 017ff9d6-6fd0-4f25-af7c-1898b01b91d2"
t=2021-02-15T15:08:35-0500 lvl=dbug msg="Started task operation: 017ff9d6-6fd0-4f25-af7c-1898b01b91d2"
t=2021-02-15T15:08:35-0500 lvl=dbug msg="CreateInstanceFromBackup started" driver=ceph instance=target optimizedStorage=false pool=ceph_pool project=default snapshots=[]
t=2021-02-15T15:08:35-0500 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0/events username=atrius
t=2021-02-15T15:08:35-0500 lvl=dbug msg="New event listener: 836f569d-bf72-4fdb-a6f6-d057412a9bd1"
t=2021-02-15T15:08:35-0500 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url=/1.0/operations/017ff9d6-6fd0-4f25-af7c-1898b01b91d2 username=atrius
t=2021-02-15T15:08:35-0500 lvl=dbug msg="Activated RBD volume" dev=/dev/rbd4 driver=ceph pool=ceph_pool vol=container_target
t=2021-02-15T15:08:36-0500 lvl=dbug msg="Mounted RBD volume" dev=/dev/rbd4 driver=ceph options=discard path=/var/snap/lxd/common/lxd/storage-pools/ceph_pool/containers/target pool=ceph_pool
t=2021-02-15T15:08:36-0500 lvl=dbug msg="Unmounted RBD volume" driver=ceph keepBlockDev=false path=/var/snap/lxd/common/lxd/storage-pools/ceph_pool/containers/target pool=ceph_pool
t=2021-02-15T15:08:37-0500 lvl=dbug msg="Deactivated RBD volume" driver=ceph pool=ceph_pool vol=container_target
t=2021-02-15T15:08:37-0500 lvl=dbug msg="Activated RBD volume" dev=/dev/rbd4 driver=ceph pool=ceph_pool vol=container_target
t=2021-02-15T15:08:37-0500 lvl=dbug msg="Mounted RBD volume" dev=/dev/rbd4 driver=ceph options=discard path=/var/snap/lxd/common/lxd/storage-pools/ceph_pool/containers/target pool=ceph_pool
t=2021-02-15T15:08:37-0500 lvl=dbug msg="Unpacking container filesystem volume" args="[-xf - --xattrs-include=* -C /var/snap/lxd/common/lxd/storage-pools/ceph_pool/containers/target --strip-components=2 backup/container]" driver=ceph pool=ceph_pool source=backup/container target=/var/snap/lxd/common/lxd/storage-pools/ceph_pool/containers/target
t=2021-02-15T15:10:57-0500 lvl=dbug msg="Unmounted RBD volume" driver=ceph keepBlockDev=false path=/var/snap/lxd/common/lxd/storage-pools/ceph_pool/containers/target pool=ceph_pool
t=2021-02-15T15:10:57-0500 lvl=dbug msg="Deactivated RBD volume" driver=ceph pool=ceph_pool vol=container_target
t=2021-02-15T15:10:59-0500 lvl=dbug msg="CreateInstanceFromBackup finished" driver=ceph instance=target optimizedStorage=false pool=ceph_pool project=default snapshots=[]
t=2021-02-15T15:11:24-0500 lvl=dbug msg="Event listener finished: 836f569d-bf72-4fdb-a6f6-d057412a9bd1"
t=2021-02-15T15:11:24-0500 lvl=dbug msg="Disconnected event listener: 836f569d-bf72-4fdb-a6f6-d057412a9bd1"
Command to export: lxc export --compression none --instance-only target target.tar
Command to import: lxc import holding/target.tar target -s ceph_pool
Error was exactly the same as during the image launch process
Well not exactly the same, its failing unpacking the tarball into containers volume and not the image volume, so now I have a scenario I can look to reproduce and confirm a bug.
Interestingly there’s also a mention of zfs in those logs, are either of your pools zfs?
And out interest did setting volume.size on the import pool to a larger size fix it?
The source system uses ZFS, so yeah. The target system also has a local pool that’s ZFS. What path in the system would it be using for that unpacking process and on which system would it be doing so? The last part is important as it’s a cluster and not all of them as much space outside of Ceph as the one I’m actually running the import command on.
Let me test that, though I’m reasonably sure it will fix it
Set it to 50GB, the source system is only 30GB, and redoing the import
Okay, not overly surprisingly, setting the size like that did allow the import to run
Thanks, thats useful info, I’ll take a look at recreating this.
OK so I have recreated this. And it occurs even if the container that was exported has its own disk device in its config (i.e not from the profile).
The issue is that container export files have to be unpacked (into their target storage volume) before we can access the backup.yaml file that contains the config of the container (e.g. the disk device size property or which profiles it belongs to).
So this becomes a chicken & egg situation where we need to create the volume before we know what size it should be. This also becomes more complicated with “optimized” exports, where the container’s filesystem is stored as a binary blob that is specific to the storage driver.
The situation is better for VM imports, because we store the VM’s disk as an image file in the tarball, and so when we get to unpacking it, we can directly compare its size with the target volume and enlarge the target volume if needed (which we do).
With standard filesystem imports we would in theory need to read through the entire tarball adding up all the file sizes from the meta data and then resize the volume, and then read through the entire tarball again to actually extract the files onto the volume.
Either that or we need to start storing something in the top-level index.yaml file (which is stored in the tarball first and is quick to access) so we can just read that file and recreate the volume at the correct size.
There is also some provision to do this, as the index.yaml file structure has a config field which could be used to store an embedded copy of the instance’s backup.yaml file, but which is currently only used to store info about custom volume exports.
@stgraber could this be something for the roadmap perhaps?
Yeah, probably. I suspect we’d want to use index.yaml for that, it’s pretty similar to the migration case where we similarly send the instance volume size out of band so we can initialize it prior to receiving the data.
2 Likes
We’d probably want it to fail in cases where no size property is set as part of the instance config and where the receiver’s volume.size is smaller.
That’s basically so we don’t offer a way for users in restricted projects to bypass quotas.
Yes for “legacy” tarballs, they should have the same restrictions as today.