"ZFS doesn't support restoring from snapshots other than the latest one"

LXD
12

Silly me, I just realised that last year I added a custom code to my restore process which removes subsequent snapshots before the restore process to prevent these kind of issues when using ZFS as at the time volume.zfs.remove_snapshots=true was not supported. It might interesting to also add version limitation to docs when it is written.

As for the error on restoring a snapshot on the ORIGINAL container after running a copy.

Snapshot \"container-test-20210203-02\" cannot be restored due to subsequent internal snapshot(s) (from a copy)",

This is what I sent to the API, as you can see it does not copy snapshots, but now I can’t restore using snapshots on the original container anymore due to error above. This seems to be a problem, anyway round this?

{
    "name": "container-clone",
    "architecture": "aarch64",
    "type": "container",
    "profiles": [
        "custom-default",
        "custom-nat"
    ],
    "config": {
        "image.architecture": "arm64",
        "image.description": "Ubuntu focal arm64 (20210120_07:42)",
        "image.os": "Ubuntu",
        "image.release": "focal",
        "image.serial": "20210120_07:42",
        "image.type": "squashfs",
        "limits.cpu": "1",
        "limits.memory": "1GB",
        "volatile.base_image": "766788f3eb910d209469ccb48109d3236d1bf60897bb2bf52e5d14e12a5a2a3d"
    },
    "source": {
        "type": "copy",
        "certificate": null,
        "base-image": "766788f3eb910d209469ccb48109d3236d1bf60897bb2bf52e5d14e12a5a2a3d",
        "source": "container-test",
        "live": false,
        "instance_only": true
    },
    "devices": {
        "root": {
            "path": "/",
            "pool": "default",
            "size": "5GB",
            "type": "disk"
        }
    },
    "ephemeral": false,
    "stateful": false
}
13

I’d expect you’re hitting a problem because your container copy (even without copying snapshots) is still dependent on the source snapshots because they form a hierarchical tree of changes (i.e source image snapshot -> oldest source snapshot -> newer source snapshot -> source instance volume -> copy of source instance volume).

Using zfs.clone_copy=rebase should help here (but a new copy will need to be made), as it will rebase the copied instance volumes ontop of the original source image rather than the source instance (but this will use more storage space).

So it would become: source image snapshot -> copy of source instance volume.

But with all of the differences between source image snapshot and source instance volume having been copied into the copy of source volume (taking up more space).

14

I ran lxc storage set default zfs.clone_copy false, and I can confirm, if I copy a container, I can now restore snapshots on the original container. Thanks.

Question: if you create a snapshot on a container that is not running, until you boot it up, there is no real snapshot other than a record, on ZFS, is that correct?

15

Great. You may also use zfs.clone_copy=rebase as a somewhat more storage efficient approach than zfs.clone_copy=false that should still allow you to restore the original snapshots, but without having to fully copy the source image volume on each instance copy.

Well technically ZFS will report a snapshot exists, but as there has been no chance of any writes to the original volume it should theoretically be zero in size. But there may be some ZFS internal usage that means there is some usage reported.

16

Does the rebase mode, somewhat rely upon the original container? I mean if I want to use the clone for a backup, I presume false is better?

17

My understanding is that it will create a new volume from the source instance volume, but use the source image volume as the basis of the new volume (i.e only the differences between the source instance volume and the source image volume will be copied to the new volume). But after that the new volume isn’t related to the source instance volume.

18

So it is still kind of connected to the original container then, which sounds like could be more head ache down the line?

19

No not connected to the original container. Only the source image.

Its just at copy time the changes made to the source instance compared to the image are duplicated into the new volume. After that they are not related.

20

Let’s say I create a ubuntu container and then install apache.
Rebase will create a new container using the ubuntu image, and then add the apache difference?
If I set to false, a new image is created with ubuntu+apache combined, which is different to actual process of manually creating.

Is my understanding correct?

21

Yeah, thats right. But keep in mind that the zfs.clone_copy setting also applies to “normal” instance creation as well (from an image), and not just copies of one instance to another.

The normal creation of an instance on a ZFS pool is to “copy” the image volume to a new instance volume. Normally this uses zfs.clone_copy=true which means it just takes a snapshot of the source image to use as the instance volume. With zfs.clone_copy=false the full image volume is duplicated for the new instance volume. With zfs.clone_copy=rebase a snapshot is taken (same as zfs.clone_copy=true), but on subsequent copies of the instance to a new instance, the new instance volume is duplicated based on only the differences between the source instance and the source image.

So zfs.clone_copy=rebase is only different to zfs.clone_copy=true when copying instances to another instance. Whereas zfs.clone_copy=false will affect normal instance creation too.

22

@stgraber does this all sound right to you?

23

hmm. so its now going to change how images are created as well.

I just want to standardise the process, offering the best integrity without having to carry out so many hacks.

So btrfs does not have these issues? If not what setting would be emulate this on ZFS pool which is the recommended one.

:confused:

24

No, zfs.clone_copy=false wont change how images are created. I believe however it will cause a fully copy of the image for new instances (like using a dir pool), but with the benefit that afterwards you can create cheap snapshots.

But this is why I suggested zfs.clone_copy=rebase as it sounded like it would fit your needs.

Have you tried it?

25

Ahh, so zfs.clone_copy=rebase so will make the copy of the container the same as if I created the container manually, rather than bundling all in a new image?

26 
lxc storage set default zfs.clone_copy rebase
Error: Invalid value for a boolean "rebase"
27

Yes. It will copy the differences between the image and the source instance into the new volume.

28

You must be running an older version.

29

I am using the stable version 4.0.5

30

Right so that is an LTS (Long Term Support) release and doesn’t get the newest features, so this will be in LXD 5 or you can switch to the latest/stable channel (But keep in mind you won’t be able to switch back as it will alter your database with latest schema).

See

Alternatively, if your not dead set on ZFS, have you looked at LVM?
In thin pool mode (the default for LXD), it supports efficient snapshots, and restoring from arbitrary snapshots, as well as copying from instances or instance snapshots, without preventing future restores or deletions.

31

I am interested in the best setup for LXD on a production server. I did not look at LVM because the documentation recommends ZFS and btrfs.