Incremental backups

Hi,
I have two LXD servers running, one production (A) and one used for backups (B). On B I added A as a remote
example.com | https://example.com:4443 | lxd | tls | NO | NO |
Both severs are running:

Name  Version    Rev    Tracking  Publisher   Notes
core  16-2.37.1  6350   stable    canonical✓  core
lxd   3.18       12224  stable    canonical✓  -

and BTRFS is used as a storage backend.

Now, on B I can do something like this copy example.com:cont01/snapshot cont01 which is working fine.
Since a cron is running hourly to take the snapshots on A, there are a few changes but the copy transfers always the full snapshot
How can I transfer the snapshots incrementally? I read about a flag --refresh but this doesn’t seem to work
Error: unknown flag: --refresh
Some containers are > 50GB thus so the snapshots, transferring them to B always takes hours.

Is there a better way to achieve this with the native LXD/BTRFS capabilities?

lxc copy should have a --refresh option which works when copying the entire container again, synchronizing the state of snapshots. LXD 3.18 supports it for sure, but maybe one of your servers is on an older release or has a mix of snap/deb causing this confusion?

You may want to run lxc version

@stgraber thanks your quick reply.

Here is the output from both servers:

Server A:

Name  Version  Rev    Tracking  Publisher   Notes
core  16-2.42  7917   stable    canonical*  core
lxd   3.18     12224  stable    canonical*  -
config:
  core.https_address: '[::]:4443'
  core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 176.xxx.xxx.xxx:4443
  - 10.159.23.1:4443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----

    -----END CERTIFICATE-----
  certificate_fingerprint: **************
  driver: lxc
  driver_version: 3.2.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "false"
    seccomp_listener: "false"
    shiftfs: "false"
    uevent_injection: "false"
    unpriv_fscaps: "true"
  kernel_version: 4.15.0-65-generic
  lxc_features:
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    seccomp_notify: "true"
  project: default
  server: lxd
  server_clustered: false
  server_name: ****-prd
  server_pid: 2268
  server_version: "3.18"
  storage: btrfs
  storage_version: "4.4"

Server B:

snap list
Name  Version    Rev    Tracking  Publisher   Notes
core  16-2.37.1  6350   stable    canonical✓  core
lxd   3.18       12224  stable    canonical✓  -
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- candid_authentication
- candid_config
- candid_config_key
- usb_optional_vendorid
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  certificate_fingerprint: *******
  driver: lxc
  driver_version: 3.0.3
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.15.0-66-generic
  server: lxd
  server_pid: 1868
  server_version: 3.0.3
  storage: btrfs
  storage_version: 4.15.1
  server_clustered: false
  server_name: ***-bckp
  project: ""

Server B shows 3.0.3 which suggests it’s not actually using the snap.
You may have a combination of the deb and snap on that system.
If that’s the case lxd.migrate should be able to move the data over to the snap and remove the remaining deb packages.

Thats odd, I’ve clean installed 18.04.03 which by default installs the lxd deb packages:

dpkg -l | grep lxd
ii  lxd                                   3.0.3-0ubuntu1~18.04.1             amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                            3.0.3-0ubuntu1~18.04.1             amd64        Container hypervisor based on LXC - client

Isn’t snap the preferred way to deploy LXD?
Thank you for pointing this out. I’ll remove the deb packages.

It is the preferred way to install LXD and the only way available on 18.10 and higher, 18.04 still has the deb available though as you’ve noticed :slight_smile: