Lxc snapshot slow

hi all,

i am experiencing slow lxc snapshot creation. a snapshot of a container takes about 10min. How can we debug this?

i think it should not be so slow. It never was before.

What storage backend?

Btrfs

Can you show:

  • lxc storage show default
  • lxc info
  • lxc config show --expanded NAME-OF-CONTAINER

As well, while running a lxc snapshot, show the ps fauxww output of the system.

root@esx:~# lxc storage show lxd
config:
size: 1500GB
source: /var/snap/lxd/common/lxd/disks/lxd.img
description: “”
name: lxd
driver: btrfs
used_by:

  • /1.0/containers/controller
  • /1.0/containers/dcfs01
  • /1.0/containers/ex02
  • /1.0/containers/kms
  • /1.0/containers/kopano
  • /1.0/containers/opsi
  • /1.0/containers/zwave
  • /1.0/profiles/default
    status: Created
    locations:
  • none

root@esx:~# lxc info
config:
core.https_address: ‘[::]:8443’
core.trust_password: true
storage.backups_volume: exports/exports_volume
api_extensions:

  • storage_zfs_remove_snapshots
  • container_host_shutdown_timeout
  • container_stop_priority
  • container_syscall_filtering
  • auth_pki
  • container_last_used_at
  • etag
  • patch
  • usb_devices
  • https_allowed_credentials
  • image_compression_algorithm
  • directory_manipulation
  • container_cpu_time
  • storage_zfs_use_refquota
  • storage_lvm_mount_options
  • network
  • profile_usedby
  • container_push
  • container_exec_recording
  • certificate_update
  • container_exec_signal_handling
  • gpu_devices
  • container_image_properties
  • migration_progress
  • id_map
  • network_firewall_filtering
  • network_routes
  • storage
  • file_delete
  • file_append
  • network_dhcp_expiry
  • storage_lvm_vg_rename
  • storage_lvm_thinpool_rename
  • network_vlan
  • image_create_aliases
  • container_stateless_copy
  • container_only_migration
  • storage_zfs_clone_copy
  • unix_device_rename
  • storage_lvm_use_thinpool
  • storage_rsync_bwlimit
  • network_vxlan_interface
  • storage_btrfs_mount_options
  • entity_description
  • image_force_refresh
  • storage_lvm_lv_resizing
  • id_map_base
  • file_symlinks
  • container_push_target
  • network_vlan_physical
  • storage_images_delete
  • container_edit_metadata
  • container_snapshot_stateful_migration
  • storage_driver_ceph
  • storage_ceph_user_name
  • resource_limits
  • storage_volatile_initial_source
  • storage_ceph_force_osd_reuse
  • storage_block_filesystem_btrfs
  • resources
  • kernel_limits
  • storage_api_volume_rename
  • macaroon_authentication
  • network_sriov
  • console
  • restrict_devlxd
  • migration_pre_copy
  • infiniband
  • maas_network
  • devlxd_events
  • proxy
  • network_dhcp_gateway
  • file_get_symlink
  • network_leases
  • unix_device_hotplug
  • storage_api_local_volume_handling
  • operation_description
  • clustering
  • event_lifecycle
  • storage_api_remote_volume_handling
  • nvidia_runtime
  • container_mount_propagation
  • container_backup
  • devlxd_images
  • container_local_cross_pool_handling
  • proxy_unix
  • proxy_udp
  • clustering_join
  • proxy_tcp_udp_multi_port_handling
  • network_state
  • proxy_unix_dac_properties
  • container_protection_delete
  • unix_priv_drop
  • pprof_http
  • proxy_haproxy_protocol
  • network_hwaddr
  • proxy_nat
  • network_nat_order
  • container_full
  • candid_authentication
  • backup_compression
  • candid_config
  • nvidia_runtime_config
  • storage_api_volume_snapshots
  • storage_unmapped
  • projects
  • candid_config_key
  • network_vxlan_ttl
  • container_incremental_copy
  • usb_optional_vendorid
  • snapshot_scheduling
  • container_copy_project
  • clustering_server_address
  • clustering_image_replication
  • container_protection_shift
  • snapshot_expiry
  • container_backup_override_pool
  • snapshot_expiry_creation
  • network_leases_location
  • resources_cpu_socket
  • resources_gpu
  • resources_numa
  • kernel_features
  • id_map_current
  • event_location
  • storage_api_remote_volume_snapshots
  • network_nat_address
  • container_nic_routes
  • rbac
  • cluster_internal_copy
  • seccomp_notify
  • lxc_features
  • container_nic_ipvlan
  • network_vlan_sriov
  • storage_cephfs
  • container_nic_ipfilter
  • resources_v2
  • container_exec_user_group_cwd
  • container_syscall_intercept
  • container_disk_shift
  • storage_shifted
  • resources_infiniband
  • daemon_storage
  • instances
  • image_types
  • resources_disk_sata
  • clustering_roles
  • images_expiry
  • resources_network_firmware
  • backup_compression_algorithm
  • ceph_data_pool_name
  • container_syscall_intercept_mount
  • compression_squashfs
  • container_raw_mount
  • container_nic_routed
  • container_syscall_intercept_mount_fuse
  • container_disk_ceph
  • virtual-machines
  • image_profiles
  • clustering_architecture
  • resources_disk_id
  • storage_lvm_stripes
  • vm_boot_priority
  • unix_hotplug_devices
  • api_filtering
    api_status: stable
    api_version: “1.0”
    auth: trusted
    public: false
    auth_methods:
  • tls
    environment:
    addresses:
    • 192.168.1.1:8443
    • 192.168.2.3:8443
      architectures:
    • x86_64
    • i686
      certificate: |
      -----BEGIN CERTIFICATE-----
      MIICBjCCAYygAwIBAgIQQ4GoWmN57sFP6bJYliUoJzAKBggqhkjOPQQDAzAxMRww
      GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMREwDwYDVQQDDAhyb290QGVzeDAe
      Fw0xOTA0MzAxOTAyMzFaFw0yOTA0MjcxOTAyMzFaMDExHDAaBgNVBAoTE2xpbnV4
      Y29udGFpbmVycy5vcmcxETAPBgNVBAMMCHJvb3RAZXN4MHYwEAYHKoZIzj0CAQYF
      K4EEACIDYgAEUPP6fBEartgMZQGUN9SabrEsCKM/keP7o98/+Mmihd7N02oLmdW9
      MmdWZVc0bDkIOTtxYmb+OAgY9aHP1Hgl7s7obr0YxnMcDDRUQAW2GWxwW+HxsTiR
      3Oy7UWTYPgUoo2kwZzAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
      AwEwDAYDVR0TAQH/BAIwADAyBgNVHREEKzApggNlc3iHBMCoAQGHECABCYFOfwAB
      agXK//4/0SaHBMCoAgOHBMCoegEwCgYIKoZIzj0EAwMDaAAwZQIwDVuX57oFGHrK
      TXTMX2q/erwb1Onc88+50DLaUxax0Joz9/BOWmowQAEOT/PJVXkbAjEAzmwhMNde
      jB2OLeGmavQ6KQ4fvPrzqjikT2osDjraPcSVSshxi6Rok/CTgcJ8QyWI
      -----END CERTIFICATE-----
      certificate_fingerprint: 3ee2a792dea3e4d3ecb29432c684a685c7aae7243300110de4ddfc947a71ed29
      driver: lxc
      driver_version: 3.2.1
      kernel: Linux
      kernel_architecture: x86_64
      kernel_features:
      netnsid_getifaddrs: “false”
      seccomp_listener: “false”
      seccomp_listener_continue: “false”
      shiftfs: “false”
      uevent_injection: “true”
      unpriv_fscaps: “true”
      kernel_version: 4.19.0-8-amd64
      lxc_features:
      cgroup2: “false”
      mount_injection_file: “true”
      network_gateway_device_route: “true”
      network_ipvlan: “true”
      network_l2proxy: “true”
      network_phys_macvlan_mtu: “true”
      network_veth_router: “true”
      seccomp_notify: “true”
      project: default
      server: lxd
      server_clustered: false
      server_name: esx
      server_pid: 1653
      server_version: “3.20”
      storage: btrfs
      storage_version: “4.4”

root@esx:~# lxc config show --expanded kopano
architecture: x86_64
config:
boot.autostart: “1”
environment.TZ: Europe/Amsterdam
image.architecture: amd64
image.description: Debian stretch amd64 (20190430_05:24)
image.os: Debian
image.release: stretch
image.serial: “20190430_05:24”
limits.disk.priority: “10”
limits.memory.swap: “false”
volatile.base_image: 0c0eedb46b6592ae53a4fd0c7efff666233b758429e62ab6489f5c009503e4f3
volatile.eth0.host_name: vethe158e61f
volatile.eth0.hwaddr: 00:16:3e:dd:bc:d8
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.power: RUNNING
devices:
eth0:
name: eth0
nictype: bridged
parent: bridge1
type: nic
root:
path: /
pool: lxd
type: disk
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”

1000000 3564 0.0 0.0 5384 940 ? Ss+ Feb10 0:00 _ /sbin/agetty -o -p – \u --noclear --keep-baud console 115200,38400,9600 linux
1000000 3595 0.0 0.0 2800 128 ? Ss Feb10 0:00 _ /usr/sbin/in.tftpd -v --ipv4 --listen --address :69 --secure /tftpboot/
1000000 3732 0.0 0.0 15852 2452 ? Ss Feb10 0:00 _ /usr/sbin/sshd -D
1000000 3959 0.0 0.0 8436 2988 ? Ss Feb10 0:03 _ /usr/sbin/apache2 -k start
1000033 10347 0.0 0.0 1999788 8724 ? Sl 00:00 0:00 | _ /usr/sbin/apache2 -k start
1000033 10348 0.0 0.0 1999788 10764 ? Sl 00:00 0:00 | _ /usr/sbin/apache2 -k start
1000107 4123 0.0 0.4 1778860 72044 ? Ssl Feb10 1:16 _ /usr/sbin/mysqld
1000000 4321 0.0 0.0 49356 6820 ? Ss Feb10 0:00 _ /usr/sbin/smbd --foreground --no-process-group
1000000 4452 0.0 0.0 46668 3528 ? S Feb10 0:00 | _ /usr/sbin/smbd --foreground --no-process-group
1000000 4453 0.0 0.0 46660 2560 ? S Feb10 0:00 | _ /usr/sbin/smbd --foreground --no-process-group
1000000 4494 0.0 0.0 49356 4228 ? S Feb10 0:00 | _ /usr/sbin/smbd --foreground --no-process-group
1000000 5820 0.0 0.1 212636 26040 ? Sl Feb10 0:51 _ /usr/bin/python /usr/bin/opsipxeconfd start
1000993 5856 0.0 0.2 142428 47332 ? S Feb10 0:01 _ /usr/bin/python /usr/bin/opsiconfd -D
root 2332 0.0 0.0 372044 16036 ? Ss Feb10 0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers zwave
1000000 2349 0.0 0.0 22980 4868 ? Ss Feb10 0:00 _ /sbin/init
1000000 2875 0.0 0.0 29680 6944 ? Ss Feb10 0:01 _ /lib/systemd/systemd-journald
1000000 3010 0.0 0.0 19548 1688 ? Ss Feb10 0:00 _ /lib/systemd/systemd-udevd
1000102 3037 0.0 0.0 20940 1500 ? Ss Feb10 0:00 _ /lib/systemd/systemd-networkd
1000103 3151 0.0 0.0 27444 1184 ? Ss Feb10 0:00 _ /lib/systemd/systemd-resolved
1000000 3401 0.0 0.0 8476 728 ? Ss Feb10 0:00 _ /usr/sbin/cron -f
1000105 3403 0.0 0.0 8972 1804 ? Ss Feb10 0:00 _ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
1000000 3414 0.0 0.0 19504 2168 ? Ss Feb10 0:00 _ /lib/systemd/systemd-logind
1000000 3521 0.4 0.7 4702268 117828 ? Ssl Feb10 6:15 _ /srv/homeassistant/bin/python3 /srv/homeassistant/bin/hass -c /home/homeassistant/.homeassistant
1000107 3522 0.0 0.0 15352 2068 ? Ss Feb10 0:28 _ /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
1000104 3543 0.0 0.0 6432 1520 ? Ss Feb10 0:00 _ /usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f
1000000 3567 0.0 0.0 5384 456 ? Ss+ Feb10 0:00 _ /sbin/agetty -o -p – \u --noclear --keep-baud console 115200,38400,9600 linux
1000000 3705 0.0 0.0 15852 2248 ? Ss Feb10 0:00 _ /usr/sbin/sshd -D
root 16723 0.0 0.0 21548 8156 ? Ss 17:28 0:00 /lib/systemd/systemd --user
root 16724 0.0 0.0 172320 3448 ? S 17:28 0:00 _ (sd-pam)

deleting snapshot takes:

root@esx:~# time lxc delete kopano/kopano

real 12m49.055s
user 0m0.016s
sys 0m0.225s

when creating a snapshot is it copying all data somewhere?

so if a container is 40gb does it copy 40gb somewhere?

Try to remove this one, it has already given you grief and it seems the only original feature in your config apart of the bridging (and this one seems mostly unrelated at first glance). For the record I use 3.20 with btrfs and snapshots are very fast (no change). And yes, there is also something else uncommon: a 1500 GB file for storage is rather big. For this size I’d expect to see a partition rather than a file, but I don’t have experience with huge sparse files, maybe it’s innocuous I don’t know.

Thanks for the reply

I think i will reinstall my system and will use a block device for storage (lxd) maybe this will make things faster again.

doing lxc config unset storage.backups_volume is probably a bit faster.

Yes i know. However i installed lxd and it now isus loop device. I heard this should be a device (block device) for better performance?

I have done this. I want my container exports to be written to external hdd. How can j perform this without first writing everything to the root?

wait, does removing the option (and possibly rebooting) change anything in the snapshot performance ?

About your question, I have never used this option myself, but as I understand it, it does not change where your final export is stored, it is only changing the place where temporary data is stored. So not using it does not stop you to export the final product wherever you want, except if your containers are too big for your root size.

No speeds is the same

real 15m58.682s user 0m0.025s sys 0m0.561s root@esx:~#

For creating snapshot

Can you create a small test container and create a snapshot on it ?
if the problem is similar, can you try to remove the snapshot, stop this test container and create a btrfs snapshot outside of LXD (I think that I had given you steps in a previous thread for the btrfs snapshot), is it slow too ?

I got one question. Say if I reinstall my debian system and i use zfs filesystem for everything (4 x ssd in raid 10) but with zfs would this increase performance?

Also now i use btrfs. If i export all my containers can i import them if i use zfs?

For my info,

If I start from scratch. I have 4 ssds of 1tb each.

-i want a raid 10

How would you partition your hdds. And would you create a zfs raid and install ontop of that the os?

What are your recommendations?