Hello,
I have a setup where a script run with crom takes snapshots of all the containers on a host and then another script does cleanup of the snapshots that are older than a specified amount of days. Also I have another host which connects to the first one and takes scheduled offsite copies of the containers. But for some reason the ZFS pool disk usage was getting higher than expected and when investigating further, I found out that one of the containers had snapshots that are not listed when calling “lxc info”.
root@lxd-2-top2:~# lxc info top2db-master
Name: top2db-master
Remote: unix://
Architecture: x86_64
Created: 2017/10/21 16:57 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 22286
Ips:
eth0: inet 10.107.211.20 vethG4157N
eth0: inet6 fe80::216:3eff:fe20:da27 vethG4157N
lo: inet 127.0.0.1
lo: inet6 ::1
Resources:
Processes: 77
Disk usage:
root: 66.77GB
Memory usage:
Memory (current): 935.36MB
Memory (peak): 1.03GB
Network usage:
eth0:
Bytes received: 36.48GB
Bytes sent: 456.65GB
Packets received: 374148287
Packets sent: 304406052
lo:
Bytes received: 8.38GB
Bytes sent: 8.38GB
Packets received: 971994
Packets sent: 971994
Snapshots:
20171021-new-prod (taken at 2017/10/21 18:39 UTC) (stateless)
remove_binlog (taken at 2017/12/17 17:10 UTC) (stateless)
20180921-before-update (taken at 2018/09/20 22:28 UTC) (stateless)
20181021-backup (taken at 2018/10/21 02:00 UTC) (stateless)
20181022-backup (taken at 2018/10/22 02:00 UTC) (stateless)
20181023-backup (taken at 2018/10/23 02:00 UTC) (stateless)
20181023-0400-restorepoint (taken at 2018/10/23 04:00 UTC) (stateless)
20181023-2000-restorepoint (taken at 2018/10/23 20:00 UTC) (stateless)
20181024-backup (taken at 2018/10/24 02:00 UTC) (stateless)
20181024-0400-restorepoint (taken at 2018/10/24 04:00 UTC) (stateless)
20181024-2000-restorepoint (taken at 2018/10/24 20:00 UTC) (stateless)
20181025-backup (taken at 2018/10/25 02:00 UTC) (stateless)
20181025-0400-restorepoint (taken at 2018/10/25 04:00 UTC) (stateless)
20181025-2000-restorepoint (taken at 2018/10/25 20:00 UTC) (stateless)
20181026-backup (taken at 2018/10/26 02:00 UTC) (stateless)
20181026-0400-restorepoint (taken at 2018/10/26 04:00 UTC) (stateless)
20181026-2000-restorepoint (taken at 2018/10/26 20:00 UTC) (stateless)
20181027-backup (taken at 2018/10/27 02:00 UTC) (stateless)
20181027-0400-restorepoint (taken at 2018/10/27 04:00 UTC) (stateless)
20181027-2000-restorepoint (taken at 2018/10/27 20:00 UTC) (stateless)
20181028-backup (taken at 2018/10/28 02:00 UTC) (stateless)
20181028-0400-restorepoint (taken at 2018/10/28 04:00 UTC) (stateless)
20181028-2000-restorepoint (taken at 2018/10/28 20:00 UTC) (stateless)
20181029-backup (taken at 2018/10/29 02:00 UTC) (stateless)
20181029-0400-restorepoint (taken at 2018/10/29 04:00 UTC) (stateless)
20181029-2000-restorepoint (taken at 2018/10/29 20:00 UTC) (stateless)
When listing the snapshots with ZFS I see the unlisted snapshots named “copy-” which obviously are the culprit for the excessive disk usage:
root@lxd-2-top2:~# zfs list -r -t snapshot -o space,creation lxd/containers/top2db-master
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD CREATION
lxd/containers/top2db-master@snapshot-20171021-new-prod - 2.03G - - - - Sat Oct 21 18:39 2017
lxd/containers/top2db-master@copy-0a1a537b-1a89-4817-9b32-2c0a57568606 - 596M - - - - Tue Nov 21 2:00 2017
lxd/containers/top2db-master@copy-493d6e4e-0315-473c-98a6-65d02a3058d5 - 462M - - - - Wed Nov 22 2:00 2017
lxd/containers/top2db-master@copy-f4d87fa9-5bf3-4085-8e9b-b4bdc42e3cf8 - 573M - - - - Thu Nov 23 2:00 2017
lxd/containers/top2db-master@copy-b2dd4f9c-81d4-4eaf-b7ec-c3f2ce69f900 - 1.25G - - - - Fri Dec 1 2:00 2017
lxd/containers/top2db-master@snapshot-remove_binlog - 1.47G - - - - Sun Dec 17 17:10 2017
lxd/containers/top2db-master@copy-deafa6f5-e474-408e-859a-25279e855742 - 1.17G - - - - Wed Jul 25 2:00 2018
lxd/containers/top2db-master@copy-b5fff947-337e-4fb8-a332-596e63ba6a0a - 932M - - - - Tue Jul 31 2:00 2018
lxd/containers/top2db-master@copy-14501791-8209-4e0c-bd65-6611c53c4199 - 979M - - - - Mon Aug 6 2:00 2018
lxd/containers/top2db-master@copy-04de4d76-f14c-4152-9931-9dc6f70e50d9 - 1014M - - - - Sun Aug 12 2:00 2018
lxd/containers/top2db-master@copy-d8ee0ff3-f1af-405a-87c0-fed3aadbdc06 - 990M - - - - Sat Aug 18 2:00 2018
lxd/containers/top2db-master@copy-0b0035e7-5acd-4ab6-8437-75f9fe724df2 - 985M - - - - Fri Aug 24 2:00 2018
lxd/containers/top2db-master@copy-d2343864-e576-4c1e-b332-cbb4bf3c03b2 - 1.04G - - - - Thu Aug 30 2:00 2018
lxd/containers/top2db-master@copy-7472222f-97ba-4b4d-bba3-2cc250cb882e - 979M - - - - Wed Sep 5 2:00 2018
lxd/containers/top2db-master@copy-92a74080-b747-4056-b3d3-6dba3264c0a5 - 926M - - - - Fri Sep 7 2:00 2018
lxd/containers/top2db-master@copy-64991c80-0257-45b3-8bcb-e3f5db315110 - 913M - - - - Tue Sep 11 2:00 2018
lxd/containers/top2db-master@copy-9435a1e8-5f43-49d5-a081-de66cc909520 - 905M - - - - Thu Sep 13 2:00 2018
lxd/containers/top2db-master@copy-0091e490-1642-42a2-b80a-d0eef223d899 - 917M - - - - Mon Sep 17 2:00 2018
lxd/containers/top2db-master@copy-fed07265-874e-43eb-9ae1-32a27861d969 - 930M - - - - Wed Sep 19 2:00 2018
lxd/containers/top2db-master@snapshot-20180921-before-update - 338M - - - - Thu Sep 20 22:28 2018
lxd/containers/top2db-master@copy-d6668fd4-36fd-4091-a36d-0806c2243fd4 - 345M - - - - Fri Sep 21 2:00 2018
lxd/containers/top2db-master@copy-c9e5e09e-5473-478c-b199-ed3050c6f2f8 - 486M - - - - Sat Sep 22 2:00 2018
lxd/containers/top2db-master@copy-10e6e0fc-dd23-4257-b0ca-a9bce9db6d1c - 427M - - - - Sun Sep 23 2:00 2018
lxd/containers/top2db-master@copy-1d8c9108-879f-4466-a170-778adbf05bb2 - 492M - - - - Mon Sep 24 2:00 2018
lxd/containers/top2db-master@copy-ce9fe8b3-4b09-4758-8287-c463c54f8d8d - 853M - - - - Tue Sep 25 2:00 2018
lxd/containers/top2db-master@copy-a9f1dc02-389c-4e5a-a8fa-dc946d7bbac3 - 839M - - - - Wed Sep 26 2:00 2018
lxd/containers/top2db-master@copy-0255e2a9-e0f5-43ec-be46-f7f0ed492bfa - 841M - - - - Thu Sep 27 2:00 2018
lxd/containers/top2db-master@copy-d3020861-8d1c-4ae7-80ca-0c79861d80ee - 840M - - - - Fri Sep 28 2:00 2018
lxd/containers/top2db-master@copy-c07c5e07-f797-4842-9ec2-9c6e2990fda9 - 482M - - - - Sat Sep 29 2:00 2018
lxd/containers/top2db-master@copy-062e4ca8-8ece-4a6b-ac14-3db244ff0482 - 435M - - - - Sun Sep 30 2:00 2018
lxd/containers/top2db-master@copy-07e9e5f6-b236-45a4-abdb-d95edfa53ce8 - 514M - - - - Mon Oct 1 2:00 2018
lxd/containers/top2db-master@copy-c4b831db-aa67-4ec9-94e4-73446eb17a28 - 871M - - - - Tue Oct 2 2:00 2018
lxd/containers/top2db-master@copy-5318b9fd-4201-45f5-a739-df939e115ae5 - 875M - - - - Wed Oct 3 2:00 2018
lxd/containers/top2db-master@copy-aca3dcde-58cd-4521-bc1d-c439a2d33971 - 789M - - - - Thu Oct 4 2:00 2018
lxd/containers/top2db-master@copy-8fdc46d8-aaaa-4fb6-9ea9-6b2561c2e83f - 783M - - - - Fri Oct 5 2:00 2018
lxd/containers/top2db-master@copy-87b5f473-2b35-4086-8ef2-dd3ab8c038b8 - 474M - - - - Sat Oct 6 2:00 2018
lxd/containers/top2db-master@copy-aece6d05-a495-4443-9f26-99e309ac5b3a - 420M - - - - Sun Oct 7 2:00 2018
lxd/containers/top2db-master@copy-dcffe52a-2719-442c-af35-a473ab941a40 - 502M - - - - Mon Oct 8 2:00 2018
lxd/containers/top2db-master@copy-539776c5-7b5a-4086-8400-12a272806042 - 855M - - - - Tue Oct 9 2:00 2018
lxd/containers/top2db-master@copy-2718305d-df26-42a2-ab52-d758bbd59603 - 857M - - - - Wed Oct 10 2:00 2018
lxd/containers/top2db-master@copy-59fe49ae-f4ea-4213-b50f-9d6fad9ff055 - 864M - - - - Thu Oct 11 2:00 2018
lxd/containers/top2db-master@copy-b3763152-a5a7-45bd-bc90-90cddfbc5a6f - 792M - - - - Fri Oct 12 2:00 2018
lxd/containers/top2db-master@copy-dbbc0c81-f0b8-438c-8b8c-995d2a3690cc - 432M - - - - Sat Oct 13 2:00 2018
lxd/containers/top2db-master@copy-d2466db2-8bcb-44a0-bfa9-fd2b7d8b2f9a - 431M - - - - Sun Oct 14 2:00 2018
lxd/containers/top2db-master@copy-e0ec7363-fafc-45e1-bb60-bbfa6511d841 - 502M - - - - Mon Oct 15 2:00 2018
lxd/containers/top2db-master@copy-7ba29723-0160-47bc-adfd-213e35e93f01 - 886M - - - - Tue Oct 16 2:00 2018
lxd/containers/top2db-master@copy-7b66e255-08e1-491a-87f1-b716664accca - 924M - - - - Wed Oct 17 2:00 2018
lxd/containers/top2db-master@copy-4dd09422-c2c0-48ed-8dd2-b622fad9d695 - 894M - - - - Fri Oct 19 2:00 2018
lxd/containers/top2db-master@copy-d873f90f-e7f5-4912-893e-9ea630917435 - 433M - - - - Sat Oct 20 2:00 2018
lxd/containers/top2db-master@snapshot-20181021-backup - 357M - - - - Sun Oct 21 2:00 2018
lxd/containers/top2db-master@snapshot-20181022-backup - 498M - - - - Mon Oct 22 2:00 2018
lxd/containers/top2db-master@snapshot-20181023-backup - 126M - - - - Tue Oct 23 2:00 2018
lxd/containers/top2db-master@snapshot-20181023-0400-restorepoint - 116M - - - - Tue Oct 23 4:00 2018
lxd/containers/top2db-master@snapshot-20181023-2000-restorepoint - 108M - - - - Tue Oct 23 20:00 2018
lxd/containers/top2db-master@snapshot-20181024-backup - 19.9M - - - - Wed Oct 24 2:00 2018
lxd/containers/top2db-master@snapshot-20181024-0400-restorepoint - 110M - - - - Wed Oct 24 4:00 2018
lxd/containers/top2db-master@snapshot-20181024-2000-restorepoint - 37.6M - - - - Wed Oct 24 20:00 2018
lxd/containers/top2db-master@snapshot-20181025-backup - 32.0M - - - - Thu Oct 25 2:00 2018
lxd/containers/top2db-master@snapshot-20181025-0400-restorepoint - 116M - - - - Thu Oct 25 4:00 2018
lxd/containers/top2db-master@snapshot-20181025-2000-restorepoint - 36.2M - - - - Thu Oct 25 20:00 2018
lxd/containers/top2db-master@snapshot-20181026-backup - 30.7M - - - - Fri Oct 26 2:00 2018
lxd/containers/top2db-master@snapshot-20181026-0400-restorepoint - 113M - - - - Fri Oct 26 4:00 2018
lxd/containers/top2db-master@snapshot-20181026-2000-restorepoint - 37.4M - - - - Fri Oct 26 20:00 2018
lxd/containers/top2db-master@snapshot-20181027-backup - 16.4M - - - - Sat Oct 27 2:00 2018
lxd/containers/top2db-master@snapshot-20181027-0400-restorepoint - 26.4M - - - - Sat Oct 27 4:00 2018
lxd/containers/top2db-master@snapshot-20181027-2000-restorepoint - 27.4M - - - - Sat Oct 27 20:00 2018
lxd/containers/top2db-master@snapshot-20181028-backup - 20.4M - - - - Sun Oct 28 2:00 2018
lxd/containers/top2db-master@snapshot-20181028-0400-restorepoint - 21.0M - - - - Sun Oct 28 4:00 2018
lxd/containers/top2db-master@snapshot-20181028-2000-restorepoint - 96.1M - - - - Sun Oct 28 20:00 2018
lxd/containers/top2db-master@snapshot-20181029-backup - 24.6M - - - - Mon Oct 29 2:00 2018
lxd/containers/top2db-master@snapshot-20181029-0400-restorepoint - 56.4M - - - - Mon Oct 29 4:00 2018
lxd/containers/top2db-master@snapshot-20181029-2000-restorepoint - 35.6M - - - - Mon Oct 29 20:00 2018
Any ideas what might be causing this? There are four other containers on the same host but they are not affected by this…