# btrfs subvolume list /var/snap/lxd/common/lxd | grep /container/
Shows many more snapshots than lxc info
So this issue isnāt related to any orphaned on disk snapshots, its only concerned with the difference between instances_snapshots and storage_volumes tables
Indeed, got it working even on 4.2, thanks a lot everyone!
1 Like
lxc storage show default
Is quite long. FWIW, the ones I did check that have this issue the counts were off and they are fairly old containers Iāve had for a while that are set to daily snapshot with 30 day retention.
Yes letās see it please
Much obliged thank you
This worked, but it shut down all the containers and started them back up in the process. It also took like 10 minutes or more.
So definitely not something you want to do if you canāt have any downtime.
Sadly not sure it would be of any value now that I downgraded to 5.1. Let me know if itās still useful.
I just want to see what the source property is as it shouldnt be trying to use /var/snap/lxd/common/shmounts for your storage pool mount, which could indicate a problem with the snap package.
osgeo7 is the name of the zpool
zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
osgeo7 14.5T 2.68T 11.8T - - 27% 18% 1.00x ONLINE -
lxc storage show default
config:
source: osgeo7
volatile.initial_source: osgeo7
zfs.pool_name: osgeo7
description: ""
name: default
driver: zfs
used_by:
I just noticed that if I run that with --debug.
I see a bunch of MISSING like these
"/1.0/instances/nginx",
"/1.0/instances/nginx%!F(MISSING)after-adding-rewrite-web-config",
"/1.0/instances/nginx%!F(MISSING)after-changing-ports",
"/1.0/instances/nginx%!F(MISSING)after-ssl-bottle",
"/1.0/instances/nginx%!F(MISSING)after-wiki-id-fdo-move",
"/1.0/instances/nginx%!F(MISSING)before-proxypass-changes",
"/1.0/instances/nginx%!F(MISSING)nginx-20181222",
"/1.0/instances/nginx%!F(MISSING)nxginx-20181225",
"/1.0/instances/nginx%!F(MISSING)snap0",
"/1.0/instances/nginx%!F(MISSING)snap14",
"/1.0/instances/nginx%!F(MISSING)snap15",
"/1.0/instances/nginx%!F(MISSING)snap17",
"/1.0/instances/nginx%!F(MISSING)snap2",
"/1.0/instances/nginx%!F(MISSING)snap3",
"/1.0/instances/nginx%!F(MISSING)snap5",
What does that MISSING mean? nginx is one of my containers that had the issue I recall.
But if I do
lxc info nginx
I do see those missing ones listed
What does lxc info nginx show?
Ah OK so this issue with creating the snapshot initially due to a problem mount is likely due to a separate long standing bug in Snapd and ZFS see https://github.com/lxc/lxd-pkg-snap/issues/61
Snapshots:
+---------------------------------+----------------------+----------------------+----------+
| NAME | TAKEN AT | EXPIRES AT | STATEFUL |
+---------------------------------+----------------------+----------------------+----------+
| after-adding-rewrite-web-config | 2019/04/30 01:41 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| after-changing-ports | 2019/04/19 23:05 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| after-gitea.osgeo.org | 2022/05/15 18:37 UTC | 2022/06/14 18:37 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| after-ssl-bottle | 2019/03/27 23:07 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| after-wiki-id-fdo-move | 2019/04/24 11:03 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| before-proxypass-changes | 2019/04/21 06:07 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| nginx-20181222 | 2018/12/23 05:05 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| nxginx-20181225 | 2018/12/25 06:07 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap0 | 2019/02/06 06:43 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap2 | 2019/03/24 00:10 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap3 | 2019/03/30 05:52 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap5 | 2019/04/18 09:39 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap14 | 2019/04/28 18:30 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap15 | 2019/04/30 18:30 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap17 | 2019/05/02 18:30 UTC | | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap660 | 2022/05/05 18:29 UTC | 2022/06/04 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap661 | 2022/05/07 18:29 UTC | 2022/06/06 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap662 | 2022/05/09 18:29 UTC | 2022/06/08 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap663 | 2022/05/11 18:29 UTC | 2022/06/10 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap664 | 2022/05/13 18:29 UTC | 2022/06/12 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap665 | 2022/05/15 18:29 UTC | 2022/06/14 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap666 | 2022/05/15 18:37 UTC | 2022/06/14 18:37 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap667 | 2022/05/17 18:29 UTC | 2022/06/16 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap668 | 2022/05/19 18:29 UTC | 2022/06/18 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap669 | 2022/05/21 18:29 UTC | 2022/06/20 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap670 | 2022/05/23 18:29 UTC | 2022/06/22 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap671 | 2022/05/25 18:29 UTC | 2022/06/24 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap672 | 2022/05/27 18:29 UTC | 2022/06/26 18:29 UTC | NO |
+---------------------------------+----------------------+----------------------+----------+
| snap673 | 2022/05/29 18:29 UTC | 2022/06/28 18:29 UTC | NO |
Which command is that for? Can you show the full output plus the command being run?
Can you also show output of lxc storage volume ls default please
I think I had tried lxc delete instance/snapshot (thinking that would do it) but it gave me the same error about them being inconsistent when I tried to delete. Does that error get triggered on any action? Unfortunately Iāve downgraded so canāt test that anymore to absolutely confirm. Maybe someone else can test and confirm that is still on 5.2 and has the issue
The list is huge. Iāll put just for the nginx one.
lxc storage volume ls default | grep nginx
| container | nginx | | filesystem | 1 |
| container (snapshot) | nginx/after-adding-rewrite-web-config | | filesystem | 1 |
| container (snapshot) | nginx/after-changing-ports | | filesystem | 1 |
| container (snapshot) | nginx/after-gitea.osgeo.org | | filesystem | 1 |
| container (snapshot) | nginx/after-ssl-bottle | | filesystem | 1 |
| container (snapshot) | nginx/after-wiki-id-fdo-move | | filesystem | 1 |
| container (snapshot) | nginx/before-proxypass-changes | | filesystem | 1 |
| container (snapshot) | nginx/for-osgeo4 | | filesystem | 1 |
| container (snapshot) | nginx/nginx-20181222 | | filesystem | 1 |
| container (snapshot) | nginx/nxginx-20181225 | | filesystem | 1 |
| container (snapshot) | nginx/snap0 | | filesystem | 1 |
| container (snapshot) | nginx/snap2 | | filesystem | 1 |
| container (snapshot) | nginx/snap3 | | filesystem | 1 |
| container (snapshot) | nginx/snap5 | | filesystem | 1 |
| container (snapshot) | nginx/snap14 | | filesystem | 1 |
| container (snapshot) | nginx/snap15 | | filesystem | 1 |
| container (snapshot) | nginx/snap17 | | filesystem | 1 |
| container (snapshot) | nginx/snap660 | | filesystem | 1 |
| container (snapshot) | nginx/snap661 | | filesystem | 1 |
| container (snapshot) | nginx/snap662 | | filesystem | 1 |
| container (snapshot) | nginx/snap663 | | filesystem | 1 |
| container (snapshot) | nginx/snap664 | | filesystem | 1 |
| container (snapshot) | nginx/snap665 | | filesystem | 1 |
| container (snapshot) | nginx/snap666 | | filesystem | 1 |
| container (snapshot) | nginx/snap667 | | filesystem | 1 |
| container (snapshot) | nginx/snap668 | | filesystem | 1 |
| container (snapshot) | nginx/snap669 | | filesystem | 1 |
| container (snapshot) | nginx/snap670 | | filesystem | 1 |
| container (snapshot) | nginx/snap671 | | filesystem | 1 |
| container (snapshot) | nginx/snap672 | | filesystem | 1 |
| container (snapshot) | nginx/snap673 | | filesystem | 1 |
| container (snapshot) | nginx/snap674 | | filesystem | 1 |
The nginx still has the mismatch, even though its not complaining anymore in 5.1, for reference, here are the queries against the sqlite db.
lxd sql global "SELECT vs.* FROM storage_volumes AS v INNER JOIN storage_volumes_snapshots AS vs ON v.id = vs.storage_volume_id WHERE v.name = 'nginx'"
+-------+-------------------+-----------------------+-------------+----------------------+
| id | storage_volume_id | name | description | expiry_date |
+-------+-------------------+-----------------------+-------------+----------------------+
| 20821 | 1 | after-gitea.osgeo.org | | 0001-01-01T00:00:00Z |
| 21257 | 1 | for-osgeo4 | | 0001-01-01T00:00:00Z |
| 20609 | 1 | snap660 | | 0001-01-01T00:00:00Z |
| 20654 | 1 | snap661 | | 0001-01-01T00:00:00Z |
| 20694 | 1 | snap662 | | 0001-01-01T00:00:00Z |
| 20733 | 1 | snap663 | | 0001-01-01T00:00:00Z |
| 20774 | 1 | snap664 | | 0001-01-01T00:00:00Z |
| 20818 | 1 | snap665 | | 0001-01-01T00:00:00Z |
| 20820 | 1 | snap666 | | 0001-01-01T00:00:00Z |
| 20859 | 1 | snap667 | | 0001-01-01T00:00:00Z |
| 20898 | 1 | snap668 | | 0001-01-01T00:00:00Z |
| 20942 | 1 | snap669 | | 0001-01-01T00:00:00Z |
| 20981 | 1 | snap670 | | 0001-01-01T00:00:00Z |
| 21020 | 1 | snap671 | | 0001-01-01T00:00:00Z |
| 21059 | 1 | snap672 | | 0001-01-01T00:00:00Z |
| 21103 | 1 | snap673 | | 0001-01-01T00:00:00Z |
| 21252 | 1 | snap674 | | 0001-01-01T00:00:00Z |
+-------+-------------------+-----------------------+-------------+----------------------+
lxd sql global "SELECT vs.* FROM instances AS v INNER JOIN instances_snapshots AS vs ON v.id = vs.instance_id WHERE v.name = 'nginx'"
+-------+-------------+---------------------------------+--------------------------------+----------+-------------+--------------------------------+
| id | instance_id | name | creation_date | stateful | description | expiry_date |
+-------+-------------+---------------------------------+--------------------------------+----------+-------------+--------------------------------+
| 138 | 1 | after-adding-rewrite-web-config | 2019-04-30T01:41:46.396381886Z | 0 | | 0001-01-01T00:00:00Z |
| 68 | 1 | after-changing-ports | 2019-04-19T23:05:35.604574266Z | 0 | | 0001-01-01T00:00:00Z |
| 19747 | 1 | after-gitea.osgeo.org | 2022-05-15T18:37:18.258357185Z | 0 | | 2022-06-14T18:37:18.248523458Z |
| 42 | 1 | after-ssl-bottle | 2019-03-27T23:07:09Z | 0 | | 0001-01-01T00:00:00Z |
| 91 | 1 | after-wiki-id-fdo-move | 2019-04-24T11:03:27.901736545Z | 0 | | 0001-01-01T00:00:00Z |
| 80 | 1 | before-proxypass-changes | 2019-04-21T06:07:55.978642091Z | 0 | | 0001-01-01T00:00:00Z |
| 20176 | 1 | for-osgeo4 | 2022-06-03T01:25:30.210702387Z | 0 | | 2022-07-03T01:25:30.197398914Z |
| 1 | 1 | nginx-20181222 | 2018-12-23T05:05:49Z | 0 | | 0001-01-01T00:00:00Z |
| 7 | 1 | nxginx-20181225 | 2018-12-25T06:07:57Z | 0 | | 0001-01-01T00:00:00Z |
| 26 | 1 | snap0 | 2019-02-06T06:43:09Z | 0 | | 0001-01-01T00:00:00Z |
| 130 | 1 | snap14 | 2019-04-28T18:30:27.7471812Z | 0 | | 0001-01-01T00:00:00Z |
| 141 | 1 | snap15 | 2019-04-30T18:30:35.265448386Z | 0 | | 0001-01-01T00:00:00Z |
| 148 | 1 | snap17 | 2019-05-02T18:30:47.115998706Z | 0 | | <nil> |
| 37 | 1 | snap2 | 2019-03-24T00:10:30Z | 0 | | 0001-01-01T00:00:00Z |
| 45 | 1 | snap3 | 2019-03-30T05:52:58Z | 0 | | 0001-01-01T00:00:00Z |
| 67 | 1 | snap5 | 2019-04-18T09:39:28.747191035Z | 0 | | 0001-01-01T00:00:00Z |
| 19535 | 1 | snap660 | 2022-05-05T18:29:27.983428Z | 0 | | 2022-06-04T18:29:27.982453535Z |
| 19580 | 1 | snap661 | 2022-05-07T18:29:22.658087232Z | 0 | | 2022-06-06T18:29:22.657113499Z |
| 19620 | 1 | snap662 | 2022-05-09T18:29:13.061714891Z | 0 | | 2022-06-08T18:29:13.060805288Z |
| 19659 | 1 | snap663 | 2022-05-11T18:29:56.693493675Z | 0 | | 2022-06-10T18:29:56.692473837Z |
| 19700 | 1 | snap664 | 2022-05-13T18:29:57.798111455Z | 0 | | 2022-06-12T18:29:57.797078934Z |
| 19744 | 1 | snap665 | 2022-05-15T18:29:46.362007621Z | 0 | | 2022-06-14T18:29:46.360959559Z |
| 19746 | 1 | snap666 | 2022-05-15T18:37:06.173887612Z | 0 | | 2022-06-14T18:37:06.160366871Z |
| 19785 | 1 | snap667 | 2022-05-17T18:29:30.873067769Z | 0 | | 2022-06-16T18:29:30.872240567Z |
| 19824 | 1 | snap668 | 2022-05-19T18:29:40.146163875Z | 0 | | 2022-06-18T18:29:40.145105714Z |
| 19868 | 1 | snap669 | 2022-05-21T18:29:36.965566274Z | 0 | | 2022-06-20T18:29:36.964610095Z |
| 19907 | 1 | snap670 | 2022-05-23T18:29:40.522181264Z | 0 | | 2022-06-22T18:29:40.520943342Z |
| 19946 | 1 | snap671 | 2022-05-25T18:29:42.809260928Z | 0 | | 2022-06-24T18:29:42.808498747Z |
| 19985 | 1 | snap672 | 2022-05-27T18:29:48.913175874Z | 0 | | 2022-06-26T18:29:48.912275577Z |
| 20029 | 1 | snap673 | 2022-05-29T18:29:45.166063596Z | 0 | | 2022-06-28T18:29:45.165208221Z |
| 20171 | 1 | snap674 | 2022-06-02T18:44:29.459936173Z | 0 | | 2022-07-02T18:44:29.445694928Z |
+-------+-------------+---------------------------------+--------------------------------+----------+-------------+--------------------------------+
So it does seem that all the missing ones are (missing from storage_volume_snapshots) are snapshots in 2019 or earlier. Then again those are the only ones with no expiry information.
I tried to launch one to see if it is still restorable, but all those were at a time when I had localhost set for connect which is no longer supported.
Although:
lxc config show nginx/snap17
Lists all the devices at the time,
lxc config edit nginx/snap17
Doesnāt show them so I can edit. Is there another way to edit the config?
Otherwise when I try to cp with
lxc cp nginx/snap17 nginx-test
I just get a
Error: Failed creating instance record: Failed initialising instance: Invalid devices: Device validation failed for "proxyssh22": Invalid value for device option "connect": Not an IP address "localhost"
Which is what I expect given I have below in the lxc config show nginx/snap17
proxyporthttps1:
bind: host
connect: tcp:localhost:443
:
type: proxy