Hi,
I’ll setup a lvmcluster but getting this error message.
Error: Failed to create storage pool “remote”: Failed to run: vgchange --addtag incus_pool incusvg: exit status 5 (Cannot access VG incusvg due to failed lock.)
I have already changed lvm.conf and lvmlocal.conf and here are my vg information. Can someone assist me? Regards.
``
Reading VG incusvg without a lock.
PV VG Fmt Attr PSize PFree
/dev/nvme0n1 incusvg lvm2 a-- 931.51g 931.26g
``
What happens if you run vgchange --lock-start? (I think that’s the right syntax, if not, try --lockstart)
Here are the commands and the result.
indiana@tnode1:~$ sudo vgchange --lockstart incusvg
VG incusvg starting sanlock lockspace
Starting locking. Waiting for sanlock may take a few seconds to 3 min…
indiana@tnode1:~$
indiana@tnode1:~$ sudo vgs
Reading VG incusvg without a lock.
VG #PV #LV #SN Attr VSize VFree
incusvg 1 0 0 wz–ns 931.51g 931.26g
indiana@tnode1:~$
indiana@tnode1:~$ incus admin init
Would you like to use clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=192.168.1.201]:
Are you joining an existing cluster? (yes/no) [default=no]:
What member name should be used to identify this server in the cluster? [default=tnode1]:
Do you want to configure a new local storage pool? (yes/no) [default=yes]: no
Do you want to configure a new remote storage pool? (yes/no) [default=no]: yes
Name of the storage backend to use (truenas, lvmcluster) [default=truenas]: lvmcluster
Create a new LVMCLUSTER pool? (yes/no) [default=yes]: no
Name of the existing LVMCLUSTER pool or dataset: incusvg
Would you like to use an existing bridge or host interface? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML “init” preseed to be printed? (yes/no) [default=no]:
Error: Failed to create storage pool “remote”: Failed to run: vgchange --addtag incus_pool incusvg: exit status 5 (Cannot access VG incusvg due to failed lock.)
And sudo vgs after that is still clean?
It’s certainly an unusual behavior. I’d expect something like this if there was a node-id conflict or something like that.
Thanks for the reply Graber,
I have figured it out at last , the board itself is arm64 single board computer and those errors lead the way.
Dec 26 22:25:31 tnode2 sanlock[818]: 2025-12-26 22:25:31 25 [1261]: s1 wdmd_connect failed -2
Dec 26 22:25:31 tnode2 sanlock[818]: 2025-12-26 22:25:31 25 [1261]: s1 connect_watchdog failed -1
Dec 26 22:25:32 tnode2 sanlock[818]: 2025-12-26 22:25:32 26 [846]: s1 add_lockspace fail result -203
I have given those clues to AI and here is the conclusion.
P.S. Make a simple documentation and share with the community soon.
Regards.
I have some problems that I’m not sure it is related with lvmcluster or not. One of them is moving one instance from one node to another. For example.
incus move webserver --target=tnode2
Error: Migration operation failure: Instance move to destination failed on source: Failed migration on source: Error from migration control target: Failed creating instance on target: Volume exists in database but not on storage
What can be wrong?
Which command format is correct, I also tried like that.
indiana@tnode2:~$ incus move webserver tnode2:
Error: The remote “tnode2” doesn’t exist
Here are some system information.
indiana@tnode1:~$ incus cluster ls
+--------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| NAME | URL | ROLES | ARCHITECTURE | FAILURE DOMAIN | DESCRIPTION | STATUS | MESSAGE |
+--------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| tnode1 | https://192.168.1.201:8443 | database-leader | aarch64 | default | | ONLINE | Fully operational |
| | | database | | | | | |
+--------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| tnode2 | https://192.168.1.202:8443 | database | aarch64 | default | | ONLINE | Fully operational |
+--------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
| tnode3 | https://192.168.1.203:8443 | database | aarch64 | default | | ONLINE | Fully operational |
+--------+----------------------------+-----------------+--------------+----------------+-------------+--------+-------------------+
indiana@tnode1:~$ incus storage ls
+--------+------------+-------------+---------+---------+
| NAME | DRIVER | DESCRIPTION | USED BY | STATE |
+--------+------------+-------------+---------+---------+
| remote | lvmcluster | | 4 | CREATED |
+--------+------------+-------------+---------+---------+
indiana@tnode1:~$ incus storage show remote
config: {}
description: ""
name: remote
driver: lvmcluster
used_by:
- /1.0/instances/test
- /1.0/instances/testvm
- /1.0/instances/webserver
- /1.0/profiles/default
status: Created
locations:
- tnode1
- tnode2
- tnode3
The command is correct (incus move NAME --target SERVER).
Does (lvs) show an identical output on both the source and target server?
The command incus move webserver --target tnode2 does not show an identical output. Tested on another hosts but the same error occurs.
indiana@tnode2:~$ incus move webserver --target tnode2Error: Migration operation failure: Instance move to destination failed on source: Failed migration on source: Error from migration control target: Failed creating instance on target: Volume exists in database but not on storageindiana@tnode2:~$ sudo lvsLV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convertvirtual-machines_testvm incusvg -wi------k 500.00mvirtual-machines_testvm.block incusvg -wi------k 10.00g
Here is the log file.
debuglog.txt (9.0 KB)
I have change the configuration a little bit and delete the old instances and recreate them. Now I get this error message.
indiana@tnode1:~$ incus move test --target tnode2
Error: Migration operation failure: Instance move to destination failed on source: Failed migration on source: migration dump failed
(00.016306) Error (criu/namespaces.c:460): Can’t dump nested uts namespace for 3300
(00.016312) Error (criu/namespaces.c:721): Can’t make utsns id
(00.021191) Error (criu/util.c:657): exited, status=1
(00.023992) Error (criu/util.c:657): exited, status=1
(00.071266) Error (criu/cr-dump.c:2128): Dumping FAILED.
Same error again.
indiana@tnode1:~$ incus ls
±----------±--------±-----±-----±----------±----------±---------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
±----------±--------±-----±-----±----------±----------±---------+
| test | STOPPED | | | CONTAINER | 0 | tnode1 |
±----------±--------±-----±-----±----------±----------±---------+
| webserver | STOPPED | | | CONTAINER | 0 | tnode3 |
±----------±--------±-----±-----±----------±----------±---------+
indiana@tnode1:~$
indiana@tnode1:~$ incus move test --target tnode2
Error: Migration operation failure: Instance move to destination failed on source: Failed migration on source: Error from migration control target: Failed creating instance on target: Volume exists in database but not on storage
indiana@tnode1:~$
You didn’t answer my question about the lvs output being identical on all systems.
The criu error is expected as live migration of containers very rarely works.
For containers you really need to stop + move + start.
Here is the output. On the tnode2 no output when command executed.
indiana@tnode1:~$ sudo lvs
[sudo] password for indiana:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
containers_test incusvg -wi------k 10.00g
indiana@tnode3:~$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
containers_webserver incusvg -wi------k 10.00g
Now, I have launched another vm and now the output looks like that.
indiana@tnode2:~$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
virtual-machines_testvm incusvg -wi-ao—k 500.00m
virtual-machines_testvm.block incusvg -wi-ao—k 10.00g
Humm, now the vm migration hangs on this state.
indiana@tnode1:~$ incus move testvm --target tnode1
Transferring instance: Live migration: 1.21GB remaining (0B/s) (0% CPU throttle)
By the way you mean, vgs?
Yeah, so there’s something very wrong with your setup.
Clustered LVM requires all systems to be connected to the same backing storage, so all servers see the same PVs, VGs and LVs. Then clustered LVM handles locking across systems to make this all work safely.
Your environment appears to have distinct storage on each server rather than shared, this can’t with with LVM cluster.
The hardware setup you have is likely better suited for Linstor or Ceph.
Thanks Stephane for the support, it is better to install Linstor for my setup.
Regards.