Incus snapshot create over ssh tunnel stuck

Incus
1

hi,

I am struggling with an issue that I don’t understand. I am not even sure if it is caused by Incus or specific to the components.

There is a backup script that creates snapshots of Incus instances over an ssh tunnel. It worked well for a long time and is currently working well in other systems with LXD. As far as I remember, it was working correctly with Incus too for a while.

Now it is stuck forever when I run the script manually at this point (copied from the script):

ssh host-machine incus snapshot create incus-instance snapshot-name

The script runs correctly if it’s started as a cron job.

Additional information:

  • other commands (eg. snapshot list) runs correctly
  • even though the command is stuck, the snapshot created (both in the Incus database and by ZFS)

The incusd.log is empty.
There is no relevant entry in lxc.log file of the instance, only this:
lxc pts101 20240901193003.282 WARN idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing (pts101 is the instance).

The output of incus warning list is empty.

# incus version
Client version: 6.0.1
Server version: 6.0.1

# zfs version
zfs-2.2.4-0arter97~ubuntu22.04.1
zfs-kmod-2.2.4-0arter97~ubuntu22.04.1

Do you have any idea how to track this issue down?

2

Our commands allow reading from stdin for YAML config details.
Most likely your script has some data in stdin or some other command piped around or the like which is causing incus snapshot create to be stuck reading from stdin.

The easy fix for this is to do ssh host-machine incus snapshot create incus-instance snapshot-name < /dev/null

This overrides stdin for the command to read from /dev/null.

3

Thanks, that is a good way to work around the problem.
But the root cause is still there and this worked before.

ssh ‘incus snapshot create proxy21 man-2024-09-13-test1’

What could a simple snapshot create waiting for? Does it not stick for you?