Hi,
I am trying to copy an instance to a different server (not clustered) and I get the following error.
Error: Failed migration on source: Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF
Systemd journal:
Mar 26 06:28:21 hoof kernel: physwc3HpI: renamed from eth0
Mar 26 06:28:21 hoof kernel: incusbr0: port 9(veth80c977ed) entered disabled state
Mar 26 06:28:21 hoof kernel: vethbd7f1f61: renamed from physwc3HpI
Mar 26 06:28:21 hoof kernel: device veth80c977ed left promiscuous mode
Mar 26 06:28:21 hoof kernel: incusbr0: port 9(veth80c977ed) entered disabled state
Mar 26 06:28:22 hoof audit[52393]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-submission_mail_</var/lib/incus>" pid=52393 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.439:165): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-submission_mail_</var/lib/incus>" pid=52393 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52396]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtps_mail_</var/lib/incus>" pid=52396 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.459:166): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtps_mail_</var/lib/incus>" pid=52396 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52399]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtp_mail_</var/lib/incus>" pid=52399 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.479:167): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtp_mail_</var/lib/incus>" pid=52399 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52402]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-imaps_mail_</var/lib/incus>" pid=52402 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.499:168): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-imaps_mail_</var/lib/incus>" pid=52402 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52404]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-mail_</var/lib/incus>" pid=52404 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.571:169): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-mail_</var/lib/incus>" pid=52404 comm="apparmor_parser"
Mar 26 06:28:34 hoof incusd[3018]: time="2025-03-26T06:28:34+01:00" level=error msg="Failed migration on source" clusterMoveSourceName= err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=mail live=false project=default push=false
Both Servers are Debian Bookworm and Incus 6.10.1.
Does anybody have an idea what the problem might be with the migration index header?
stgraber
March 26, 2025, 4:27pm
2
Can you try running incus monitor --pretty on both source and target?
Sometimes, it’s also network related, in which case you may want to give --mode=relay a go.
Thank you!
That helped to find the Problem. I set a different ipv4 address as the network is different. However, this does not apply to the snapshots.
not working:
incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158
working
incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158 --instance-only
log serverb:
DEBUG [2025-03-26T20:03:48+01:00] Instance operation lock created action=create instance=instance/212122-08-03_23-22-47 project=default reusable=false
INFO [2025-03-26T20:03:48+01:00] Creating instance snapshot ephemeral=false instance=instance/212122-08-03_23-22-47 instanceType=container project=default
ERROR [2025-03-26T20:03:48+01:00] Failed initializing instance err="Invalid devices: Device validation failed for \"eth0\": Device IP address \"10.16.128.158\" not within network \"incusbr0\" subnet" instance=instance/212122-08-03_23-22-47 project=default type=container
DEBUG [2025-03-26T20:03:48+01:00] Instance operation lock finished action=create err="Invalid devices: Device validation failed for \"eth0\": Device IP address \"10.16.128.158\" not within network \"incusbr0\" subnet" instance=instance/212122-08-03_23-22-47 project=default reusable=false
log servera:
DEBUG [2025-03-26T20:03:48+01:00] Starting storage migration phase instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Migrate send transfer started instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Websocket: Sending barrier message address="10.0.0.4:39968"
DEBUG [2025-03-26T20:03:48+01:00] Sent migration index header, waiting for response args="&{IndexHeaderVersion:1 Name:instance Snapshots:[212122-08-03_23-22-47 222222-08-03_23-22-00 232322-08-03_23-22-05 242422-08-03_23-22-06 252522-08-03_23-22-50] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes header_subvolume_uuids]} TrackProgress:true MultiSync:false FinalSync:false Data:<nil> ContentType: AllowInconsistent:false Refresh:false Info:0xc000f0c2b8 VolumeOnly:false ClusterMove:false StorageMove:false}" driver=btrfs instance=instance pool=default project=default version=1
DEBUG [2025-03-26T20:03:48+01:00] Instance operation lock finished action=migrate err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=instance project=default reusable=false
DEBUG [2025-03-26T20:03:48+01:00] Migration send stopped instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Migrate send control monitor finished instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Migrate send transfer finished instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Migration channels disconnected on source clusterMoveSourceName= instance=instance live=false project=default push=false
ERROR [2025-03-26T20:03:48+01:00] Failed migration on source clusterMoveSourceName= err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=instance live=false project=default push=false
DEBUG [2025-03-26T20:03:48+01:00] MigrateInstance finished args="&{IndexHeaderVersion:1 Name:instance Snapshots:[212122-08-03_23-22-47 222222-08-03_23-22-00 232322-08-03_23-22-05 242422-08-03_23-22-06 252522-08-03_23-22-50] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes header_subvolume_uuids]} TrackProgress:true MultiSync:false FinalSync:false Data:<nil> ContentType: AllowInconsistent:false Refresh:false Info:0xc000f0c2b8 VolumeOnly:false ClusterMove:false StorageMove:false}" driver=btrfs instance=instance pool=default project=default
DEBUG [2025-03-26T20:03:48+01:00] Failure for operation class=websocket description="Migrating instance" err="Failed migration on source: Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" operation=684ab5f7-b30e-4dd0-8232-65d07cac1056 project=default
lt
Is there a way for this copy to work to include the snapshots? I guess maybe re-configuring the network on serverb to use the same ipv4 range would work.
stgraber
March 26, 2025, 7:32pm
4
Do you get a more useful log when using the --mode=relay option to incus copy?
If I do that, I get the relevant part directly in the output to the incus copy command.
incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158 --mode=relay --refresh
Error: Error transferring instance data: Failed migration on target: Failed creating instance snapshot record "instance/212122-08-03_23-22-47": Failed initializing instance: Invalid devices: Device validation failed for "eth0": Device IP address "10.16.128.158" not within network "incusbr0" subnet
stgraber
March 26, 2025, 8:01pm
6
Okay, so that does explain the issue.
We’ve historically relaxed some of those checks when dealing with a snapshot, so we may be able to do something like that here too.
Could you file an issue at GitHub · Where software is built with that last error message?
1 Like