Hi,
I am trying to copy an instance to a different server (not clustered) and I get the following error.
Error: Failed migration on source: Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF
Systemd journal:
Mar 26 06:28:21 hoof kernel: physwc3HpI: renamed from eth0
Mar 26 06:28:21 hoof kernel: incusbr0: port 9(veth80c977ed) entered disabled state
Mar 26 06:28:21 hoof kernel: vethbd7f1f61: renamed from physwc3HpI
Mar 26 06:28:21 hoof kernel: device veth80c977ed left promiscuous mode
Mar 26 06:28:21 hoof kernel: incusbr0: port 9(veth80c977ed) entered disabled state
Mar 26 06:28:22 hoof audit[52393]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-submission_mail_</var/lib/incus>" pid=52393 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.439:165): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-submission_mail_</var/lib/incus>" pid=52393 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52396]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtps_mail_</var/lib/incus>" pid=52396 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.459:166): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtps_mail_</var/lib/incus>" pid=52396 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52399]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtp_mail_</var/lib/incus>" pid=52399 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.479:167): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtp_mail_</var/lib/incus>" pid=52399 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52402]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-imaps_mail_</var/lib/incus>" pid=52402 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.499:168): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-imaps_mail_</var/lib/incus>" pid=52402 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52404]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-mail_</var/lib/incus>" pid=52404 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.571:169): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-mail_</var/lib/incus>" pid=52404 comm="apparmor_parser"
Mar 26 06:28:34 hoof incusd[3018]: time="2025-03-26T06:28:34+01:00" level=error msg="Failed migration on source" clusterMoveSourceName= err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=mail live=false project=default push=false
Both Servers are Debian Bookworm and Incus 6.10.1.
I can copy all other instances between these two servers without problems.
Does anybody have an idea what the problem might be with the migration index header?
Thanks!
stgraber
(Stéphane Graber)
March 26, 2025, 4:27pm
2
Can you try running incus monitor --pretty
on both source and target?
Sometimes, it’s also network related, in which case you may want to give --mode=relay
a go.
Thank you!
That helped to find the Problem. I set a different ipv4 address as the network is different. However, this does not apply to the snapshots.
servera ipv4.address=10.16.128.158
serverb ipv4.address=10.118.21.158
not working:
incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158
working
incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158 --instance-only
log serverb:
DEBUG [2025-03-26T20:03:48+01:00] Instance operation lock created action=create instance=instance/212122-08-03_23-22-47 project=default reusable=false
INFO [2025-03-26T20:03:48+01:00] Creating instance snapshot ephemeral=false instance=instance/212122-08-03_23-22-47 instanceType=container project=default
ERROR [2025-03-26T20:03:48+01:00] Failed initializing instance err="Invalid devices: Device validation failed for \"eth0\": Device IP address \"10.16.128.158\" not within network \"incusbr0\" subnet" instance=instance/212122-08-03_23-22-47 project=default type=container
DEBUG [2025-03-26T20:03:48+01:00] Instance operation lock finished action=create err="Invalid devices: Device validation failed for \"eth0\": Device IP address \"10.16.128.158\" not within network \"incusbr0\" subnet" instance=instance/212122-08-03_23-22-47 project=default reusable=false
log servera:
DEBUG [2025-03-26T20:03:48+01:00] Starting storage migration phase instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Migrate send transfer started instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Websocket: Sending barrier message address="10.0.0.4:39968"
DEBUG [2025-03-26T20:03:48+01:00] Sent migration index header, waiting for response args="&{IndexHeaderVersion:1 Name:instance Snapshots:[212122-08-03_23-22-47 222222-08-03_23-22-00 232322-08-03_23-22-05 242422-08-03_23-22-06 252522-08-03_23-22-50] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes header_subvolume_uuids]} TrackProgress:true MultiSync:false FinalSync:false Data:<nil> ContentType: AllowInconsistent:false Refresh:false Info:0xc000f0c2b8 VolumeOnly:false ClusterMove:false StorageMove:false}" driver=btrfs instance=instance pool=default project=default version=1
DEBUG [2025-03-26T20:03:48+01:00] Instance operation lock finished action=migrate err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=instance project=default reusable=false
DEBUG [2025-03-26T20:03:48+01:00] Migration send stopped instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Migrate send control monitor finished instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Migrate send transfer finished instance=instance instanceType=container project=default
DEBUG [2025-03-26T20:03:48+01:00] Migration channels disconnected on source clusterMoveSourceName= instance=instance live=false project=default push=false
ERROR [2025-03-26T20:03:48+01:00] Failed migration on source clusterMoveSourceName= err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=instance live=false project=default push=false
DEBUG [2025-03-26T20:03:48+01:00] MigrateInstance finished args="&{IndexHeaderVersion:1 Name:instance Snapshots:[212122-08-03_23-22-47 222222-08-03_23-22-00 232322-08-03_23-22-05 242422-08-03_23-22-06 252522-08-03_23-22-50] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes header_subvolume_uuids]} TrackProgress:true MultiSync:false FinalSync:false Data:<nil> ContentType: AllowInconsistent:false Refresh:false Info:0xc000f0c2b8 VolumeOnly:false ClusterMove:false StorageMove:false}" driver=btrfs instance=instance pool=default project=default
DEBUG [2025-03-26T20:03:48+01:00] Failure for operation class=websocket description="Migrating instance" err="Failed migration on source: Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" operation=684ab5f7-b30e-4dd0-8232-65d07cac1056 project=default
lt
Is there a way for this copy to work to include the snapshots? I guess maybe re-configuring the network on serverb to use the same ipv4 range would work.
stgraber
(Stéphane Graber)
March 26, 2025, 7:32pm
4
Do you get a more useful log when using the --mode=relay
option to incus copy
?
If I do that, I get the relevant part directly in the output to the incus copy command.
incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158 --mode=relay --refresh
Error: Error transferring instance data: Failed migration on target: Failed creating instance snapshot record "instance/212122-08-03_23-22-47": Failed initializing instance: Invalid devices: Device validation failed for "eth0": Device IP address "10.16.128.158" not within network "incusbr0" subnet
stgraber
(Stéphane Graber)
March 26, 2025, 8:01pm
6
Okay, so that does explain the issue.
We’ve historically relaxed some of those checks when dealing with a snapshot, so we may be able to do something like that here too.
Could you file an issue at GitHub · Where software is built with that last error message?
1 Like