Incus copy error: Failed reading migration index header

Hi,

I am trying to copy an instance to a different server (not clustered) and I get the following error.

Error: Failed migration on source: Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF

Systemd journal:

Mar 26 06:28:21 hoof kernel: physwc3HpI: renamed from eth0
Mar 26 06:28:21 hoof kernel: incusbr0: port 9(veth80c977ed) entered disabled state
Mar 26 06:28:21 hoof kernel: vethbd7f1f61: renamed from physwc3HpI
Mar 26 06:28:21 hoof kernel: device veth80c977ed left promiscuous mode
Mar 26 06:28:21 hoof kernel: incusbr0: port 9(veth80c977ed) entered disabled state
Mar 26 06:28:22 hoof audit[52393]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-submission_mail_</var/lib/incus>" pid=52393 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.439:165): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-submission_mail_</var/lib/incus>" pid=52393 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52396]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtps_mail_</var/lib/incus>" pid=52396 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.459:166): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtps_mail_</var/lib/incus>" pid=52396 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52399]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtp_mail_</var/lib/incus>" pid=52399 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.479:167): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-smtp_mail_</var/lib/incus>" pid=52399 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52402]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-imaps_mail_</var/lib/incus>" pid=52402 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.499:168): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_forkproxy-imaps_mail_</var/lib/incus>" pid=52402 comm="apparmor_parser"
Mar 26 06:28:22 hoof audit[52404]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-mail_</var/lib/incus>" pid=52404 comm="apparmor_parser"
Mar 26 06:28:22 hoof kernel: audit: type=1400 audit(1742966902.571:169): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-mail_</var/lib/incus>" pid=52404 comm="apparmor_parser"
Mar 26 06:28:34 hoof incusd[3018]: time="2025-03-26T06:28:34+01:00" level=error msg="Failed migration on source" clusterMoveSourceName= err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=mail live=false project=default push=false

Both Servers are Debian Bookworm and Incus 6.10.1.
I can copy all other instances between these two servers without problems.

Does anybody have an idea what the problem might be with the migration index header?
Thanks!

Can you try running incus monitor --pretty on both source and target?

Sometimes, it’s also network related, in which case you may want to give --mode=relay a go.

Thank you!

That helped to find the Problem. I set a different ipv4 address as the network is different. However, this does not apply to the snapshots.
servera ipv4.address=10.16.128.158
serverb ipv4.address=10.118.21.158

not working:

incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158

working

incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158 --instance-only

log serverb:

DEBUG  [2025-03-26T20:03:48+01:00] Instance operation lock created               action=create instance=instance/212122-08-03_23-22-47 project=default reusable=false
INFO   [2025-03-26T20:03:48+01:00] Creating instance snapshot                    ephemeral=false instance=instance/212122-08-03_23-22-47 instanceType=container project=default
ERROR  [2025-03-26T20:03:48+01:00] Failed initializing instance                  err="Invalid devices: Device validation failed for \"eth0\": Device IP address \"10.16.128.158\" not within network \"incusbr0\" subnet" instance=instance/212122-08-03_23-22-47 project=default type=container
DEBUG  [2025-03-26T20:03:48+01:00] Instance operation lock finished              action=create err="Invalid devices: Device validation failed for \"eth0\": Device IP address \"10.16.128.158\" not within network \"incusbr0\" subnet" instance=instance/212122-08-03_23-22-47 project=default reusable=false

log servera:

DEBUG  [2025-03-26T20:03:48+01:00] Starting storage migration phase              instance=instance instanceType=container project=default
DEBUG  [2025-03-26T20:03:48+01:00] Migrate send transfer started                 instance=instance instanceType=container project=default
DEBUG  [2025-03-26T20:03:48+01:00] Websocket: Sending barrier message            address="10.0.0.4:39968"
DEBUG  [2025-03-26T20:03:48+01:00] Sent migration index header, waiting for response  args="&{IndexHeaderVersion:1 Name:instance Snapshots:[212122-08-03_23-22-47 222222-08-03_23-22-00 232322-08-03_23-22-05 242422-08-03_23-22-06 252522-08-03_23-22-50] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes header_subvolume_uuids]} TrackProgress:true MultiSync:false FinalSync:false Data:<nil> ContentType: AllowInconsistent:false Refresh:false Info:0xc000f0c2b8 VolumeOnly:false ClusterMove:false StorageMove:false}" driver=btrfs instance=instance pool=default project=default version=1
DEBUG  [2025-03-26T20:03:48+01:00] Instance operation lock finished              action=migrate err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=instance project=default reusable=false
DEBUG  [2025-03-26T20:03:48+01:00] Migration send stopped                        instance=instance instanceType=container project=default
DEBUG  [2025-03-26T20:03:48+01:00] Migrate send control monitor finished         instance=instance instanceType=container project=default
DEBUG  [2025-03-26T20:03:48+01:00] Migrate send transfer finished                instance=instance instanceType=container project=default
DEBUG  [2025-03-26T20:03:48+01:00] Migration channels disconnected on source     clusterMoveSourceName= instance=instance live=false project=default push=false
ERROR  [2025-03-26T20:03:48+01:00] Failed migration on source                    clusterMoveSourceName= err="Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" instance=instance live=false project=default push=false
DEBUG  [2025-03-26T20:03:48+01:00] MigrateInstance finished                      args="&{IndexHeaderVersion:1 Name:instance Snapshots:[212122-08-03_23-22-47 222222-08-03_23-22-00 232322-08-03_23-22-05 242422-08-03_23-22-06 252522-08-03_23-22-50] MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes header_subvolume_uuids]} TrackProgress:true MultiSync:false FinalSync:false Data:<nil> ContentType: AllowInconsistent:false Refresh:false Info:0xc000f0c2b8 VolumeOnly:false ClusterMove:false StorageMove:false}" driver=btrfs instance=instance pool=default project=default
DEBUG  [2025-03-26T20:03:48+01:00] Failure for operation                         class=websocket description="Migrating instance" err="Failed migration on source: Failed reading migration index header: websocket: close 1006 (abnormal closure): unexpected EOF" operation=684ab5f7-b30e-4dd0-8232-65d07cac1056 project=default
lt

Is there a way for this copy to work to include the snapshots? I guess maybe re-configuring the network on serverb to use the same ipv4 range would work.

Do you get a more useful log when using the --mode=relay option to incus copy?

If I do that, I get the relevant part directly in the output to the incus copy command.

incus copy instance serverb:instance -d eth0,ipv4.address=10.118.21.158 --mode=relay --refresh
Error: Error transferring instance data: Failed migration on target: Failed creating instance snapshot record "instance/212122-08-03_23-22-47": Failed initializing instance: Invalid devices: Device validation failed for "eth0": Device IP address "10.16.128.158" not within network "incusbr0" subnet

Okay, so that does explain the issue.

We’ve historically relaxed some of those checks when dealing with a snapshot, so we may be able to do something like that here too.

Could you file an issue at GitHub · Where software is built with that last error message?

1 Like