above is the output with mode=relay
Try running lxc monitor --type=logging --pretty on both source and target while trying another lxc move --mode=relay see if that gives us a bit more details.
DBUG[10-06|12:41:57] New event listener: 327324d1-d5f6-4556-9bc3-9d7305781f19
DBUG[10-06|12:42:15] Handling ip=@ method=GET url=/1.0 user=
DBUG[10-06|12:42:15] Handling ip=@ method=GET url=/1.0/instances/test user=
DBUG[10-06|12:42:15] Handling user= ip=@ method=GET url=/1.0/events
DBUG[10-06|12:42:15] New event listener: f800e32e-6340-46fa-b99b-e3d9c86d7122
DBUG[10-06|12:42:15] Handling ip=@ method=POST url=/1.0/instances/test user=
DBUG[10-06|12:42:15] New websocket Operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
DBUG[10-06|12:42:15] Started websocket operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
DBUG[10-06|12:42:15] Handling user= ip=@ method=GET url="/1.0/operations/5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d/websocket?secret=21b56e55a633e3ede0a9946e306b47e0a6e52d12d4a4d8ceb54cc44b3bc901e4"
DBUG[10-06|12:42:15] Connected websocket Operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
DBUG[10-06|12:42:15] Handled websocket Operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
DBUG[10-06|12:42:15] Handling url="/1.0/operations/5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d/websocket?secret=5d5d9d460df7246dbaea903cfe07d6bcf82a3c427dc17b8397cf6a3338cdfdd3" user= ip=@ method=GET
DBUG[10-06|12:42:15] Connected websocket Operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
DBUG[10-06|12:42:15] Handled websocket Operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
DBUG[10-06|12:42:15] Handling ip=@ method=GET url="/1.0/operations/5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d/websocket?secret=cd571fd6c80db949e3a00a23eedbeac51a30e602ae949152db159ffb25092b62" user=
DBUG[10-06|12:42:15] Connected websocket Operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
DBUG[10-06|12:42:15] Handled websocket Operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
INFO[10-06|12:42:15] Migrating container actionscript=false features=1 name=test project=default statedir= created="2020-10-06 07:15:24.797099714 +0000 UTC" ephemeral=false predumpdir= stop=false used="2020-10-06 07:15:25.403411621 +0000 UTC"
DBUG[10-06|12:42:15] Using maximal 10 iterations for pre-dumping
DBUG[10-06|12:42:15] MigrateInstance started pool=default project=default args="&{test [] {ZFS [compress]} true true false <nil> }" driver=zfs instance=test
DBUG[10-06|12:42:16] Updated metadata for websocket Operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d
DBUG[10-06|12:42:16] MigrateInstance finished args="&{test [] {ZFS [compress]} true true false map[filesystem:default/containers/test@migration-8ec808b1-ed1a-498e-8ea1-1c4bfa30ae30] }" driver=zfs instance=test pool=default project=default
DBUG[10-06|12:42:16] New websocket Operation: 8d7dd5a9-a315-4d1b-8347-e4e702eb36e4
DBUG[10-06|12:42:16] The other side does support pre-copy
DBUG[10-06|12:42:16] Doing another pre-dump in
INFO[10-06|12:42:16] Migrating container name=test predumpdir= project=default statedir=/tmp/lxd_checkpoint_277170320 stop=false ephemeral=false created="2020-10-06 07:15:24.797099714 +0000 UTC" features=0 used="2020-10-06 07:15:25.403411621 +0000 UTC" actionscript=false
INFO[10-06|12:42:27] Failed migrating container used="2020-10-06 07:15:25.403411621 +0000 UTC" actionscript=false created="2020-10-06 07:15:24.797099714 +0000 UTC" features=0 predumpdir= project=default statedir=/tmp/lxd_checkpoint_277170320 stop=false ephemeral=false name=test
DBUG[10-06|12:42:27] Failure for websocket operation: 5750a9b8-02f0-4ed5-8b3e-f94d7adbf45d: migration pre-dump failed
(00.000140) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.000147) Error (criu/seize.c:288): zombie found while seizing
(00.100216) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.100223) Error (criu/seize.c:288): zombie found while seizing
(00.200292) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.200298) Error (criu/seize.c:288): zombie found while seizing
(00.300367) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.300373) Error (criu/seize.c:288): zombie found while seizing
(00.400440) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.400446) Error (criu/seize.c:288): zombie found while seizing
(00.500515) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.500521) Error (criu/seize.c:288): zombie found while seizing
(00.600590) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.600596) Error (criu/seize.c:288): zombie found while seizing
(00.700663) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.700669) Error (criu/seize.c:288): zombie found while seizing
(00.800736) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.800742) Error (criu/seize.c:288): zombie found while seizing
(00.900810) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(00.900816) Error (criu/seize.c:288): zombie found while seizing
(01.000884) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.000890) Error (criu/seize.c:288): zombie found while seizing
(01.100957) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.100963) Error (criu/seize.c:288): zombie found while seizing
(01.201030) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.201036) Error (criu/seize.c:288): zombie found while seizing
(01.301103) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.301109) Error (criu/seize.c:288): zombie found while seizing
(01.401176) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.401182) Error (criu/seize.c:288): zombie found while seizing
(01.501253) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.501259) Error (criu/seize.c:288): zombie found while seizing
(01.601332) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.601338) Error (criu/seize.c:288): zombie found while seizing
(01.701404) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.701410) Error (criu/seize.c:288): zombie found while seizing
(01.801477) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.801483) Error (criu/seize.c:288): zombie found while seizing
(01.901550) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(01.901556) Error (criu/seize.c:288): zombie found while seizing
(02.001623) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.001628) Error (criu/seize.c:288): zombie found while seizing
(02.101696) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.101702) Error (criu/seize.c:288): zombie found while seizing
(02.201769) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.201774) Error (criu/seize.c:288): zombie found while seizing
(02.301840) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.301846) Error (criu/seize.c:288): zombie found while seizing
(02.401912) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.401918) Error (criu/seize.c:288): zombie found while seizing
(02.501984) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.501990) Error (criu/seize.c:288): zombie found while seizing
(02.602057) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.602063) Error (criu/seize.c:288): zombie found while seizing
(02.702130) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.702138) Error (criu/seize.c:288): zombie found while seizing
(02.802206) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.802212) Error (criu/seize.c:288): zombie found while seizing
(02.902279) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(02.902285) Error (criu/seize.c:288): zombie found while seizing
(03.002351) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.002356) Error (criu/seize.c:288): zombie found while seizing
(03.102423) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.102429) Error (criu/seize.c:288): zombie found while seizing
(03.202496) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.202502) Error (criu/seize.c:288): zombie found while seizing
(03.302568) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.302574) Error (criu/seize.c:288): zombie found while seizing
(03.402640) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.402646) Error (criu/seize.c:288): zombie found while seizing
(03.502713) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.502718) Error (criu/seize.c:288): zombie found while seizing
(03.602785) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.602791) Error (criu/seize.c:288): zombie found while seizing
(03.702870) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.702881) Error (criu/seize.c:288): zombie found while seizing
(03.802949) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.802955) Error (criu/seize.c:288): zombie found while seizing
(03.903021) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(03.903033) Error (criu/seize.c:288): zombie found while seizing
(04.003100) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.003105) Error (criu/seize.c:288): zombie found while seizing
(04.103192) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.103204) Error (criu/seize.c:288): zombie found while seizing
(04.203273) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.203280) Error (criu/seize.c:288): zombie found while seizing
(04.303347) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.303353) Error (criu/seize.c:288): zombie found while seizing
(04.403420) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.403426) Error (criu/seize.c:288): zombie found while seizing
(04.503492) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.503498) Error (criu/seize.c:288): zombie found while seizing
(04.603565) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.603571) Error (criu/seize.c:288): zombie found while seizing
(04.703637) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.703644) Error (criu/seize.c:288): zombie found while seizing
(04.803711) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.803717) Error (criu/seize.c:288): zombie found while seizing
(04.903783) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(04.903788) Error (criu/seize.c:288): zombie found while seizing
(05.003855) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.003863) Error (criu/seize.c:288): zombie found while seizing
(05.103934) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.103941) Error (criu/seize.c:288): zombie found while seizing
(05.204008) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.204014) Error (criu/seize.c:288): zombie found while seizing
(05.304081) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.304086) Error (criu/seize.c:288): zombie found while seizing
(05.404153) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.404158) Error (criu/seize.c:288): zombie found while seizing
(05.504225) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.504231) Error (criu/seize.c:288): zombie found while seizing
(05.604297) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.604303) Error (criu/seize.c:288): zombie found while seizing
(05.704373) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.704379) Error (criu/seize.c:288): zombie found while seizing
(05.804445) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.804451) Error (criu/seize.c:288): zombie found while seizing
(05.904517) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(05.904523) Error (criu/seize.c:288): zombie found while seizing
(06.004590) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.004596) Error (criu/seize.c:288): zombie found while seizing
(06.104662) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.104668) Error (criu/seize.c:288): zombie found while seizing
(06.204736) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.204742) Error (criu/seize.c:288): zombie found while seizing
(06.304809) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.304818) Error (criu/seize.c:288): zombie found while seizing
(06.404885) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.404891) Error (criu/seize.c:288): zombie found while seizing
(06.504958) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.504964) Error (criu/seize.c:288): zombie found while seizing
(06.605031) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.605037) Error (criu/seize.c:288): zombie found while seizing
(06.705103) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.705109) Error (criu/seize.c:288): zombie found while seizing
(06.805175) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.805181) Error (criu/seize.c:288): zombie found while seizing
(06.905247) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(06.905253) Error (criu/seize.c:288): zombie found while seizing
(07.005321) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.005329) Error (criu/seize.c:288): zombie found while seizing
(07.105401) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.105408) Error (criu/seize.c:288): zombie found while seizing
(07.205475) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.205481) Error (criu/seize.c:288): zombie found while seizing
(07.305549) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.305555) Error (criu/seize.c:288): zombie found while seizing
(07.405621) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.405627) Error (criu/seize.c:288): zombie found while seizing
(07.505693) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.505699) Error (criu/seize.c:288): zombie found while seizing
(07.605765) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.605771) Error (criu/seize.c:288): zombie found while seizing
(07.705838) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.705843) Error (criu/seize.c:288): zombie found while seizing
(07.805910) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.805916) Error (criu/seize.c:288): zombie found while seizing
(07.905983) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(07.905988) Error (criu/seize.c:288): zombie found while seizing
(08.006055) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.006061) Error (criu/seize.c:288): zombie found while seizing
(08.106127) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.106133) Error (criu/seize.c:288): zombie found while seizing
(08.206200) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.206206) Error (criu/seize.c:288): zombie found while seizing
(08.306273) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.306279) Error (criu/seize.c:288): zombie found while seizing
(08.406345) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.406351) Error (criu/seize.c:288): zombie found while seizing
(08.506416) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.506422) Error (criu/seize.c:288): zombie found while seizing
(08.606487) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.606492) Error (criu/seize.c:288): zombie found while seizing
(08.706559) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.706569) Error (criu/seize.c:288): zombie found while seizing
(08.806647) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.806657) Error (criu/seize.c:288): zombie found while seizing
(08.906726) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(08.906733) Error (criu/seize.c:288): zombie found while seizing
(09.006801) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.006807) Error (criu/seize.c:288): zombie found while seizing
(09.106891) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.106902) Error (criu/seize.c:288): zombie found while seizing
(09.206971) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.206977) Error (criu/seize.c:288): zombie found while seizing
(09.307043) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.307049) Error (criu/seize.c:288): zombie found while seizing
(09.407115) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.407121) Error (criu/seize.c:288): zombie found while seizing
(09.507187) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.507193) Error (criu/seize.c:288): zombie found while seizing
(09.607259) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.607265) Error (criu/seize.c:288): zombie found while seizing
(09.707331) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.707337) Error (criu/seize.c:288): zombie found while seizing
(09.807403) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.807409) Error (criu/seize.c:288): zombie found while seizing
(09.907474) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(09.907479) Error (criu/seize.c:288): zombie found while seizing
(10.000108) Error (criu/cr-dump.c:1454): Timeout reached. Try to interrupt: 0
(10.000127) Warn (compel/src/lib/infect.c:127): Unable to interrupt task: 17928 (Operation not permitted)
(10.000131) Error (criu/seize.c:288): zombie found while seizing
(10.000165) Error (compel/src/lib/infect.c:236): Unseizable non-zombie 17928 found, state t, err -1/10
(10.000178) Error (compel/src/lib/infect.c:346): Unable to detach from 17928: No such process
(10.000211) Error (criu/cr-dump.c:1567): Pre-dumping FAILED.
DBUG[10-06|12:42:27] Event listener finished: f800e32e-6340-46fa-b99b-e3d9c86d7122
DBUG[10-06|12:42:27] Disconnected event listener: f800e32e-6340-46fa-b99b-e3d9c86d7122
DBUG[10-06|12:42:52] Handling ip=@ method=GET url=/1.0 user=
DBUG[10-06|12:42:52] Handling ip=@ method=GET url=/1.0 user=
INFO[10-06|12:42:59] Starting shutdown sequence
INFO[10-06|12:42:59] Received 'terminated signal', exiting
DBUG[10-06|12:42:59] Cancel ongoing or future gRPC connection attempts
INFO[10-06|12:42:59] Closing the database
INFO[10-06|12:42:59] Stop database gateway
INFO[10-06|12:42:59] Failed migrating container features=0 predumpdir=003 stop=true used="2020-10-06 07:15:25.403411621 +0000 UTC" actionscript=true ephemeral=false project=default statedir=/tmp/lxd_checkpoint_802152523 created="2020-10-06 07:15:24.797099714 +0000 UTC" name=test
Error: websocket: close 1006 (abnormal closure): unexpected EOF
root@cpu-6225:~#
root@cpu-6225:~#
root@cpu-6225:~#
root@cpu-6225:~# lxc monitor --type=logging --pretty
DBUG[10-06|12:43:23] New event listener: 447382f0-b6e5-4cd3-988e-71c6556c2d74
DBUG[10-06|12:43:31] Handling url=/1.0 user= ip=@ method=GET
DBUG[10-06|12:43:31] Handling ip=@ method=GET url=/1.0/instances/test user=
DBUG[10-06|12:43:31] Handling ip=@ method=GET url=/1.0/events user=
DBUG[10-06|12:43:31] New event listener: 22221935-6d81-4757-bdc2-e5232a06b0db
DBUG[10-06|12:43:31] Handling ip=@ method=POST url=/1.0/instances/test user=
DBUG[10-06|12:43:31] New websocket Operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
DBUG[10-06|12:43:31] Started websocket operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
DBUG[10-06|12:43:31] Connected websocket Operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
DBUG[10-06|12:43:31] Handling ip=@ method=GET url="/1.0/operations/e3916462-85eb-4003-9471-acc7c2c2b0a6/websocket?secret=76ad8a113d75ac6e5adea6a64fb3c7f456e322c08f4ffb9cdf2ab8a8e34fb3c6" user=
DBUG[10-06|12:43:31] Handled websocket Operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
DBUG[10-06|12:43:31] Handling ip=@ method=GET url="/1.0/operations/e3916462-85eb-4003-9471-acc7c2c2b0a6/websocket?secret=f7e2866be96220eeb0f4c085d86f744ae52ab4cfb81bb6ff3e1ac66c8218335d" user=
DBUG[10-06|12:43:31] Connected websocket Operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
DBUG[10-06|12:43:31] Handled websocket Operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
DBUG[10-06|12:43:31] Handling ip=@ method=GET url="/1.0/operations/e3916462-85eb-4003-9471-acc7c2c2b0a6/websocket?secret=c722fb9cd4bd198102bc2293a269e35125013cd0cecea5ee78e3f3f68c3e3546" user=
DBUG[10-06|12:43:31] Connected websocket Operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
DBUG[10-06|12:43:31] Handled websocket Operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
INFO[10-06|12:43:31] Migrating container project=default created="2020-10-06 07:15:24.797099714 +0000 UTC" ephemeral=false name=test statedir= stop=false used="2020-10-06 07:15:25.403411621 +0000 UTC" actionscript=false features=1 predumpdir=
DBUG[10-06|12:43:31] Using maximal 10 iterations for pre-dumping
DBUG[10-06|12:43:31] MigrateInstance started driver=zfs instance=test pool=default project=default args="&{test [] {ZFS [compress]} true true false <nil> }"
DBUG[10-06|12:43:32] Updated metadata for websocket Operation: e3916462-85eb-4003-9471-acc7c2c2b0a6
DBUG[10-06|12:43:33] MigrateInstance finished args="&{test [] {ZFS [compress]} true true false map[filesystem:default/containers/test@migration-2cf3987c-940c-414b-b96d-132273beeb81] }" driver=zfs instance=test pool=default project=default
DBUG[10-06|12:43:33] New websocket Operation: 9ccb2341-ed0c-46da-9407-360f747a232c
DBUG[10-06|12:43:33] The other side does support pre-copy
DBUG[10-06|12:43:33] Doing another pre-dump in
INFO[10-06|12:43:33] Migrating container features=0 stop=false used="2020-10-06 07:15:25.403411621 +0000 UTC" actionscript=false created="2020-10-06 07:15:24.797099714 +0000 UTC" ephemeral=false name=test predumpdir= project=default statedir=/tmp/lxd_checkpoint_059314377
INFO[10-06|12:43:33] Migrated container created="2020-10-06 07:15:24.797099714 +0000 UTC" features=0 name=test predumpdir= statedir=/tmp/lxd_checkpoint_059314377 stop=false used="2020-10-06 07:15:25.403411621 +0000 UTC" actionscript=false ephemeral=false project=default
DBUG[10-06|12:43:33] CRIU pages written 1419
DBUG[10-06|12:43:33] CRIU pages skipped 0
DBUG[10-06|12:43:33] CRIU pages skipped percentage 0%
DBUG[10-06|12:43:33] Sending another header
DBUG[10-06|12:43:33] Sending another header done
DBUG[10-06|12:43:33] Doing another pre-dump in 001
INFO[10-06|12:43:33] Migrating container used="2020-10-06 07:15:25.403411621 +0000 UTC" features=0 statedir=/tmp/lxd_checkpoint_059314377 ephemeral=false name=test predumpdir=001 project=default stop=false actionscript=false created="2020-10-06 07:15:24.797099714 +0000 UTC"
INFO[10-06|12:43:33] Migrated container actionscript=false created="2020-10-06 07:15:24.797099714 +0000 UTC" predumpdir=001 statedir=/tmp/lxd_checkpoint_059314377 ephemeral=false features=0 name=test project=default stop=false used="2020-10-06 07:15:25.403411621 +0000 UTC"
DBUG[10-06|12:43:33] CRIU pages written 24
DBUG[10-06|12:43:33] CRIU pages skipped 1395
DBUG[10-06|12:43:33] Memory pages skipped (99%) due to pre-copy is larger than threshold (70%)
DBUG[10-06|12:43:33] CRIU pages skipped percentage 99%
DBUG[10-06|12:43:33] This was the last pre-dump; next dump is the final dump
DBUG[10-06|12:43:33] Sending another header
DBUG[10-06|12:43:33] Sending another header done
DBUG[10-06|12:43:33] Started websocket operation: 9ccb2341-ed0c-46da-9407-360f747a232c
INFO[10-06|12:43:33] Migrating container predumpdir=003 statedir=/tmp/lxd_checkpoint_059314377 used="2020-10-06 07:15:25.403411621 +0000 UTC" created="2020-10-06 07:15:24.797099714 +0000 UTC" name=test features=0 project=default stop=true actionscript=true ephemeral=false
DBUG[10-06|12:43:35] Connected websocket Operation: 9ccb2341-ed0c-46da-9407-360f747a232c
DBUG[10-06|12:43:35] Handling ip=@ method=GET url="/1.0/operations/9ccb2341-ed0c-46da-9407-360f747a232c/websocket?secret=e452ff489f1a9176ee3622996d9a3028e10f6287130750644eee7d8e6bfc38b1" user=
DBUG[10-06|12:43:35] Handled websocket Operation: 9ccb2341-ed0c-46da-9407-360f747a232c
DBUG[10-06|12:43:35] Dump finished, continuing with restore...
DBUG[10-06|12:43:35] Handling user= ip=@ method=GET url=/1.0/operations/9ccb2341-ed0c-46da-9407-360f747a232c/wait
DBUG[10-06|12:43:35] MigrateInstance started instance=test pool=default project=default args="&{test [] {ZFS [compress]} true true true map[filesystem:default/containers/test@migration-2cf3987c-940c-414b-b96d-132273beeb81] }" driver=zfs
DBUG[10-06|12:43:36] Event listener finished: 22221935-6d81-4757-bdc2-e5232a06b0db
DBUG[10-06|12:43:36] Disconnected event listener: 22221935-6d81-4757-bdc2-e5232a06b0db
DBUG[10-06|12:43:41] Handling ip=@ method=GET url=/1.0 user=
DBUG[10-06|12:43:41] Handling ip=@ method=GET url=/1.0 user=
from source
root@cpu-5228:~#
root@cpu-5228:~# lxc monitor --type=logging --pretty
DBUG[10-06|12:43:15] New event listener: fc2ee2f5-73d4-497b-b9c2-60996797a6ab
DBUG[10-06|12:43:28] Handling ip=@ method=GET url=/1.0 user=
DBUG[10-06|12:43:28] Handling method=GET url=/1.0 user= ip=@
DBUG[10-06|12:43:31] Found cert name=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Handling ip=54.37.245.179:40648 method=GET url=/1.0 user=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Found cert name=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] New event listener: b44aed43-01fa-4500-9a59-6359356aaca0
DBUG[10-06|12:43:31] Handling ip=54.37.245.179:40650 method=GET url=/1.0/events user=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Found cert name=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Found cert name=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Handling ip=54.37.245.179:40652 method=POST url=/1.0/instances user=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Responding to instance create
INFO[10-06|12:43:31] Creating container ephemeral=false name=test project=default
INFO[10-06|12:43:31] Created container ephemeral=false name=test project=default
DBUG[10-06|12:43:31] New websocket Operation: 5486310d-f5c8-4938-944d-be27d44a5b99
DBUG[10-06|12:43:31] Started websocket operation: 5486310d-f5c8-4938-944d-be27d44a5b99
DBUG[10-06|12:43:31] Found cert name=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Handling ip=54.37.245.179:40654 method=GET url="/1.0/operations/5486310d-f5c8-4938-944d-be27d44a5b99/websocket?secret=c5a5ea767efe330bacf01d818855d0054a8cea23dc5cff2115766f301abccbd4" user=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Handled websocket Operation: 5486310d-f5c8-4938-944d-be27d44a5b99
DBUG[10-06|12:43:31] Connected websocket Operation: 5486310d-f5c8-4938-944d-be27d44a5b99
DBUG[10-06|12:43:31] Connected websocket Operation: 5486310d-f5c8-4938-944d-be27d44a5b99
DBUG[10-06|12:43:31] Handling method=GET url="/1.0/operations/5486310d-f5c8-4938-944d-be27d44a5b99/websocket?secret=31ecb195b145481d04c81184eff1e2c2c50424699f28151731adc5c95d50bb21" user=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930 ip=54.37.245.179:40656
DBUG[10-06|12:43:31] Handled websocket Operation: 5486310d-f5c8-4938-944d-be27d44a5b99
DBUG[10-06|12:43:31] Found cert name=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Found cert name=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Handled websocket Operation: 5486310d-f5c8-4938-944d-be27d44a5b99
DBUG[10-06|12:43:31] Connected websocket Operation: 5486310d-f5c8-4938-944d-be27d44a5b99
DBUG[10-06|12:43:31] Handling ip=54.37.245.179:40658 method=GET url="/1.0/operations/5486310d-f5c8-4938-944d-be27d44a5b99/websocket?secret=55d70ac5b7c03ae9180b7c33bdd1fc9523a681c460a62cd4466957db9183b237" user=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] CreateInstanceFromMigration started driver=zfs instance=test pool=default project=default args="{test map[] [] {ZFS [compress]} false false true 0}"
DBUG[10-06|12:43:31] About to receive rsync
DBUG[10-06|12:43:31] Handling ip=54.37.245.179:40660 method=GET url=/1.0/operations/5486310d-f5c8-4938-944d-be27d44a5b99 user=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:31] Found cert name=6b701ed2713e018ec932914da0b3c15ed3e6f14718ca18db45af4be822742930
DBUG[10-06|12:43:33] CreateInstanceFromMigration finished args="{test map[] [] {ZFS [compress]} false false true 0}" driver=zfs instance=test pool=default project=default
DBUG[10-06|12:43:33] Done receiving from rsync
DBUG[10-06|12:43:33] About to receive header
DBUG[10-06|12:43:33] About to receive rsync
DBUG[10-06|12:43:33] About to receive header
DBUG[10-06|12:43:33] Done receiving from rsync
INFO[10-06|12:43:35] Migrating container features=0 name=test project=default used="1970-01-01 00:00:00 +0000 UTC" statedir=/tmp/lxd_restore_055338199 stop=false actionscript=false created="2020-10-06 12:43:31.718262266 +0000 UTC" ephemeral=false predumpdir=
DBUG[10-06|12:43:35] Scheduler: network: veth482ba876 has been added: updating network priorities
DBUG[10-06|12:43:35] Scheduler: network: veth9f4df22c has been added: updating network priorities
DBUG[10-06|12:43:35] MountInstance started pool=default project=default driver=zfs instance=test
DBUG[10-06|12:43:35] MountInstance finished driver=zfs instance=test pool=default project=default
DBUG[10-06|12:43:35] UpdateInstanceBackupFile started driver=zfs instance=test pool=default project=default
DBUG[10-06|12:43:35] Mounted ZFS dataset dev=default/containers/test driver=zfs path=/var/lib/lxd/storage-pools/default/containers/test pool=default
DBUG[10-06|12:43:35] UpdateInstanceBackupFile finished driver=zfs instance=test pool=default project=default
DBUG[10-06|12:43:35] MountInstance started instance=test pool=default project=default driver=zfs
DBUG[10-06|12:43:35] MountInstance finished driver=zfs instance=test pool=default project=default
DBUG[10-06|12:43:35] Got error reading migration control socket read tcp 164.132.202.62:8448->54.37.245.179:40654: use of closed network connection
EROR[10-06|12:43:35] Error collecting checkpoint log file err="lstat /tmp/lxd_restore_055338199/final/restore.log: no such file or directory"
INFO[10-06|12:43:35] Deleting container project=default used="1970-01-01 00:00:00 +0000 UTC" created="2020-10-06 12:43:31.718262266 +0000 UTC" ephemeral=false name=test
DBUG[10-06|12:43:35] DeleteInstance started pool=default project=default driver=zfs instance=test
DBUG[10-06|12:43:35] Deleting instance volume pool=default project=default volName=test driver=zfs instance=test
DBUG[10-06|12:43:36] DeleteInstance finished driver=zfs instance=test pool=default project=default
INFO[10-06|12:43:36] Deleted container created="2020-10-06 12:43:31.718262266 +0000 UTC" ephemeral=false name=test project=default used="1970-01-01 00:00:00 +0000 UTC"
DBUG[10-06|12:43:36] Failure for websocket operation: 5486310d-f5c8-4938-944d-be27d44a5b99: Error transferring instance data: Failed to run: /root/go/bin/lxd forkmigrate test /var/lib/lxd/containers /var/log/lxd/test/lxc.conf /tmp/lxd_restore_055338199/final true:
DBUG[10-06|12:43:36] Event listener finished: b44aed43-01fa-4500-9a59-6359356aaca0
DBUG[10-06|12:43:36] Disconnected event listener: b44aed43-01fa-4500-9a59-6359356aaca0
DBUG[10-06|12:44:28] Handling ip=@ method=GET url=/1.0 user=
DBUG[10-06|12:44:28] Handling ip=@ method=GET url=/1.0 user=
from destination
Can you look at what’s in /var/log/lxd/test/ on the destination?
root@cpu-5228:/var/log/lxd/test# tail lxc.log
lxc test 20201006124335.823 ERROR criu - criu.c:criu_ok:894 - Found un-dumpable network: phys (eth0)```root@cpu-5228:/var/log/lxd/test# ls
lxc.conf lxc.log lxc.log.old
Ok, so that’s the usual issue then, containers with network interfaces cannot currently be dumped.
We have planned work that @brauner will be doing over the coming months to fix such issues.
okay , we need to wait for a fix then ?
Yeah, there are two main issues with live migration right now which we’ll be working on:
- A container with a
nicdevice attached cannot be migrated - Containers using apparmor namespacing cannot be migrated
For the former, it’s been okay for some users to detach the nic, move the container, re-attach the nic and then manually reconfigure the address, routing, … on it, but that’s really a huge workaround and quite annoying.
The apparmor namespacing issue prevents a lot of even empty containers based on modern distros from migrating.
3 Likes
okay Thank you for your help @stgraber 