Hi all,
Setup:
- Debian bullseye
- snap LXD (5.0.2)
- btrfs-progs v5.10.1
I have a problem with lxc copy on btrfs.
I copy my container to another host with:
lxc snapshot host:container snap_name
lxc copy host:container/snap_name snap_name_local
I managed to successfully make one copy but the rest of the times (tried at least 20 times, probably more) I end up with the following error:
Error: Failed instance creation: Error transferring instance data: Got error reading migration source: read tcp 1.2.3.4:32970->1.2.3.5:8443: read: connection timed out
Using: lxc mointor
I get:
Failed sending volume ideaprod/ideaprod-20230206-143040:/: Btrfs send failed: [signal: killed write tcp 1.2.3.5:8443->1.2.3.4:32978: write: broken pipe] (At subvol /var/snap/lxd/common/lxd/storage-pools/default/containers/migration.3863231056/.migration-send
I am not able to find any other errors, not in dmesg and there are no errors reported by btrfs.
On both hosts:
btrfs scrub start -B /var/snap/lxd β No errors
I have the same setup on another host and lxc copy is done every night and that never fails (the containers are extremely small though).
The size of this container is: 2.65MiB
I did a test with just btrfs, created a lxc snapshot and used btrfs send/reveive.
I tried that twice and both of the times it worked fine, no timeout.
btrfs send /var/snap/lxd/common/lxd/storage-pools/default/containers/my_snap | ssh host btrfs receive /var/snap/lxd/common/lxd/storage-pools/default/containers/
Is there some type of timeout that happens in LXD?
As I do not seem to be able to find any other errors I really struggle here to try to figure this one out.
Many thanks in advance!
I did some more testing using lxc monitoring and it seems that it stops on the exact same place every time. Every time there is the same message before the first connection reset by peer.
message: MigrateInstance finished
Here is the last part of the lxc monitor output:
location: none
metadata:
context:
args: β&{IndexHeaderVersion:1 Name:ideaprod/ideaprod-20230207-134402 Snapshots:[]
MigrationType:{FSType:BTRFS Features:[migration_header header_subvolumes header_subvolume_uuids]}
TrackProgress:true MultiSync:false FinalSync:false Data: ContentType: AllowInconsistent:false
Refresh:false Info:0xc000014a40 VolumeOnly:true}β
instance: ideaprod/ideaprod-20230207-134402
project: default
level: debug
message: MigrateInstance finished
timestamp: β2023-02-07T13:44:51.124827787+01:00β
type: logging
location: none
metadata:
context:
err: |-
Failed sending volume ideaprod/ideaprod-20230207-134402:/: Btrfs send failed: [signal: killed write tcp 1.2.3.4:8443->1.2.3.5:35860: write: connection reset by peer] (At subvol /var/snap/lxd/common/lxd/st
orage-pools/default/containers/migration.1812165211/.migration-send
)
instance: ideaprod/ideaprod-20230207-134402
project: default
level: error
message: Migration failed on source
timestamp: β2023-02-07T13:44:51.125105799+01:00β
type: logging
location: none
metadata:
context:
instance: ideaprod/ideaprod-20230207-134402
project: default
level: info
message: Migration channels disconnected on source
timestamp: β2023-02-07T13:44:51.125307033+01:00β
type: logging
location: none
metadata:
context:
class: websocket
description: Transferring snapshot
err: |-
Failed sending volume ideaprod/ideaprod-20230207-134402:/: Btrfs send failed: [signal: killed write tcp 1.2.3.4:8443->1.2.3.5:35860: write: connection reset by peer] (At subvol /var/snap/lxd/common/lxd/st
orage-pools/default/containers/migration.1812165211/.migration-send
)
operation: fdf4df78-d7cd-4f13-85d0-e69a82fc2373
project: default
level: debug
message: Failure for operation
timestamp: β2023-02-07T13:44:51.125386762+01:00β
type: logging
location: none
metadata:
class: websocket
created_at: β2023-02-07T13:44:04.168983296+01:00β
description: Transferring snapshot
err: |-
Failed sending volume ideaprod/ideaprod-20230207-134402:/: Btrfs send failed: [signal: killed write tcp 1.2.3.4:8443->1.2.3.5:35860: write: connection reset by peer] (At subvol /var/snap/lxd/common/lxd/stor
age-pools/default/containers/migration.1812165211/.migration-send
)
id: fdf4df78-d7cd-4f13-85d0-e69a82fc2373
location: none
may_cancel: false
metadata:
control: f3008a4d5464c1310bc14f7e5adcc5c52202a2ca3ea5c18072252166ba057d6a
fs: 05ae55ee9a051f9b54ae488de91c36b5d0bea8593544a4423088152aacb11843
fs_progress: βideaprod/ideaprod-20230207-134402: 5.16GB (114.51MB/s)β
resources:
containers:
- /1.0/containers/ideaprod
instances:
- /1.0/instances/ideaprod
status: Failure
status_code: 400
updated_at: β2023-02-07T13:44:50.46370078+01:00β
project: default
timestamp: β2023-02-07T13:44:51.125490093+01:00β
type: operation