Cannot copy container

Hello,

I cannot copy container, once i do

lxc copy hostname:container-name/snapshot local:container-name

and i do lxc list the container is in the list but the copy status only hangs forever, i left the command running for more than 12 hours and it is still hanging. Before i did copy some other containers and it was quite quick but now i’m unable to do so.

Also when i do
ll /var/snap/lxd/common/lxd/containers/
ll /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/Pool-1/containers/
ll /var/snap/lxd/common/lxd/storage-pools/Pool-1/

There’s no container that i’m trying to copy.

Thanks in advance!

What version of LXD on source and target and what storage driver on source and target?

Hello @stgraber,

Thanks for the quick reply!

I’m not sure if that helps but the cause of my issue is related to my previous post since everything started when i interrupted the processes from my source container

LXD version on source: 4.4
LXD version on target: 4.4
storage driver from source: btrfs
storage driver from target: btrfs

Ok, so I’d recommend to:

  • Check ps aux | grep btrfs on source and target to make sure no running send/receive are left running
  • systemctl reload snap.lxd.daemon on both sides to be sure there’s no funky state left around
  • Run lxc monitor --type=logging --pretty on both side (keep running in a terminal)
  • Run the copy again.

This should give us some clearer output by seeing what’s going on on both sides.

You may also want to check dmesg on source and target to make sure there’s nothing wrong going on with btrfs itself.

Hi @stgraber,

Sorry for the late reply,
I left the copy command running the whole weekend. However, it keeps hanging. I reran everything again and these are the outputs.

This is the output of ps aux | grep btrfs from the target:

root      1877  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-worker]
root      1878  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-worker-hi]
root      1879  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-delalloc]
root      1880  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-flush_del]
root      1881  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-cache]
root      1882  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-submit]
root      1883  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-fixup]
root      1884  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio]
root      1885  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-met]
root      1886  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-met]
root      1887  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-rai]
root      1888  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-rep]
root      1889  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-rmw]
root      1890  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-wri]
root      1891  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-freespace]
root      1892  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-delayed-m]
root      1893  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-readahead]
root      1894  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-qgroup-re]
root      1895  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-extent-re]
root      1897  0.0  0.0      0     0 ?        S    Aug19   0:00 [btrfs-cleaner]
root      1898  0.0  0.0      0     0 ?        S    Aug19   0:09 [btrfs-transacti]
cesar    15043  0.0  0.0  13136  1028 pts/2    S+   10:20   0:00 grep --color=auto btrfs

This is the output of ps aux | grep btrfs from the source:

root      1390  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-worker]
root      1391  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-worker-hi]
root      1392  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-delalloc]
root      1393  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-flush_del]
root      1394  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-cache]
root      1395  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-submit]
root      1396  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-fixup]
root      1397  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio]
root      1398  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-met]
root      1399  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-met]
root      1400  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-rai]
root      1401  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-rep]
root      1402  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-rmw]
root      1403  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-endio-wri]
root      1404  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-freespace]
root      1405  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-delayed-m]
root      1406  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-readahead]
root      1407  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-qgroup-re]
root      1408  0.0  0.0      0     0 ?        I<   Aug19   0:00 [btrfs-extent-re]
root      1489  0.0  0.0      0     0 ?        S    Aug19   2:45 [btrfs-cleaner]
root      1490  3.8  0.0      0     0 ?        S    Aug19 658:50 [btrfs-transacti]
cesar    31525  0.0  0.0  13136  1092 pts/1    S+   10:21   0:00 grep --color=auto btrf

This is the output of lxc monitor --type=logging --pretty from target:

DBUG[08-31|11:07:19] New event listener: 0d051589-48a5-4439-bf60-62c4c35135a3 
DBUG[08-31|11:07:43] Handling                                 ip=@ method=GET url=/1.0 user=
DBUG[08-31|11:07:43] Handling                                 ip=@ method=GET url="/1.0/instances?recursion=1" user=
DBUG[08-31|11:07:43] Handling                                 user= ip=@ method=GET url=/1.0
DBUG[08-31|11:07:43] Handling                                 user= ip=@ method=GET url="/1.0/instances?recursion=1"
DBUG[08-31|11:07:43] Handling                                 ip=@ method=GET url=/1.0 user=
DBUG[08-31|11:07:44] Handling                                 ip=@ method=GET url=/1.0/events user=
DBUG[08-31|11:07:44] New event listener: 16d8fda5-1f5f-4694-8ad6-19e6816dca94 
DBUG[08-31|11:07:44] Responding to instance create 
DBUG[08-31|11:07:44] Handling                                 ip=@ method=POST url=/1.0/instances user=
INFO[08-31|11:07:44] Creating container                       name=container-name project=default ephemeral=false
INFO[08-31|11:07:44] Created container                        ephemeral=false name=container-name project=default
DBUG[08-31|11:07:44] New task Operation: 4c8ba937-2e16-4492-a416-4e0dfcce426b 
DBUG[08-31|11:07:44] Started task operation: 4c8ba937-2e16-4492-a416-4e0dfcce426b 
DBUG[08-31|11:07:44] Handling                                 ip=@ method=GET url=/1.0/operations/4c8ba937-2e16-4492-a416-4e0dfcce426b user=
DBUG[08-31|11:07:44] CreateInstanceFromMigration started      pool=Pool-1 project=default args="{instance-name  map[] [] {BTRFS [migration_header header_subvolumes]} false false false 0 }" driver=btrfs instance=instance-name
DBUG[08-31|11:07:56] Received migration meta data header      driver=btrfs name=container-name pool=Pool-1
DBUG[08-31|11:07:56] Receiving volume                         driver=btrfs name=container-name path=/var/snap/lxd/common/lxd/storage-pools/Pool-1/containers/container-name pool=Pool-1 receivePath=/var/snap/lxd/common/lxd/storage-pools/Pool-1/containers/migration.286674634
DBUG[08-31|11:22:57] New task Operation: 15c83655-6ed5-41ba-ae44-5cdf99aff921 
DBUG[08-31|11:22:57] Started task operation: 15c83655-6ed5-41ba-ae44-5cdf99aff921 
INFO[08-31|11:22:57] Pruning expired instance backups 
INFO[08-31|11:22:57] Done pruning expired instance backups 
DBUG[08-31|11:22:57] Success for task operation: 15c83655-6ed5-41ba-ae44-5cdf99aff921 

This is the output of lxc monitor --type=logging --pretty from source:

DBUG[08-31|11:07:23] New event listener: 5b933436-384a-4330-889d-afcb218eb1f0 
DBUG[08-31|11:07:43] Handling                                 ip=0.0.0.0:59860 method=GET url=/1.0 user=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:43] Found cert                               name=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:43] Found cert                               name=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:44] Found cert                               name=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:44] Handling                                 user=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57 ip=0.0.0.0:59862 method=GET url=/1.0/instances/instance-name/snapshots/snapshot-name
DBUG[08-31|11:07:44] GetInstanceUsage started                 driver=btrfs instance=instance-name/snapshot-name pool=Pool-1 project=default
DBUG[08-31|11:07:44] GetInstanceUsage finished                instance=instance-name/snapshot-name pool=Pool-1 project=default driver=btrfs
DBUG[08-31|11:07:44] Found cert                               name=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:44] Handling                                 ip=0.0.0.0:59864 method=GET url=/1.0/instances/instance-name user=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:44] Found cert                               name=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:44] Handling                                 ip=0.0.0.0:59866 method=GET url=/1.0/events user=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:44] New event listener: 09b86511-db9f-4b41-b75e-91a96557624a 
DBUG[08-31|11:07:44] Handling                                 method=POST url=/1.0/instances/instance-name/snapshots/snapshot-name user=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57 ip=0.0.0.0:59868
DBUG[08-31|11:07:44] Found cert                               name=5859fa11a2866401559a347196235570bde566607e52d11f2737d30c68d7dd57
DBUG[08-31|11:07:44] 
	{
		"name": "container-name",
		"migration": true,
		"target": null
	} 
DBUG[08-31|11:07:44] New websocket Operation: c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7 
DBUG[08-31|11:07:44] Started websocket operation: c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7 
DBUG[08-31|11:07:44] 
	{
		"type": "async",
		"status": "Operation created",
		"status_code": 100,
		"operation": "/1.0/operations/c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7",
		"error_code": 0,
		"error": "",
		"metadata": {
			"id": "c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7",
			"class": "websocket",
			"description": "Transferring snapshot",
			"created_at": "2020-08-31T11:07:44.259578398-04:00",
			"updated_at": "2020-08-31T11:07:44.259578398-04:00",
			"status": "Running",
			"status_code": 103,
			"resources": {
				"containers": [
					"/1.0/containers/container-name"
				]
			},
			"metadata": {
				"control": "26d9cd2237959353924a5e1208dff51b6dc050cddcedc3e2f78c3134a1ab4c1c",
				"fs": "a8bfc059e03212ee2f1a6556f9d5d0b8025f8aec6be3858943d8196e7c048733"
			},
			"may_cancel": false,
			"err": "",
			"location": "none"
		}
	} 
DBUG[08-31|11:07:44] Connected websocket Operation: c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7 
DBUG[08-31|11:07:44] Handled websocket Operation: c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7 
DBUG[08-31|11:07:44] Allowing untrusted GET                   ip=0.0.0.0:59870 url="/1.0/operations/c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7/websocket?secret=26d9cd2237959353924a5e1208dff51b6dc050cddcedc3e2f78c3134a1ab4c1c"
DBUG[08-31|11:07:44] Allowing untrusted GET                   ip=0.0.0.0:59872 url="/1.0/operations/c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7/websocket?secret=a8bfc059e03212ee2f1a6556f9d5d0b8025f8aec6be3858943d8196e7c048733"
DBUG[08-31|11:07:44] Handled websocket Operation: c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7 
DBUG[08-31|11:07:44] Connected websocket Operation: c2cbb59b-c46c-48e7-9aa5-c41bd2bc19a7 
DBUG[08-31|11:07:44] MigrateInstance started                  instance=instance-name/snapshot-name pool=Pool-1 project=default args="&{instance-name/snapshot-name [] {BTRFS [migration_header header_subvolumes]} true false false <nil> }" driver=btrfs
DBUG[08-31|11:07:56] Sent migration meta data header          driver=btrfs name=instance-name/snapshot-name pool=Pool-1
DBUG[08-31|11:08:00] Sending subvolume                        pool=Pool-1 source=/var/snap/lxd/common/lxd/storage-pools/Pool-1/containers/migration.532287686/.migration-send driver=btrfs name=instance-name/snapshot-name parent= path=/

Thanks for your help!

@tomp can you take over on this one?
Otherwise it’d have to wait almost a week for me to get back to it.

Yep will take over.

I’ve gone through and quoted the output messages as it makes it a lot easier to read and parse. Looking into this now.

So I’ve just tested this with the LXD 4.4 snap on Ubuntu Focal using a BTRFS storage pool on both sides:

On node v1:

lxc config set core.https_address [::]:8443
lxc config set core.trust_password pw

lxc launch images:alpine/3.12 c1
lxc snapshot c1

On node v2:

lxc config set core.https_address [::]:8443
lxc config set core.trust_password pw
lxc remote add v1 10.135.120.69

lxc copy v1:c1/snap0 local:c2

This completed OK.

So I’m happy its working in a general sense.

Have you tried running the command from the source to the target, i.e on the source node:

lxc copy container-name/snapshot target:container-name

Hello @tomp

Thank you for your help. A reboot from the target fixed the issue.

Again, I really appreciate your help.

1 Like