Copying/Migrating lxc vm from LXD host 5.2 & ZFS storage to LXD host 5.2 & ZFS storage - Failing

,

Hello all,

I am trying to migrate the lxc vm named: ansible from the lxd host kumo2 to lxd host lxc1

I followed the instructions here: https://www.cyberciti.biz/faq/how-to-movemigrate-lxd-vm-to-another-host-on-linux/ “Method # 2 LXD VM container migration using LXD API and Simplestreams”

CONTEXT:

LXD host: kumo2
lxc-vm to migrate: ansible
Version: 5.2-79c3c3b
Storage:
kumo2:~$ lxc storage list
±------±-------±-----------------------------------------±------------±--------±--------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY | STATE |
±------±-------±-----------------------------------------±------------±--------±--------+
| DATA1 | zfs | /var/snap/lxd/common/lxd/disks/DATA1.img | | 4 | CREATED |
±------±-------±-----------------------------------------±------------±--------±--------+

LXD host: lxc1
Version: 5.2-79c3c3b
Storage:
lxc1:~$ lxc storage list
±------±-------±-------±------------±--------±--------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY | STATE |
±------±-------±-------±------------±--------±--------+
| RAID1 | zfs | RAID1 | | 9 | CREATED |
±------±-------±-------±------------±--------±--------+

The set up seems to be fine no errors there, but when I try to migrate the snapshot I get this message:

kumo2:~$ lxc copy ansible/snap0 lxc1:ansible --verbose
Error: Failed instance creation: Error transferring instance data: Failed decoding migration header: invalid character ‘\x00’ looking for beginning of value

Any suggestions or comments will be really appreciated, in the meantime I will see if there are other methods that allow me to complete the migration.

Hello all,

I tried following the configuration again and this time I got this message:

kumo2:~$ lxc copy ansible/snap0 lxc1:ansible
Error: Failed instance creation: Error transferring instance data: websocket: close 1000 (normal)

I am not sure what’s causing it, but I will continuing investigating.

Any suggestions or comments will be greatly appreciated.

Hello all,

When trying to use a different method of migration, publishing the image/snap I get this:

kumo2:~$ lxc publish ansible/snap0 --alias ansible-image
Error: Failed getting disk path: Could not locate a zvol for DATA1/virtual-machines/ansible.block@snapshot-snap0

That message about the storage at the source LXD host concerns me…

Here is what I see on the source LXD host profile:

kumo2:~$ lxc profile show default
config:
boot.autostart: “false”
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: DATA1
type: disk
name: default
used_by:

  • /1.0/instances/ansible

I will continue investigating.

There have been numerous issues with optimized migration since optimized refresh was added (see https://github.com/lxc/lxd/issues/10186).

Are both source and target running same LXD version?

There have been several fixes added since LXD 5.2 was released, so perhaps this will be fixed in LXD 5.3.

Hello @tomp,

Thank you for the update, as you can see in the start of this thread I shared the context:

CONTEXT:

LXD host: kumo2
lxc-vm to migrate: ansible
Version: 5.2-79c3c3b
Storage:
kumo2:~$ lxc storage list
±------±-------±-----------------------------------------±------------±--------±--------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY | STATE |
±------±-------±-----------------------------------------±------------±--------±--------+
| DATA1 | zfs | /var/snap/lxd/common/lxd/disks/DATA1.img | | 4 | CREATED |
±------±-------±-----------------------------------------±------------±--------±--------+

LXD host: lxc1
Version: 5.2-79c3c3b
Storage:
lxc1:~$ lxc storage list
±------±-------±-------±------------±--------±--------+
| NAME | DRIVER | SOURCE | DESCRIPTION | USED BY | STATE |
±------±-------±-------±------------±--------±--------+
| RAID1 | zfs | RAID1 | | 9 | CREATED |
±------±-------±-------±------------±--------±--------+

I am not sure about that error message I am getting on the destination LXD host:

kumo2:~$ lxc publish ansible/snap0 --alias ansible-image
Error: Failed getting disk path: Could not locate a zvol for DATA1/virtual-machines/ansible.block@snapshot-snap0

I am also in irc.libera.chat willing to investigate and troubleshoot, this might be related to the way the storage pool is created on my LXD hosts?

Sincerely,

Hello @tomp,

Any workaround I can follow in order to:

  • Backup the lxc VM from that LXD host.
  • Remove the full configuration of LXD from the Ubuntu 20.04 LXD host and convert the host to a container using p2c tool.

I am trying to migrate the lxc VM out of that host and concert the host from LDX host into a normal Ubtuntu 20.04 lxc container, and then re-install Ubuntu 24.04 LTS in that box.

I may end up backing up the whole server including the lxc VM or try to convert it into a lxc container with a nested LXD configuration and lxc VM running in it.

I look forward to any suggestions.

Sincerely,

You could do lxc export transfer the file to the new host and do lxc import there.

Also you could try upgrading to the edge snap on both sides and see if any of the recent fixes resolve it:

snap refresh lxd --channel=latest/edge
1 Like

Alright, I will try the lxc export method first, but I believe that is what I tried and got different errors I listed on the thread. I will then upgrade both LXD servers and try again.

Any suggestions regarding the lxd-p2c options?

Thank you for all you do!

Right,

This is the error I get when trying to export the lxc VM in the source LXD host:

kumo2:~$ lxc export ansible ansible.tar.gz
Error: Create backup: Backup create: Error getting VM block volume disk path: Could not locate a zvol for DATA1/virtual-machines/ansible.block@snapshot-snap0

I will go with the snap refresh option and try the first method of:

LXD VM container migration using LXD API and Simplestreams

Let’s see how it goes,

Hmm, no luck.

source

kumo2:~$ sudo snap refresh lxd --channel=latest/edge
lxd (edge) git-d139b55 from Canonical✓ refreshed

destination

lxc1:~$ sudo snap refresh lxd --channel=latest/edge
lxd (edge) git-d139b55 from Canonical✓ refreshed

lxc1:~$ lxc config set core.https_address 192.168.0.10:8443
lxc config set core.trust_password secretpassword

remote LXD host added to source

kumo2:~$ lxc remote add lxc1 192.168.0.10
Certificate fingerprint: 56d897cc1e84e73d68b8380344633481d995b5c8c69579781ebc75da38da436f
ok (y/n/[fingerprint])? y

kumo2:~$ lxc remote list

±----------------±-----------------------------------------±--------------±------------±-------±-------±-------+
| NAME | URL | PROTOCOL | AUTH TYPE | PUBLIC | STATIC | GLOBAL |
±----------------±-----------------------------------------±--------------±------------±-------±-------±-------+
| images | https://images.linuxcontainers.org | simplestreams | none | YES | NO | NO |
±----------------±-----------------------------------------±--------------±------------±-------±-------±-------+
| local (current) | unix:// | lxd | file access | NO | YES | NO |
±----------------±-----------------------------------------±--------------±------------±-------±-------±-------+
| lxc1 | https://192.168.0.10:8443 | lxd | tls | NO | NO | NO |
±----------------±-----------------------------------------±--------------±------------±-------±-------±-------+
| ubuntu | Ubuntu Cloud Images | simplestreams | none | YES | YES | NO |
±----------------±-----------------------------------------±--------------±------------±-------±-------±-------+
| ubuntu-daily | Ubuntu Cloud Images | simplestreams | none | YES | YES | NO |
±----------------±-----------------------------------------±--------------±------------±-------±-------±-------+

kumo2:~$ lxc list

±--------±--------±-----±-----±----------------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±--------±--------±-----±-----±----------------±----------+
| ansible | STOPPED | | | VIRTUAL-MACHINE | 1 |
±--------±--------±-----±-----±----------------±----------+

migration command

kumo2:~$ lxc copy ansible/snap0 lxc1:ansible --verbose
Error: Failed instance creation: Error transferring instance data: websocket: close 1000 (normal)

Any ideas?

So focusing on the lxc export problem atm:

Can you show:

  • lxc info ansible
  • lxc storage volume ls <pool> | grep ansible
  • sudo zfs list -t snapshot | grep ansible

So we can ascertain what we have in the instance snapshot records, instance volume snapshot records and the actual snapshots on disk.

1 Like

kumo2:~$ lxc info ansible

Name: ansible
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2022/05/30 22:41 JST
Last Used: 2022/05/31 12:11 JST

Snapshots:
±------±---------------------±-----------±---------+
| NAME | TAKEN AT | EXPIRES AT | STATEFUL |
±------±---------------------±-----------±---------+
| snap0 | 2022/06/13 09:06 JST | | NO |
±------±---------------------±-----------±---------+

kumo2:~$ lxc storage volume ls DATA1 | grep ansible

| virtual-machine | ansible | | block | 1 |
| virtual-machine (snapshot) | ansible/snap0 | | block | 1 |

kumo2:~$ sudo zfs list -t snapshot | grep ansible

DATA1/virtual-machines/ansible@snapshot-snap0 15.5K - 5.87M -
DATA1/virtual-machines/ansible.block@migration-5a3e5b67-bda8-4e8a-b49d-daade77a4df1 0B - 2.71G -
DATA1/virtual-machines/ansible.block@snapshot-snap0 0B - 2.71G -

Thank you in advance,

Can you remove this one please, I’m wondering if its preventing the other ones from being activated?

1 Like

Sounds good, removed please see below:

kumo2:~$ sudo zfs destroy DATA1/virtual-machines/ansible.block@migration-5a3e5b67-bda8-4e8a-b49d-daade77a4df1

Now I see this:

kumo2:~$ sudo zfs list -t snapshot | grep ansible
DATA1/virtual-machines/ansible@snapshot-snap0 15.5K - 5.87M -
DATA1/virtual-machines/ansible.block@snapshot-snap0 1K - 2.71G -

I tried exporting again and got a similar error:

kumo2:~$ lxc export ansible ansible.tar.gz
Error: Create backup: Backup create: Error getting VM block volume disk path: Could not locate a zvol for DATA1/virtual-machines/ansible.block@snapshot-snap0

Hmm,

Can you try rebooting that machine and trying again, I want to rule out an issue with the mount namespace inside the snap becoming out of sync with the host.

1 Like

Alright,

Here is what I am seeing now:

kumo2:~$ sudo zfs list -t snapshot | grep ansible
DATA1/virtual-machines/ansible@snapshot-snap0 15.5K - 5.87M -
DATA1/virtual-machines/ansible.block@snapshot-snap0 1K - 2.71G -

kumo2:~$ lxc export ansible ansible.tar.gz
Backing up instance: 46.69MB (1.31MB/s)

I like how it changed from just showing the size as you can see above, then to:

kumo2:~$ lxc export ansible ansible.tar.gz
Exporting the backup: 91% (34.43MB/s)

and then:

kumo2:~$ lxc export ansible ansible.tar.gz
Backup exported successfully!

Looking better,

So it looks like reboot fixed the lxc export case.
Likely this is caused by a known issue related to ZFS and snap packaging:

To answer your question about physical machine migration, there is a tool called lxd-migrate that you run on the machine/VM you want to migrate into LXD and specify the address of a LXD server that is configured to listen on the network.

See What's new in LXD 4.23? - YouTube

There’s no dedicated lxd-migrate package currently, but you can download a fresh build from the build assets of any pull-request, e.g. doc: add link to video about network forwards · lxc/lxd@09d8bb6 · GitHub

1 Like

Excellent @tomp,

I am still waiting for the export to complete before I attempt the migration following the LXD API method, and if that fails I will report the outcome here, and try to import the exported lxc VM into the destination LXD host manually.

I will keep you posted, and thank you for the references, those are really useful!

Hello @tomp,

It worked!

destination lxd host:

lxc1:~$ lxc list
±--------±--------±-----±-----±----------------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±--------±--------±-----±-----±----------------±----------+
| ansible | STOPPED | | | VIRTUAL-MACHINE | 0 |
±--------±--------±-----±-----±----------------±----------+
| c1 | STOPPED | | | CONTAINER | 0 |
±--------±--------±-----±-----±----------------±----------+
| c2 | STOPPED | | | CONTAINER | 0 |
±--------±--------±-----±-----±----------------±----------+
| vm1 | STOPPED | | | VIRTUAL-MACHINE | 0 |
±--------±--------±-----±-----±----------------±----------+
| vm2 | STOPPED | | | VIRTUAL-MACHINE | 0 |
±--------±--------±-----±-----±----------------±----------+
| vm3 | STOPPED | | | VIRTUAL-MACHINE | 0 |
±--------±--------±-----±-----±----------------±----------+
| vm4 | STOPPED | | | VIRTUAL-MACHINE | 0 |
±--------±--------±-----±-----±----------------±----------+

lxc1:~$ lxc config show ansible
architecture: x86_64
config:
image.architecture: amd64
image.description: Ubuntu jammy amd64 (20220530_08:32)
image.os: Ubuntu
image.release: jammy
image.serial: “20220530_08:32”
image.type: disk-kvm.img
image.variant: default
limits.cpu: “1”
limits.memory: 2GB
volatile.apply_template: copy
volatile.base_image: dddb4c8d1b25ad693722babbc4726da56ccfcb307859b59070206f78547fcd13
volatile.cloud-init.instance-id: 82430879-382e-4d1c-9505-7b9c287856ed
volatile.eth0.hwaddr: 00:16:3e:9f:18:5e
volatile.uuid: 433a1988-e492-4a8b-b78b-b02d0a55d22e
devices:
root:
path: /
pool: RAID1
size: 20GiB
type: disk
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”

Thank you so much I can mark this one as solved.

I really appreciate all your time and help!

1 Like