How to move LXC Container on another partition

Hello there, my computer has two partitions, one to store the OS and libraries and another partition for data storage aka the larger partition. For reference, one partition is 100GB and the other one is 1TB.

I need to spin a large cluster therefore I need my containers to reside on the larger partition and take advantage of the storage there but I don’t know how to do that. LXD is installed on the base partition and every container I spin resides in that partition. I am using the dir storage method for my lxd.

Is there a command for me to migrate my containers from one partition to the other?

I tried to change my default profile. root.path: / was changed to root.path: /home/containers/ however this change is not accepted and throws me the error:

Config parsing error: Device validation failed for “root”: Disk entry is missing the required “source” or “path” property
Press enter to open the editor again or ctrl+c to abort change


config:
  limits.cpu: "6"
  limits.memory: 30GB
description: Default LXD profile
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /home/containers/
    pool: default
    size: 100GB
    type: disk
name: default

What else can I do to migrate or at least be able to create my VMs on my other bigger partition?
Any help is greatly appreciated.

1 Like

Hi cli0,

I was having the same challenge and my need was exactly the same as yours. I understand from your description, that you ran lxd init and you went with the option for local directory storage, right?

1- Anyways, even if this is not the case, let’s see how to create a new storage pool

$ lxc storage list

The above command will show you the storage pools available for your containers and VMs. In my setup, the output is

+----------+--------+------------------------------------------------+-------------+---------+---------+
|   NAME   | DRIVER |                     SOURCE                     | DESCRIPTION | USED BY |  STATE  |
+----------+--------+------------------------------------------------+-------------+---------+---------+
| default  | dir    | /var/snap/lxd/common/lxd/storage-pools/default |             | 1       | CREATED |
+----------+--------+------------------------------------------------+-------------+---------+---------+
| zfs-pool | zfs    | zfs-pool                                       |             | 9       | CREATED |
+----------+--------+------------------------------------------------+-------------+---------+---------+

So I have 3 storage pools that I can use with my profiles or I can use directly in the command line when launching a new instance (container or VM).

Ref: https://linuxcontainers.org/lxd/docs/master/howto/storage_pools/
You can create a new storage pool in command line as mentioned in the doc link above, choose the driver you prefer (zfs, dir, btrfs, etc…), let’s quickly create a new pool with dir driver

$ lxc storage create dir-pool dir source=/virtual-hdd/test-dir/
Storage pool dir-pool created

$ lxc storage list
+----------+--------+------------------------------------------------+-------------+---------+---------+
|   NAME   | DRIVER |                     SOURCE                     | DESCRIPTION | USED BY |  STATE  |
+----------+--------+------------------------------------------------+-------------+---------+---------+
| default  | dir    | /var/snap/lxd/common/lxd/storage-pools/default |             | 1       | CREATED |
+----------+--------+------------------------------------------------+-------------+---------+---------+
| dir-pool | dir    | /virtual-hdd/test-dir/                         |             | 0       | CREATED |
+----------+--------+------------------------------------------------+-------------+---------+---------+
| zfs-pool | zfs    | zfs-pool                                       |             | 9       | CREATED |
+----------+--------+------------------------------------------------+-------------+---------+---------+

Now I have my new storage pool created and ready to be used. Let’s use it directly to launch a new container

$ lxc launch images:ubuntu/20.04 -s dir-pool c1
Creating c2
Starting c2

$ lxc list
+-----------------+---------+-----------------------+------+-----------+-----------+
|      NAME       |  STATE  |         IPV4          | IPV6 |   TYPE    | SNAPSHOTS |
+-----------------+---------+-----------------------+------+-----------+-----------+
| c2              | RUNNING | 10.252.104.51 (eth0)  |      | CONTAINER | 0         |

To make sure that my new container is really in the new dir-pool, run the following

$ lxc storage info dir-pool

info:
  description: ""
  driver: dir
  name: dir-pool
  space used: 20.82GiB
  total space: 491.08GiB
used by:
  instances:
  - c2

2- Now to the second part of your question, how to move existing containers and VMs to the new storage pool, to free the space on your OS partition. You can just run lxc move as follows

lxc move <container|VM name> -s <destination_storage_pool>

I created the container c1 in dir-pool and I want to move it to zfs-pool

$ lxc move c1 -s zfs-pool

You might need to stop the instance before moving, or you will get the error I got
VM

$ lxc move v1 -s dir-pool
Error: Migration operation failure: Stateful stop requires migration.stateful to be set to true

Container

$ lxc move c1 -s dir-pool
Error: Migration operation failure: Unable to perform live container migration. CRIU isn't installed. To migrate the container, stop the container before migration or install CRIU

3- Finally, how to make it the default behaviour to create new instances in the new storage pool? Answer: You can: a- just modify the default profile to point to the newly created storage pool

$ lxc profile edit default

This will open an editor to the profile yaml config

..
  root:
    path: /
    pool: zfs-pool                    # change this to be -> dir-pool (or whatever you called your new storage pool. And do not edit the previous and next lines. Just this one.
    type: disk
..

or, b- create a new profile (if for some reason you want to keep the default profile as is), by copying the default one to a new one and modifying the new profile, and then you will need to specify it with every instance launch

$ lxc profile copy default new-profile

$ lxc profile edit new-profile      #then edit the new profile the same way you would have done it with the default profile

$ lxc launch images:ubuntu/20.04 -p new-profile c3    #you will need to specify the profile every time you launch a new instance

References:

2 Likes