Live migration between 2 hosts fails


(Ibrahim Menem) #1

Hi, I’m trying to test live migration but im having hard time doing it, I tested different versions and configurations with no success, Migration between two hosts is failing with this message:

$ lxc move test1 ibra3:
error: Failed container creation:
 - https://10.40.128.12:8443: Error transferring container data: Failed to run: /usr/bin/lxd forkmigrate test1 /var/lib/lxd/containers /var/log/lxd/test1/lxc.conf /tmp/lxd_restore_820736425 true:

container image is ubuntu 16.04

host is ubuntu 17.04 with kernel:
Linux ibratest-2 4.13.0-21-generic #24-Ubuntu SMP Mon Dec 18 17:29:16 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

on destination host I got this log:

$ cat /var/log/lxd/lxd.log 

ephemeral=false lvl=info msg="Creating container" name=test2 t=2018-01-08T13:23:54+0000
ephemeral=false lvl=info msg="Created container" name=test2 t=2018-01-08T13:23:54+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=test2 t=2018-01-08T13:23:54+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=test2 t=2018-01-08T13:23:54+0000
ephemeral=false lvl=info msg="Creating container" name=test2/s3 t=2018-01-08T13:23:55+0000
ephemeral=false lvl=info msg="Created container" name=test2/s3 t=2018-01-08T13:23:55+0000
ephemeral=false lvl=info msg="Creating container" name=test2/s4 t=2018-01-08T13:24:00+0000
ephemeral=false lvl=info msg="Created container" name=test2/s4 t=2018-01-08T13:24:00+0000
actionscript=false created=2018-01-08T13:23:54+0000 ephemeral=false lvl=info msg="Migrating container" name=test2 statedir=/tmp/lxd_restore_982407591 stop=false t=2018-01-08T13:24:04+0000 used=1970-01-01T00:00:00+0000
action=stop created=2018-01-08T13:23:54+0000 ephemeral=false lvl=info msg="Container initiated stop" name=test2 stateful=false t=2018-01-08T13:24:05+0000 used=2018-01-08T13:24:04+0000
action=stop created=2018-01-08T13:23:54+0000 ephemeral=false lvl=info msg="Container initiated stop" name=test2 stateful=false t=2018-01-08T13:24:06+0000 used=2018-01-08T13:24:04+0000
err="remove /sys/kernel/security/apparmor/policy/namespaces/lxd-test2_<var-lib-lxd>: no such file or directory" lvl=eror msg="error removing apparmor namespace" ns=/sys/kernel/security/apparmor/policy/namespaces/lxd-test2_<var-lib-lxd> t=2018-01-08T13:24:06+0000
err="open /tmp/lxd_restore_982407591/restore.log: no such file or directory" lvl=eror msg="Error collecting checkpoint log file" t=2018-01-08T13:24:06+0000
err="Failed to run: /usr/bin/lxd forkmigrate test2 /var/lib/lxd/containers /var/log/lxd/test2/lxc.conf /tmp/lxd_restore_982407591 true: " lvl=eror msg="Error during migration sink" t=2018-01-08T13:24:06+0000
created=2018-01-08T13:23:54+0000 ephemeral=false lvl=info msg="Deleting container" name=test2 t=2018-01-08T13:24:06+0000 used=1970-01-01T00:00:00+0000
created=2018-01-08T13:23:55+0000 ephemeral=false lvl=info msg="Deleting container" name=test2/s3 t=2018-01-08T13:24:06+0000 used=1970-01-01T00:00:00+0000
created=2018-01-08T13:23:55+0000 ephemeral=false lvl=info msg="Deleted container" name=test2/s3 t=2018-01-08T13:24:06+0000 used=1970-01-01T00:00:00+0000
created=2018-01-08T13:24:00+0000 ephemeral=false lvl=info msg="Deleting container" name=test2/s4 t=2018-01-08T13:24:06+0000 used=1970-01-01T00:00:00+0000
created=2018-01-08T13:24:00+0000 ephemeral=false lvl=info msg="Deleted container" name=test2/s4 t=2018-01-08T13:24:06+0000 used=1970-01-01T00:00:00+0000
action=R err="Failed to run: apparmor_parser -RWL /var/lib/lxd/security/apparmor/cache /var/lib/lxd/security/apparmor/profiles/lxd-test2: apparmor_parser: Unable to remove \"lxd-test2_</var/lib/lxd>\".  Profile doesn't exist" lvl=eror msg="Running apparmor" output="apparmor_parser: Unable to remove \"lxd-test2_</var/lib/lxd>\".  Profile doesn't exist\n" t=2018-01-08T13:24:06+0000
container=test2 err="Failed to run: apparmor_parser -RWL /var/lib/lxd/security/apparmor/cache /var/lib/lxd/security/apparmor/profiles/lxd-test2: apparmor_parser: Unable to remove \"lxd-test2_</var/lib/lxd>\".  Profile doesn't exist" lvl=eror msg="Failed to destroy apparmor namespace" t=2018-01-08T13:24:06+0000
created=2018-01-08T13:23:54+0000 ephemeral=false lvl=info msg="Deleted container" name=test2 t=2018-01-08T13:24:06+0000 used=1970-01-01T00:00:00+0000

and

cat /var/log/lxd/test2/lxc.log
            lxc 20180108132404.933 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132404.933 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132405.144 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132405.144 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132405.213 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132405.213 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132405.871 ERROR    lxc_criu - criu.c:do_restore:978 - criu process exited 1, output:

            lxc 20180108132405.871 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132405.871 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132405.871 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:125 - Command get_cgroup failed to receive response: Connection reset by peer.
            lxc 20180108132405.871 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:125 - Command get_cgroup failed to receive response: Connection reset by peer.
            lxc 20180108132405.878 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/unified//lxc/test2
            lxc 20180108132405.885 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/systemd//lxc/test2
            lxc 20180108132405.891 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/hugetlb//lxc/test2
            lxc 20180108132405.898 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/cpu//lxc/test2
            lxc 20180108132405.904 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/rdma//lxc/test2
            lxc 20180108132405.911 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/freezer//lxc/test2
            lxc 20180108132405.917 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/blkio//lxc/test2
            lxc 20180108132405.925 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/devices//lxc/test2
            lxc 20180108132405.931 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/net_cls//lxc/test2
            lxc 20180108132405.938 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/cpuset//lxc/test2
            lxc 20180108132405.945 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/perf_event//lxc/test2
            lxc 20180108132405.952 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/pids//lxc/test2
            lxc 20180108132405.958 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/memory//lxc/test2
            lxc 20180108132405.958 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132405.958 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180108132406.557 ERROR    lxc_criu - criu.c:__criu_restore:1284 - restore process died

source and destination instalation are identical;

lxc info
config:
  core.https_address: 10.40.128.15:8443
  core.trust_password: true
  images.auto_update_interval: "0"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
  addresses:
  - 10.40.128.15:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFUzCCAzugAwIBAgC52cS2kFym5/VLWL3S/RWMgvmX5qAJ 
...
    W6s5spwdW8LbT4m75iggmGh3dEF0uACHifAhHnbJazavHIgp7F17zy/8DUnytW0T
    kcYDIJhOH7ytrM1kWKP22ulnQ9H5ryw=
    -----END CERTIFICATE-----
  certificate_fingerprint: 4a5b56e3e0f260c86a81106ee7dbd5b92862401bd56cdabd56a50018285ea612
  driver: lxc
  driver_version: 2.1.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.13.0-21-generic
  server: lxd
  server_pid: 4072
  server_version: "2.18"
  storage: zfs
  storage_version: 0.6.5.11-1ubuntu3

criu installed from source:

sudo criu --version
Version: 3.7
GitID: v3.7

Any hint?
Many thanks


(Stéphane Graber) #2

I’d recommend upgrading to the latest LXD first, just in case this helps:

apt install -t artful-backports lxd lxd-client

Then the best first step is to try just a local stateful snapshot or stateful stop, that tends to isolate things to just CRIU and make it easier to look at logs:

lxc snapshot test1 --stateful
lxc stop test1 --stateful

Either of those will have LXD tell CRIU to dump the container to disk.
Any failure should result in a log file in /var/log/lxd/test1/


(Ibrahim Menem) #3

Hi Stéphane,

Thank you for answering!
I installed the version from artful-backports, restarted lxd and redid the test, but still having the same error :frowning_face:. the stateful snapshot is working the problem is in the restore process

ubuntu@ibratest-2:~$ lxc snapshot test2 s2 --stateful
ubuntu@ibratest-2:~$ lxc stop test2 --stateful
ubuntu@ibratest-2:~$ lxc start test2 
error: Failed to run: /usr/bin/lxd forkmigrate test2 /var/lib/lxd/containers /var/log/lxd/test2/lxc.conf /var/lib/lxd/containers/test2/state true: 
Try `lxc info --show-log test2` for more info
  driver: lxc
  driver_version: 2.1.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.13.0-21-generic
  server: lxd
  server_pid: 14000
  server_version: "2.21"
  storage: zfs

exactly the same error but now in the same host:

lxc info --show-log test2
Name: test2
Remote: unix://
Architecture: x86_64
Created: 2018/01/08 13:17 UTC
Status: Stopped
Type: persistent
Profiles: default
Snapshots:
  s2 (taken at 2018/01/09 09:37 UTC) (stateful)

Log:

            lxc 20180109093745.309 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093745.309 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093745.437 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093745.437 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093745.524 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093745.524 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093746.488 ERROR    lxc_conf - conf.c:run_buffer:438 - Script exited with status 1.
            lxc 20180109093746.488 ERROR    lxc_start - start.c:lxc_fini:753 - Failed to run lxc.hook.post-stop for container "test2".
            lxc 20180109093746.673 ERROR    lxc_criu - criu.c:do_restore:978 - criu process exited 1, output:

            lxc 20180109093746.675 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093746.675 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093746.676 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:125 - Command get_cgroup failed to receive response: Connection reset by peer.
            lxc 20180109093746.743 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/unified//lxc/test2
            lxc 20180109093746.798 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/systemd//lxc/test2
            lxc 20180109093746.858 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/net_cls//lxc/test2
            lxc 20180109093746.914 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/hugetlb//lxc/test2
            lxc 20180109093746.974 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/memory//lxc/test2
            lxc 20180109093746.102 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/pids//lxc/test2
            lxc 20180109093746.107 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/cpuset//lxc/test2
            lxc 20180109093746.113 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/cpu//lxc/test2
            lxc 20180109093746.119 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/devices//lxc/test2
            lxc 20180109093746.126 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/blkio//lxc/test2
            lxc 20180109093746.132 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/perf_event//lxc/test2
            lxc 20180109093746.139 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/freezer//lxc/test2
            lxc 20180109093746.145 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/rdma//lxc/test2
            lxc 20180109093746.145 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093746.145 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20180109093746.668 ERROR    lxc_conf - conf.c:run_buffer:438 - Script exited with status 1.
            lxc 20180109093746.668 ERROR    lxc_start - start.c:lxc_fini:753 - Failed to run lxc.hook.post-stop for container "test2".
            lxc 20180109093746.669 ERROR    lxc_criu - criu.c:__criu_restore:1284 - restore process died


(Stéphane Graber) #4

What do you have in /var/log/lxd/test2/ ?
CRIU should create one log file per attempt on top of logging to lxc.log.


(Ibrahim Menem) #5

Hi again,

ubuntu@ibratest-2:~$ ls -lart /var/log/lxd/test2/

total 24824
-rw-r--r-- 1 root root 1775739 Jan  8 13:17 snapshot_dump_2018-01-08T13:17:50Z.log
-rw-r--r-- 1 root root 1774936 Jan  8 13:18 migration_dump_2018-01-08T13:18:17Z.log
-rw-r--r-- 1 root root 1774233 Jan  8 13:18 migration_dump_2018-01-08T13:18:52Z.log
-rw-r--r-- 1 root root 1653936 Jan  8 13:21 migration_dump_2018-01-08T13:21:16Z.log
-rw-r--r-- 1 root root 1654160 Jan  8 13:24 migration_dump_2018-01-08T13:24:06Z.log
-rw-r--r-- 1 root root 1683124 Jan  8 13:36 migration_dump_2018-01-08T13:36:19Z.log
-rw-r--r-- 1 root root       0 Jan  8 13:48 forkexec.log
-rw-r--r-- 1 root root 1682906 Jan  8 15:30 migration_dump_2018-01-08T15:30:33Z.log
-rw-r--r-- 1 root root 1682899 Jan  8 15:30 migration_dump_2018-01-08T15:30:56Z.log
-rw-r--r-- 1 root root 1683046 Jan  8 16:02 migration_dump_2018-01-08T16:02:31Z.log
-rw-r--r-- 1 root root       0 Jan  8 16:37 forkstart.log
-rw-r--r-- 1 root root 1658825 Jan  8 16:42 migration_dump_2018-01-08T16:42:58Z.log
-rw-r--r-- 1 root root 1658862 Jan  8 16:45 migration_dump_2018-01-08T16:45:05Z.log
-rw-r--r-- 1 root root 1663884 Jan  8 16:49 migration_dump_2018-01-08T16:49:14Z.log
drwxr-xr-x 8 root root    4096 Jan  9 06:25 ..
-rw-r--r-- 1 root root       0 Jan  9 08:33 netcat.log
-rw-r--r-- 1 root root 1677111 Jan  9 08:33 migration_dump_2018-01-09T08:33:48Z.log
-rw-r--r-- 1 root root 1676568 Jan  9 09:37 snapshot_dump_2018-01-09T09:37:02Z.log
-rw-r--r-- 1 root root 1676418 Jan  9 09:37 snapshot_dump_2018-01-09T09:37:35Z.log
-rw-r--r-- 1 root root       0 Jan  9 09:37 lxc.log.old
-rw-r--r-- 1 root root    1634 Jan  9 09:37 lxc.conf
drwxr-xr-x 2 root root    4096 Jan  9 09:37 .
-rw-r--r-- 1 root root    4407 Jan  9 09:37 lxc.log

ubuntu@ibratest-2:~$ tail /var/log/lxd/test2/snapshot_dump_2018-01-09T09:37:35Z.log

(03.882616) 	Unseizing 5294 into 2
(03.882694) 	Unseizing 5300 into 2
(03.883049) 	Unseizing 5330 into 2
(03.883418) 	Unseizing 5332 into 2
(03.883626) 	Unseizing 5338 into 2
(03.883814) 	Unseizing 5341 into 2
(03.884138) 	Unseizing 5443 into 2
(03.884170) 	Unseizing 5457 into 2
(03.944707) Writing stats
(03.944900) Dumping finished successfully

the log is huge, I’m looking for a way to share it !


(Ibrahim Menem) #6

the full logs here https://drive.google.com/open?id=1Wvpaekj4PZT6BNPWe3_r_dT6qyJYJJnt


(Stéphane Graber) #7

Can you post the output of lxc info? That should give us an idea of the liblxc version in use, in case it matters somehow.


(Ibrahim Menem) #8

lxc info

config:
  core.https_address: 10.40.128.12:8443
  core.trust_password: true
  images.auto_update_interval: "0"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 10.40.128.12:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----

    -----END CERTIFICATE-----
  certificate_fingerprint: 0acab152caaf6ca62044c634378a55cf061831bf7e514bd47d844aaae8ab0220
  driver: lxc
  driver_version: 2.1.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.13.0-21-generic
  server: lxd
  server_pid: 14000
  server_version: "2.21"
  storage: zfs
  storage_version: 0.6.5.11-1ubuntu3

also

dpkg -l | grep lx

ii  libgl1-mesa-glx:amd64                      17.2.2-0ubuntu1                              amd64        free implementation of the OpenGL API -- GLX runtime
ii  libgolang-gopkg-lxc-go-lxc.v2-1            0.0~git20161126.1.82a07a6-0ubuntu7           amd64        Go bindings for LXC -- shared library
ii  liblxc1                                    2.1.1-0ubuntu1~ubuntu17.10.1~ppa1            amd64        Linux Containers userspace tools (library)
ii  libxcb-glx0:amd64                          1.12-1ubuntu1                                amd64        X C Binding, glx extension
ii  lxc-common                                 2.1.1-0ubuntu1~ubuntu17.10.1~ppa1            amd64        Linux Containers userspace tools (common tools)
ii  lxcfs                                      2.0.8-1ubuntu2~ubuntu17.10.1~ppa1            amd64        FUSE based filesystem for LXC
ii  lxd                                        2.21-0ubuntu2~17.10.1                        amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                                 2.21-0ubuntu2~17.10.1                        amd64        Container hypervisor based on LXC - client

(Stéphane Graber) #9

@brauner do you remember what version of CRIU, LXC and LXD you were using when you got it working last (probably when working with Adrian)?


(Christian Brauner) #10

I asked Adrian just now but I think we simply used the latest CRIU release and liblxc and LXD from master.


(Ibrahim Menem) #11

Thank you @brauner and @stgraber
It would be great if you can confirm that some combination is working for you
on the other hand, do you have any estimation of when Live migration will be a non-experimental feature? or it is all in hands of CRIU people?


(Christian Brauner) #12

That is indeed mostly in the hands of the CRIU people.


#13

@ibrahim_menem It works perfectly for me with the latest LXC/LXD/CRIU from git. My base system is CentOS VM my containers are alpine and Fedora.


#14

In the logs you attached I see action-script errors and https://github.com/lxc/lxd/issues/4190 also seems related to CRIU’s action-script. So those two might be related.