lxc-checkpoint/restore does not work in aarch64 platform

chenglin-li · March 5, 2020, 7:59am

Development environment:

in Linux PC(x86_64) side:

LXC version: 3.0.3
criu version: 3.6
lxc-checkconfig

srcx-lcl@SRCX-LCL:~$ lxc-checkconfig
Kernel configuration not found at /proc/config.gz; searching…
Kernel configuration found at /boot/config-4.15.0-88-generic
— Namespaces —
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

— Control groups —
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/memory
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/pids
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/blkio
/sys/fs/cgroup/rdma
/sys/fs/cgroup/freezer
/sys/fs/cgroup/devices
/sys/fs/cgroup/hugetlb

Cgroup v2 mount points:
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

— Misc —
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: enabled, loaded
CONFIG_NF_NAT_IPV6: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

— Checkpoint/Restore —
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

in target board (aarch64) side:

LXC version: 3.10
criu version: 3.13
- lxc-checkconfig
root@euto-v9:~# lxc-checkconfig
— Namespaces —
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

— Control groups —
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/freezer
/sys/fs/cgroup/memory
/sys/fs/cgroup/devices

Cgroup v2 mount points:
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

— Misc —
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: enabled, not loaded
CONFIG_NF_NAT_IPV6: enabled, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

— Checkpoint/Restore —
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

uname -a

root@euto-v9:~# uname -a
Linux euto-v9 4.14.137 #1 SMP PREEMPT Wed Mar 4 06:28:10 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

cat /proc/cpuinfo`: its CPU architecture is armv8

root@euto-v9:~# cat /proc/cpuinfo
processor : 0
BogoMIPS : 52.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0b
CPU revision : 0

Issue description

I want to run lxc-checkpoint/restore feature in target board(aarch64), there are errors, but if i run lxc-checkpoint/restore in Linux PC(x86_64), it works fine.
for details please check below reproduce steps.

my questions are as below:

as LXC provide default container template, like busybox, when creating checkpoint using busybox, why it does not work? and need to change config file (commnet 2 lines: #lxc.tty.max = 1 ,
#lxc.pty.max = 1) to make it works?
after comment 2 lines in config files, why create checkpoint in target board(aarch64) does not work? the same process works fine in Linux PC(x86_64)? are there any issues on aarch64 platform?
i found that in criu makefile, there are ARM architecture config as below: this means criu does not support ARMV8, right?

ifeq ($(ARCH),arm)
ARMV := $(shell echo $(SUBARCH) | sed -nr ‘s/armv([[:digit:]]).*/\1/p; t; i7’)

    ifeq ($(ARMV),6)
            USERCFLAGS += -march=armv6
    endif

    ifeq ($(ARMV),7)
            USERCFLAGS += -march=armv7-a
    endif

    ifeq ($(ARMV),8)
            # Running 'setarch linux32 uname -m' returns armv8l on travis aarch64.
            # This tells CRIU to handle armv8l just as armv7hf. Right now this is
            # only used for compile testing. No further verification of armv8l exists.
            USERCFLAGS += -march=armv7-a
            ARMV := 7
    endif

if i want to support ARMV8 in target board side, what shoud i do?

could you please give commens and help? thanks in advance!

Steps to reproduce

in Linux PC (x86_64) side:
step 1. create container:
srcx-lcl@SRCX-LCL:~$ sudo lxc-create -n andybusybox -t busybox

step 2. start container:
srcx-lcl@SRCX-LCL:~$ sudo lxc-start -n andybusybox

step 3. check container status: it is running
srcx-lcl@SRCX-LCL:~$ sudo lxc-info -n andybusybox
Name: andybusybox
State: RUNNING
PID: 25297
CPU use: 0.01 seconds
BlkIO use: 0 bytes
Memory use: 1.45 MiB
KMem use: 1.20 MiB
Link: vethIQOI2S
TX bytes: 858 bytes
RX bytes: 2.52 KiB
Total bytes: 3.36 KiB

step 4. create checkpoint: there are errors
srcx-lcl@SRCX-LCL:~$ sudo lxc-checkpoint -n andybusybox -D /tmp/andydump
lxc-checkpoint: andybusybox: criu.c: do_dump: 1328 dump failed with 1
lxc-checkpoint: andybusybox: criu.c: do_dump: 1342 criu output: Will skip in-flight TCP connections

lxc-checkpoint: andybusybox: tools/lxc_checkpoint.c: checkpoint: 200 Checkpointing andybusybox failed

step 5. comment the below lines in config file
#lxc.tty.max = 1
#lxc.pty.max = 1
step 6. after commnet above 2 lines, the config file is as below:

srcx-lcl@SRCX-LCL:/var/lib/lxc/andybusybox$ cat config

Template used to create this container: /usr/share/lxc/templates/lxc-busybox

Parameters passed to the template:

Template script checksum (SHA-1): 21abc1440b73cdb95d96d5459b27c3a87df9976f

For additional config options, please look at lxc.container.conf(5)

Uncomment the following line to support nesting containers:

#lxc.include = /usr/share/lxc/config/nesting.conf

lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:ee:50:4f
lxc.rootfs.path = dir:/var/lib/lxc/lclbusybox/rootfs
lxc.signal.halt = SIGUSR1
lxc.signal.reboot = SIGTERM
lxc.uts.name = “lclbusybox”
#lxc.tty.max = 1
#lxc.pty.max = 1
lxc.cap.drop = sys_module mac_admin mac_override sys_time

When using LXC with apparmor, uncomment the next line to run unconfined:

#lxc.apparmor.profile = unconfined

lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed
lxc.mount.entry = shm /dev/shm tmpfs defaults 0 0
lxc.mount.entry = /lib lib none ro,bind 0 0
lxc.mount.entry = /usr/lib usr/lib none ro,bind 0 0
lxc.mount.entry = /lib64 lib64 none ro,bind 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none ro,bind,optional 0 0

step 7. create checkpoint again: it can create successfully, and image files are created.
srcx-lcl@SRCX-LCL:~$ sudo lxc-checkpoint -n lclbusybox -D /tmp/lcldump

step 8. check the checkpoint dump file, they are created successfully.
srcx-lcl@SRCX-LCL:/tmp$ cd lcldump
srcx-lcl@SRCX-LCL:/tmp/lcldump$ ls
cgroup.img fdinfo-3.img fs-9.img ip6tables-9.img mountpoints-12.img pages-1.img route6-9.img tty-info.img
core-10.img fdinfo-4.img ids-10.img ipcns-var-10.img netdev-9.img pages-2.img route-9.img utsns-11.img
core-1.img fdinfo-5.img ids-1.img iptables-9.img netns-9.img pages-3.img rule-9.img
core-4.img files.img ids-4.img mm-10.img pagemap-10.img pages-4.img stats-dump
core-9.img fs-10.img ids-9.img mm-1.img pagemap-1.img pipes-data.img stats-restore
dump.log fs-1.img ifaddr-9.img mm-4.img pagemap-4.img pstree.img tmpfs-dev-67.tar.gz.img
fdinfo-2.img fs-4.img inventory.img mm-9.img pagemap-9.img restore.log tty.info

step 9. restore the checkpoint, and the container runs successfully

srcx-lcl@SRCX-LCL:~$ sudo lxc-checkpoint -n lclbusybox -D /tmp/lcldump -r
srcx-lcl@SRCX-LCL:~$ sudo lxc-info -n lclbusybox
Name: lclbusybox
State: RUNNING
PID: 25703
CPU use: 0.04 seconds
BlkIO use: 0 bytes
Memory use: 2.15 MiB
KMem use: 1.96 MiB
Link: vethO8VK2M
TX bytes: 426 bytes
RX bytes: 3.14 KiB
Total bytes: 3.56 KiB

when i repeat above steps in target board side (aarch64):

step 1-6 are fine.
for step 7: when create checkpoint, there are errors as below, and dump files are failed to create.

root@euto-v9:~# lxc-checkpoint -n mybusybox -D /tmp/ck
lxc-checkpoint: mybusybox: criu.c: do_dump: 1329 dump failed with 1
lxc-checkpoint: mybusybox: criu.c: do_dump: 1343 criu output:
lxc-checkpoint: mybusybox: lxc_checkpoint.c: checkpoint: 200 Checkpointing mybusybox failed

try to restore checkpoint does not work.

stgraber · March 5, 2020, 8:16am

I’d recommend against cross-posting between forum/github as we’re looking at both