Migration fails - Found un-dumpable network: phys (eth0)

I am trying to migrate a container.

If the container has no IP → works.
If the container has IP → “Error. Found un-dumpable network: phys (eth0)”. Happens on the target side!

Should I configure something?

criu --version
Version: 3.13

uname -a

Linux vm-VirtualBox 5.0.0-36-generic #39~18.04.1-Ubuntu SMP Tue Nov 12 11:09:50 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

lxc info (same for both vms)

config:
  core.https_address: <ip>:8443
  core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - <ip>:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  certificate_fingerprint: 12d5f2a801412b16c908bd1d7aac4b9f8d59572ec413e178b80f61788c395efb
  driver: lxc
  driver_version: 3.2.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.0.0-36-generic
  lxc_features:
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    seccomp_notify: "true"
  project: default
  server: lxd
  server_clustered: false
  server_name: vm-VirtualBox
  server_pid: 2285
  server_version: "3.18"
  storage: zfs
  storage_version: 0.7.12-1ubuntu5

Found in the sources: “We only know how to restore containers with veth networks”

How to configure lxd to use veth?

veths are the default. I wonder if we have an issue with CRIU when using veths that are generated by LXD itself rather than by liblxc.

I just did lxc init, and then launch container. It gets type=phys:

lxc.net.0.name = eth0
lxc.net.0.type = phys
lxc.net.0.flags = up
lxc.net.0.link = veth25e67193

ip link

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:37:c0:41 brd ff:ff:ff:ff:ff:ff
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 02:8a:aa:af:0b:3c brd ff:ff:ff:ff:ff:ff
5: vethbf4745c9@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 02:8a:aa:af:0b:3c brd ff:ff:ff:ff:ff:ff link-netnsid 0
7: vethfa129c4e@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 46:24:f7:c9:75:56 brd ff:ff:ff:ff:ff:ff link-netnsid 1
9: vethb8311a08@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ea:55:b1:b9:75:de brd ff:ff:ff:ff:ff:ff link-netnsid 3

Do you know what I did wrong?

You didn’t do anything wrong. LXD changed a little while back from having liblxc use veths to having itself generate veths and pass them through phys to liblxc.

This then causes this issue as CRIU cannot dump/restore such interfaces.

@brauner thoughts on how to handle that?

I wonder if we can detect that the interface is veth from looking at its properties in /sys/class/net (I notice that veth devices symlink to ../../devices/virtual/net/vethcaaa9763 whereas physical devices symlink to ../../devices/pci0000:00/0000:00:1c.3/0000:03:00.0/net/enp3s0

But I don’t know whether CRIU only works with certain virtual devices or can work with all of them.

Phew, I think it’s time to rope in Adrian…

Can I do something now in order to make it migratable?
I do not need a production ready solution

I cannot give a solution to the problem here, but a general description how CRIU handles network namespaces.

If CRIU is told to checkpoint and restore the network namespace it will try to do that, including the network interfaces. LXC gives CRIU the information to which veth the container internal veth should be connected. If it is not veth based it might fail like it can be seen here.
Docker does the same thing for network namespaces in combination with CRIU.

My recommendation for LXC/LXD would be to do the same as Podman. The network namespace setup is done outside of Podman (CNI). CRIU checkpoints the container knowing that it should ignore the network namespace. The same happens during restore. The network namespace is pre-created and CRIU restores the container into that network namespace. That way any network namespace configuration is supported as CRIU does not touch the network namespace. The network namespace setup can even change during migration (if migrating without --tcp-established).

Unfortunately no solution from me.

The network namespace setup is done outside of Podman (CNI).

I want to know the method of namespace setup of Podman