Creating new container from a snapshot constantly fails

Hi there,

Ok so this is not the first time this has happened, i’ve got this problem before when using ceph for storage, but right now i’m using ZFS on a LVM logical volume. But whenever i use `lxc copy / , i get the following error.

Error: not found

here is the result of lxc info:

config:
  core.https_address: 178.128.31.165:8443
  core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 178.128.31.165:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFZDCCA0ygAwIBAgIQX+PZA8qRHU+XqWjHIm/eizANBgkqhkiG9w0BAQsFADA8
    MRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRwwGgYDVQQDDBNyb290QGx4
    Yy1jbHVzdGVyLTAxMB4XDTE4MTAyODE2NDI1NloXDTI4MTAyNTE2NDI1NlowPDEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEcMBoGA1UEAwwTcm9vdEBseGMt
    Y2x1c3Rlci0wMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMvnSYfa
    ebx2tRmgNY1d/xw71W0ymUPv78J9sT86BKCCwoPkFoM9Gur68PsvZPAyKXT2qcxT
    OXn37GWziTN9jwGoT6wUP9ollGe0myJmyVG5TalKK3ININ7CW9djPc9qAvagXObQ
    EC1ri4qIrFydUzGw/vuv4i3j7FO2Vn3QLIK/qnA6zhgEKfufvi8mRsV/QOFVeVAF
    Ik9/o4ie4oPoZ0zHpduhsm1woHlEhuq7Sbs8ScmdfU41MNi1JXc84HJzIoVRw+8f
    kg3Ziw4glSC5OjszujB/DXIaop8QyMLnSHGcXkh4aAbYvJT6LCXst93CP8USsQmb
    S1Rk8b2EJkBmw6W84G0fxruOjr8VFU//+YHvmauXcMi2P7oT7Tmg1vevmk/pbv7S
    BZF8qM41tccs4xBcdkoasPVdrefIHMB/LXWYRR/5ateeZ5psZZMNWppyd5TKSKj8
    SeEuJVLL6hvl0iK8e7EZ9f/b7LBM6/AFd42cukmKxnsjSopUeDPhzJV0uZfsPQYf
    gMJt/IOIeiK/KDXMuAGslazQ/RkoVf+zGqm4cA/HSsahnGu93IZZAwnqh55ArXe4
    C0Ts9r8ASC1GqCu1d95j6meoSg9c8htXi2ba5T4QrDJRmlXs3zZkFadHj8N00DaI
    G0D7nL6bDXVqOnWmzPqISB2yNqJK4BmLIUHRAgMBAAGjYjBgMA4GA1UdDwEB/wQE
    AwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCsGA1UdEQQk
    MCKCDmx4Yy1jbHVzdGVyLTAxhwSygB+lhwQKDwAGhwQKgmbLMA0GCSqGSIb3DQEB
    CwUAA4ICAQACWMbb+q6h+Hc3sFN0zZIS9BMpp0mNmgQoiIpN5ZTluAQZR3M+06WR
    jV4yXaz6RjY3fzhxF/PySGdiADmw9I/utVWgxjclJR3cxZ80Geap22XBF+J2hCVq
    98taUZ9Z1Hs5fqCWkD4r1zb434KauRVnpyL1XURVxjbjGCyMC6gcDRBM846HhG2f
    oeGNp2Hg3xeJX0WmkUBScmgajwsW5VtTOVmSs+RjF/HgJG6v9spzcK61V0xrxnxz
    stDIEvqaSgCcw9/twdTjatJ0eAy0XrTdhIEr1s9nWddsoTTA/KbooGeD2g7HPvP6
    uI0ElTSHMMafLaYWbKfwig6fkz4WEThc7PnU3kHKPKdl1LovKB3u9LCejetzq7u3
    quNfjdcg1HGe7Cjt+igAOYWzNYiKWrrAPHZoihkHRvGBFuEiX9buoZkB1rgb5zil
    eQ4+wJe/9ErZ8FxCjenVDFPpy2+82CGOJxI50lVMcB6/dFhg2f5NE81HrmkKQ1az
    nc3JDPgha8K8j5X6EOvmRYSrxv7NriKkDkyUxs6kOJg/gdFlRInXZsUpe5SJEkIB
    y7CgOIRuCXaQyiwjNbmUrzJk1uno2pRDLtlp0sErfPkvksNq5yZG1tbZFW33+a0S
    Cs7kJFkWUDQTo8XDdpuCuDoykL5ePdY3F8JzC7OA219AD388ommlfg==
    -----END CERTIFICATE-----
  certificate_fingerprint: baa6af1ce78c26b05c3b2e64c0f482a39655ae315cb3d9309db10d32a300a586
  driver: lxc
  driver_version: 3.0.2
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.4.0-138-generic
  server: lxd
  server_pid: 8676
  server_version: "3.6"
  storage: zfs
  storage_version: 0.6.5.6-0ubuntu25
  server_clustered: true
  server_name: lxc-cluster-01
  project: default

and for lxc info haproxy:

Name: haproxy
Location: lxc-cluster-01
Remote: unix://
Architecture: x86_64
Created: 2018/10/28 17:28 UTC
Status: Running
Type: persistent
Profiles: default
Pid: 7970
Ips:
  eth0: inet    240.102.203.135 veth03N0KJ
  eth0: inet6   fe80::216:3eff:feb3:5e76        veth03N0KJ
  lo:   inet    127.0.0.1
  lo:   inet6   ::1
Resources:
  Processes: 5
  Disk usage:
    root: 147.97kB
  CPU usage:
    CPU usage (in seconds): 0
  Memory usage:
    Memory (current): 1.52MB
    Memory (peak): 2.27MB
  Network usage:
    eth0:
      Bytes received: 2.10kB
      Bytes sent: 1.72kB
      Packets received: 18
      Packets sent: 12
    lo:
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
Snapshots:
  snap0 (taken at 2018/10/28 17:28 UTC) (stateless)

and btw yes i’m using lxd clustering but i’m executing the shell scripts from within the local machine on which the container is running on, the error is still the same.

Can you show the --debug output of that lxc copy?

here you go:

DBUG[10-29|06:18:44] Connecting to a local LXD over a Unix socket
DBUG[10-29|06:18:44] Sending request to LXD                   etag= method=GET u                                                                                        rl=http://unix.socket/1.0
DBUG[10-29|06:18:44] Got response struct from LXD
DBUG[10-29|06:18:44]
        {
                "config": {
                        "core.https_address": "178.128.31.165:8443",
                        "core.trust_password": true
                },
                "api_extensions": [
                        "storage_zfs_remove_snapshots",
                        "container_host_shutdown_timeout",
                        "container_stop_priority",
                        "container_syscall_filtering",
                        "auth_pki",
                        "container_last_used_at",
                        "etag",
                        "patch",
                        "usb_devices",
                        "https_allowed_credentials",
                        "image_compression_algorithm",
                        "directory_manipulation",
                        "container_cpu_time",
                        "storage_zfs_use_refquota",
                        "storage_lvm_mount_options",
                        "network",
                        "profile_usedby",
                        "container_push",
                        "container_exec_recording",
                        "certificate_update",
                        "container_exec_signal_handling",
                        "gpu_devices",
                        "container_image_properties",
                        "migration_progress",
                        "id_map",
                        "network_firewall_filtering",
                        "network_routes",
                        "storage",
                        "file_delete",
                        "file_append",
                        "network_dhcp_expiry",
                        "storage_lvm_vg_rename",
                        "storage_lvm_thinpool_rename",
                        "network_vlan",
                        "image_create_aliases",
                        "container_stateless_copy",
                        "container_only_migration",
                        "storage_zfs_clone_copy",
                        "unix_device_rename",
                        "storage_lvm_use_thinpool",
                        "storage_rsync_bwlimit",
                        "network_vxlan_interface",
                        "storage_btrfs_mount_options",
                        "entity_description",
                        "image_force_refresh",
                        "storage_lvm_lv_resizing",
                        "id_map_base",
                        "file_symlinks",
                        "container_push_target",
                        "network_vlan_physical",
                        "storage_images_delete",
                        "container_edit_metadata",
                        "container_snapshot_stateful_migration",
                        "storage_driver_ceph",
                        "storage_ceph_user_name",
                        "resource_limits",
                        "storage_volatile_initial_source",
                        "storage_ceph_force_osd_reuse",
                        "storage_block_filesystem_btrfs",
                        "resources",
                        "kernel_limits",
                        "storage_api_volume_rename",
                        "macaroon_authentication",
                        "network_sriov",
                        "console",
                        "restrict_devlxd",
                        "migration_pre_copy",
                        "infiniband",
                        "maas_network",
                        "devlxd_events",
                        "proxy",
                        "network_dhcp_gateway",
                        "file_get_symlink",
                        "network_leases",
                        "unix_device_hotplug",
                        "storage_api_local_volume_handling",
                        "operation_description",
                        "clustering",
                        "event_lifecycle",
                        "storage_api_remote_volume_handling",
                        "nvidia_runtime",
                        "container_mount_propagation",
                        "container_backup",
                        "devlxd_images",
                        "container_local_cross_pool_handling",
                        "proxy_unix",
                        "proxy_udp",
                        "clustering_join",
                        "proxy_tcp_udp_multi_port_handling",
                        "network_state",
                        "proxy_unix_dac_properties",
                        "container_protection_delete",
                        "unix_priv_drop",
                        "pprof_http",
                        "proxy_haproxy_protocol",
                        "network_hwaddr",
                        "proxy_nat",
                        "network_nat_order",
                        "container_full",
                        "candid_authentication",
                        "backup_compression",
                        "candid_config",
                        "nvidia_runtime_config",
                        "storage_api_volume_snapshots",
                        "storage_unmapped",
                        "projects",
                        "candid_config_key"
                ],
                "api_status": "stable",
                "api_version": "1.0",
                "auth": "trusted",
                "public": false,
                "auth_methods": [
                        "tls"
                ],
                "environment": {
                        "addresses": [
                                "178.128.31.165:8443"
                        ],
                        "architectures": [
                                "x86_64",
                                "i686"
                        ],
                        "certificate": "-----BEGIN CERTIFICATE-----\nMIIFZDCCA0y                                                                                        gAwIBAgIQX+PZA8qRHU+XqWjHIm/eizANBgkqhkiG9w0BAQsFADA8\nMRwwGgYDVQQKExNsaW51eGNvb                                                                                        nRhaW5lcnMub3JnMRwwGgYDVQQDDBNyb290QGx4\nYy1jbHVzdGVyLTAxMB4XDTE4MTAyODE2NDI1Nlo                                                                                        XDTI4MTAyNTE2NDI1NlowPDEc\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEcMBoGA1UEAwwTc                                                                                        m9vdEBseGMt\nY2x1c3Rlci0wMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMvnSYfa\ne                                                                                        bx2tRmgNY1d/xw71W0ymUPv78J9sT86BKCCwoPkFoM9Gur68PsvZPAyKXT2qcxT\nOXn37GWziTN9jwG                                                                                        oT6wUP9ollGe0myJmyVG5TalKK3ININ7CW9djPc9qAvagXObQ\nEC1ri4qIrFydUzGw/vuv4i3j7FO2V                                                                                        n3QLIK/qnA6zhgEKfufvi8mRsV/QOFVeVAF\nIk9/o4ie4oPoZ0zHpduhsm1woHlEhuq7Sbs8ScmdfU4                                                                                        1MNi1JXc84HJzIoVRw+8f\nkg3Ziw4glSC5OjszujB/DXIaop8QyMLnSHGcXkh4aAbYvJT6LCXst93CP                                                                                        8USsQmb\nS1Rk8b2EJkBmw6W84G0fxruOjr8VFU//+YHvmauXcMi2P7oT7Tmg1vevmk/pbv7S\nBZF8q                                                                                        M41tccs4xBcdkoasPVdrefIHMB/LXWYRR/5ateeZ5psZZMNWppyd5TKSKj8\nSeEuJVLL6hvl0iK8e7E                                                                                        Z9f/b7LBM6/AFd42cukmKxnsjSopUeDPhzJV0uZfsPQYf\ngMJt/IOIeiK/KDXMuAGslazQ/RkoVf+zG                                                                                        qm4cA/HSsahnGu93IZZAwnqh55ArXe4\nC0Ts9r8ASC1GqCu1d95j6meoSg9c8htXi2ba5T4QrDJRmlX                                                                                        s3zZkFadHj8N00DaI\nG0D7nL6bDXVqOnWmzPqISB2yNqJK4BmLIUHRAgMBAAGjYjBgMA4GA1UdDwEB/                                                                                        wQE\nAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCsGA1UdEQQk\nMCKCDmx4Y                                                                                        y1jbHVzdGVyLTAxhwSygB+lhwQKDwAGhwQKgmbLMA0GCSqGSIb3DQEB\nCwUAA4ICAQACWMbb+q6h+Hc                                                                                        3sFN0zZIS9BMpp0mNmgQoiIpN5ZTluAQZR3M+06WR\njV4yXaz6RjY3fzhxF/PySGdiADmw9I/utVWgx                                                                                        jclJR3cxZ80Geap22XBF+J2hCVq\n98taUZ9Z1Hs5fqCWkD4r1zb434KauRVnpyL1XURVxjbjGCyMC6g                                                                                        cDRBM846HhG2f\noeGNp2Hg3xeJX0WmkUBScmgajwsW5VtTOVmSs+RjF/HgJG6v9spzcK61V0xrxnxz\                                                                                        nstDIEvqaSgCcw9/twdTjatJ0eAy0XrTdhIEr1s9nWddsoTTA/KbooGeD2g7HPvP6\nuI0ElTSHMMafL                                                                                        aYWbKfwig6fkz4WEThc7PnU3kHKPKdl1LovKB3u9LCejetzq7u3\nquNfjdcg1HGe7Cjt+igAOYWzNYi                                                                                        KWrrAPHZoihkHRvGBFuEiX9buoZkB1rgb5zil\neQ4+wJe/9ErZ8FxCjenVDFPpy2+82CGOJxI50lVMc                                                                                        B6/dFhg2f5NE81HrmkKQ1az\nnc3JDPgha8K8j5X6EOvmRYSrxv7NriKkDkyUxs6kOJg/gdFlRInXZsU                                                                                        pe5SJEkIB\ny7CgOIRuCXaQyiwjNbmUrzJk1uno2pRDLtlp0sErfPkvksNq5yZG1tbZFW33+a0S\nCs7                                                                                        kJFkWUDQTo8XDdpuCuDoykL5ePdY3F8JzC7OA219AD388ommlfg==\n-----END CERTIFICATE-----                                                                                        \n",
                        "certificate_fingerprint": "baa6af1ce78c26b05c3b2e64c0f4                                                                                        82a39655ae315cb3d9309db10d32a300a586",
                        "driver": "lxc",
                        "driver_version": "3.0.2",
                        "kernel": "Linux",
                        "kernel_architecture": "x86_64",
                        "kernel_version": "4.4.0-138-generic",
                        "server": "lxd",
                        "server_pid": 8676,
                        "server_version": "3.6",
                        "storage": "zfs",
                        "storage_version": "0.6.5.6-0ubuntu25",
                        "server_clustered": true,
                        "server_name": "lxc-cluster-01",
                        "project": "default"
                }
        }
DBUG[10-29|06:18:44] Sending request to LXD                   etag= method=GET u                                                                                        rl=http://unix.socket/1.0/containers/haproxy/snapshots/snap0
DBUG[10-29|06:18:44] Got response struct from LXD
DBUG[10-29|06:18:44]
        {
                "architecture": "x86_64",
                "config": {
                        "image.architecture": "amd64",
                        "image.description": "Alpine edge amd64 (20181004_13:00)                                                                                        ",
                        "image.os": "Alpine",
                        "image.release": "edge",
                        "image.serial": "20181004_13:00",
                        "volatile.base_image": "87530b2678f03cbc51a7e1732acfed67                                                                                        9c4a15ba1939197cbda733346ef28ad9",
                        "volatile.eth0.hwaddr": "00:16:3e:b3:5e:76",
                        "volatile.idmap.base": "0",
                        "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":true,                                                                                        \"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]",
                        "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\"                                                                                        :true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]",
                        "volatile.last_state.power": "RUNNING"
                },
                "created_at": "2018-10-28T17:28:56.623874691Z",
                "devices": {},
                "ephemeral": false,
                "expanded_config": {
                        "image.architecture": "amd64",
                        "image.description": "Alpine edge amd64 (20181004_13:00)                                                                                        ",
                        "image.os": "Alpine",
                        "image.release": "edge",
                        "image.serial": "20181004_13:00",
                        "volatile.base_image": "87530b2678f03cbc51a7e1732acfed67                                                                                        9c4a15ba1939197cbda733346ef28ad9",
                        "volatile.eth0.hwaddr": "00:16:3e:b3:5e:76",
                        "volatile.idmap.base": "0",
                        "volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":true,                                                                                        \"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]",
                        "volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\"                                                                                        :true,\"Hostid\":1000000,\"Nsid\":0,\"Maprange\":1000000000}]",
                        "volatile.last_state.power": "RUNNING"
                },
                "expanded_devices": {
                        "eth0": {
                                "name": "eth0",
                                "nictype": "bridged",
                                "parent": "lxdfan0",
                                "type": "nic"
                        },
                        "root": {
                                "path": "/",
                                "pool": "local",
                                "type": "disk"
                        }
                },
                "last_used_at": "1970-01-01T00:00:00Z",
                "name": "snap0",
                "profiles": [
                        "default"
                ],
                "stateful": false
        }
DBUG[10-29|06:18:44] Connected to the websocket
DBUG[10-29|06:18:44] Sending request to LXD                   etag= method=POST                                                                                         url=http://unix.socket/1.0/containers
DBUG[10-29|06:18:44]
        {
                "architecture": "x86_64",
                "config": {
                        "image.architecture": "amd64",
                        "image.description": "Alpine edge amd64 (20181004_13:00)                                                                                        ",
                        "image.os": "Alpine",
                        "image.release": "edge",
                        "image.serial": "20181004_13:00",
                        "volatile.base_image": "87530b2678f03cbc51a7e1732acfed67                                                                                        9c4a15ba1939197cbda733346ef28ad9"
                },
                "devices": {},
                "ephemeral": false,
                "profiles": [
                        "default"
                ],
                "stateful": false,
                "description": "",
                "name": "haproxy1",
                "source": {
                        "type": "copy",
                        "certificate": "",
                        "base-image": "87530b2678f03cbc51a7e1732acfed679c4a15ba1                                                                                        939197cbda733346ef28ad9",
                        "source": "haproxy/snap0"
                },
                "instance_type": ""
        }
DBUG[10-29|06:18:44] Got operation from LXD
DBUG[10-29|06:18:44]
        {
                "id": "7e33fa7a-8326-40c6-9e48-015800f1a7f0",
                "class": "task",
                "description": "Creating container",
                "created_at": "2018-10-29T06:18:44.52579756Z",
                "updated_at": "2018-10-29T06:18:44.52579756Z",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/haproxy1",
                                "/1.0/containers/haproxy/snap0"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": ""
        }
DBUG[10-29|06:18:44] Sending request to LXD                   etag= method=GET u                                                                                        rl=http://unix.socket/1.0/operations/7e33fa7a-8326-40c6-9e48-015800f1a7f0
DBUG[10-29|06:18:44] Got response struct from LXD
DBUG[10-29|06:18:44]
        {
                "id": "7e33fa7a-8326-40c6-9e48-015800f1a7f0",
                "class": "task",
                "description": "Creating container",
                "created_at": "2018-10-29T06:18:44.52579756Z",
                "updated_at": "2018-10-29T06:18:44.52579756Z",
                "status": "Running",
                "status_code": 103,
                "resources": {
                        "containers": [
                                "/1.0/containers/haproxy1",
                                "/1.0/containers/haproxy/snap0"
                        ]
                },
                "metadata": null,
                "may_cancel": false,
                "err": ""
        }
Error: not found

What does lxc info haproxy show you?