I was tracking an issue that I can’t find right now… semi-related to https://github.com/lxc/lxd/issues/3775 but was more about checking if the btrfs subvolume exists before deleting…
I ran across a similar issue in 2.18 and have been waiting for the fix to come through in 2.19… well with the repo freeze, my metal never saw 2.19, but I finally got 2.20 the other day and the issue still exists so that didn’t fix it…
The issue that I’m facing only appears to happen when I delete a container through the REST api… I’ve never seen it happen when I delete via the CLI.
And it’s sporadic, but frequent… ‘sometimes’ I can delete without error, but usually the REST api pops a 400. After which, the container is actually deleted, and the subvolume is actually gone.
So again, my metal is 2.20 with btrfs storage… reproducing now…
lxc info:
root@wyzsrv:~# lxc info
config:
core.https_address: '[::]:8443'
core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses:
- 192.168.1.75:8443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIIFQTCCAymgAwIBAgIRALo7K41xPnu2nKOEJiWH3jowDQYJKoZIhvcNAQELBQAw
NDEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEUMBIGA1UEAwwLcm9vdEB3
eXpzcnYwHhcNMTcxMDE4MTgxNDI4WhcNMjcxMDE2MTgxNDI4WjA0MRwwGgYDVQQK
ExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QHd5enNydjCCAiIw
DQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMBYv9/zg+yPeBR/l+wco1hkr6Zl
TVaBamgcCbySoYYqoBCQUiRig/37gzESMJScFLtYPAelPZKbUlo+1gldr3eKXgG8
1rK2CWRuTj0oCAgknZC4fQWObDEHJKqvr7BjZRNJachs7fz+pMtoH6zg3NSmsFnU
+LuRuk+tSozpTpqBBjsBwXlAL+nWAr6UfjreiZamqC1kVb/gMYKvqYaTUQ7+uv7B
DR4ABNsOarkGiAXmgfUevIgXnDkekOIC8JybrxNvRJn2ifyPI4AMwH2R9l2KUX7j
K8w7pKhzsfyVxeLKFAdD8m5QyGiaAWmIumNLX+t6nKATK2RToU2NtVlWZNePVp6D
3PER8dCXJ29ek3e5vYN2wAX9ixGScTJVk/aG7Jrij5mY538IsRb+BMm+U1S5NQVw
iWMl3HYyo24Fc3E3+SNDDL/D+hewU0Zp7KMCu0ACte+ltXjDIr84FnV+hWAmru4d
PWELhhqoRBZQG+3WGKDyuz0gXZmjlLa+PQKCFQRmy35DAqa0+NTzwzjhfWCpQHc4
84DiWZ7FSX/yObKk0ts0h9sBU/k0/NRnvlbBtts8dujbVcBBghyvbXlIaV0GpHCJ
Z24iPjVrmLHTdd+0mKaFHnJ8P+Fyesnghl4DkznurcRPt3f15wGfgkQEiGeTo4IE
Sn0rDo7T7muwIg1XAgMBAAGjTjBMMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAK
BggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMBcGA1UdEQQQMA6CBnd5enNydocEwKgB
SzANBgkqhkiG9w0BAQsFAAOCAgEAai3yF7lYVUwH08NWXFWwYyxuApLxWtDjYEC4
5Titd5w1SERkznAWEe6G88q7InDqIF5w6z7In46mMjfd8BPSdV0+0Z0m2IYBul7L
rvYEuT3vywJWNZuidzFLacLQVoMoQiAK6eFUWZ13s9NDxDXOw34fLIEoDmylFf/W
ilZ/4He8HFRCikFG67F1qrVTJw0ODDdDjuuWqsl9pzjsHDub5qpMFQvyMMRE76lJ
fnjGylEtrUpUxIfwtd/bU9iEFU6NNvaM/eZCkL5C3hK6Qihj8MIG/e79b4FXUt8U
bHTFZyjIy7QuQN6tJYp+PqjwNCtnoHXP36mCTp+0eTR5g9o0XkFNzR+FaBYtHNBA
4O44EQDOxfrOwaGcC5HvSQWLjVSZwbvcZXTZ65ZFJgkQtwaCwUmgTlAY0QcqA+Mz
l3bC5SDzLa6tf/xYFFs6aJzsOJemgUkeZDWCgYdDX/53NZDd6ju19/V3H0k4JhKW
zyy3ZdnWMYkV1s70WB/gScoOzLPNJgD3Ai4YL036xRxXE1xt6zqa7z10F54xQUVL
T5r3HB3x/FFjW+ipUAFVGMV7fP4HCnpcVks2qX1I9PE6bVACaLf2pFSUKhPn5dSq
ZgAQeCLDCwDNEIg1NsLxAEWnAHusytGbCnXfOgxi8rY8zgbcoRkTfR69OTpZbEOB
z8Ls2V8=
-----END CERTIFICATE-----
certificate_fingerprint: 332a47ba4f50d5365fb13fb2fc16281f14a577bc3b9e22a42f970908707b8e08
driver: lxc
driver_version: 2.0.8
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.4.0-101-generic
server: lxd
server_pid: 1785
server_version: "2.20"
storage: btrfs
storage_version: "4.4"
so this next command loops through ruby test-kitchen, & through my driver that I’m writing which via the REST api does a force stop, followed by a delete… no real diagnostic info available without rewriting something… though my CI on the submodules may be more helpful - let me know if needed. (That’s how I know it’s a 400 at least)
C:\Users\Sean\Documents\projects\lxd_nexus>kitchen destroy 16
-----> Starting Kitchen (v1.17.0)
WARN: Unresolved specs during Gem::Specification.reset:
minitest (~> 5.1)
rake (>= 0)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
-----> Destroying <lxd-ubuntu-1604>...
Utilizing REST interface at https://wyzsrv:8443
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Failed to complete #destroy action: [An existing connection was forcibly closed by the remote host.] on lxd-ubuntu-1604
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
and on the host we get:
root@wyzsrv:~# systemctl status lxd
● lxd.service - LXD - main daemon
Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
Active: active (running) since Fri 2017-11-24 12:59:55 MST; 7h ago
Docs: man:lxd(1)
Process: 1789 ExecStartPost=/usr/bin/lxd waitready --timeout=600 (code=exited, status=0/SUCCESS)
Process: 1758 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
Main PID: 1785 (lxd)
Tasks: 20
Memory: 123.4M
CPU: 53.441s
CGroup: /system.slice/lxd.service
├─1785 /usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
├─1956 [lxc monitor] /var/lib/lxd/containers test
├─2105 [lxc monitor] /var/lib/lxd/containers test2
└─4938 [lxc monitor] /var/lib/lxd/containers lxd-ubuntu-1404-25cb89ff3f43e9e4
Nov 24 12:59:52 wyzsrv systemd[1]: Starting LXD - main daemon...
Nov 24 12:59:53 wyzsrv lxd[1785]: lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2017-11-24T12
Nov 24 12:59:55 wyzsrv systemd[1]: Started LXD - main daemon.
Nov 24 19:58:13 wyzsrv lxd[1785]: err="Failed to run: btrfs subvolume delete /var/lib/lxd/storage-pools/default/containers/lxd-ubuntu-16
lines 1-20/20 (END)
and again the container is deleted and the subvolume is properly deleted
what more log output might you need for this one?