Deletion failed, now I can't delete

LXD
1

Hi

I’m running lxd v 2.21. When I deleted container1 I received the following output.

error: error removing /var/lib/lxd/storage-pools/default/containers/container1: rm: cannot remove ‘/var/lib/lxd/storage-pools/default/containers/container1/rootfs/run’: Device or resource busy

Container1 is still in the stop state and cannot be started. I get the following error when starting.

error: Failed to run: /usr/bin/lxd forkstart container1 /var/lib/lxd/containers /var/log/lxd/container1/lxc.conf:

Any ideas how I can get rid of this?

2

Have you tried with

lxc delete --force container1

The –force parameter might be able to clean up the messy state.

3

No change unfortunately

4

What error do you get with lxc delete --force container1?

The issue is that there is a process running in the container.
There are a few things to try, like restarting LXD,

sudo systemctl restart lxd

Can you share some info on how the container got in a stuck state? Which OS did you run in it, and what apps.

5

The error doesn’t change when forced. Here it is in full

lxc delete --force container1
error: error removing /var/lib/lxd/storage-pools/default/containers/container1: rm: cannot remove ‘/var/lib/lxd/storage-pools/default/containers/container1/rootfs/run’: Device or resource busy
rm: cannot remove ‘/var/lib/lxd/storage-pools/default/containers/container1/rootfs/sys’: Device or resource busy
rm: cannot remove ‘/var/lib/lxd/storage-pools/default/containers/container1/rootfs/dev’: Device or resource busy
rm: cannot remove ‘/var/lib/lxd/storage-pools/default/containers/container1/rootfs/proc’: Device or resource busy

restarting the host makes no difference. The host and container are both running ubuntu xenial. The host in this case is a container on another host. The container in question is running salt and mysql.

6

The set of /run, /sys, /dev, and /proc are partitions that are mounted by the init process.
If the container is really stopped, then when LXD is about to restart, it should not start the container.
My understanding is that LXD tries to restart this container because it did not registered earlier that it has been stopped.

Try the following:

$ lxc config get container1 boot.autostart
true

If you get true, it means that container1 autostarts when LXD starts.
In that case, change to not autostart:

$ lxc config get container1 boot.autostart false

and then restart LXD.

7

I managed to break another one when deleting.

lxc info --show-log mms-live-1
Name: mms-live-1
Remote: unix://
Architecture: x86_64
Created: 2018/01/16 09:25 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

        lxc 20180118124735.421 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124735.421 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124736.217 ERROR    lxc_start - start.c:start:1535 - No such file or directory - Failed to exec "/sbin/init".
        lxc 20180118124736.218 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 7)
        lxc 20180118124736.218 ERROR    lxc_start - start.c:__lxc_start:1459 - Failed to spawn container "mms-live-1".
        lxc 20180118124736.219 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124736.219 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124736.219 ERROR    lxc_container - lxccontainer.c:wait_on_daemonized_start:760 - Received container state "STOPPING" instead of "RUNNING"
        lxc 20180118124736.271 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/blkio/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.271 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/blkio/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.291 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/cpuacct/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.291 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/cpuacct/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.316 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/cpuset/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.317 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/cpuset/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.329 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/devices/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.329 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/devices/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.343 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/freezer/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.343 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/freezer/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.358 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/hugetlb/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.358 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/hugetlb/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.391 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/memory/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.391 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/memory/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.401 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/systemd/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.401 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/systemd/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.412 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/perf_event/lxc/salt-tester-wayne-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.412 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/perf_event/lxc/salt-tester-wayne-1/lxc/mms-live-1
        lxc 20180118124736.412 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124736.412 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
8

Something is really wrong. Let’s grab as much info as possible and post here, so that when someone more knowledgeable has a look, they can easily figure out what is going on.

lxc info

lxc info --show-log container1

9

lxc info

config: {}
api_extensions:

  • storage_zfs_remove_snapshots
  • container_host_shutdown_timeout
  • container_stop_priority
  • container_syscall_filtering
  • auth_pki
  • container_last_used_at
  • etag
  • patch
  • usb_devices
  • https_allowed_credentials
  • image_compression_algorithm
  • directory_manipulation
  • container_cpu_time
  • storage_zfs_use_refquota
  • storage_lvm_mount_options
  • network
  • profile_usedby
  • container_push
  • container_exec_recording
  • certificate_update
  • container_exec_signal_handling
  • gpu_devices
  • container_image_properties
  • migration_progress
  • id_map
  • network_firewall_filtering
  • network_routes
  • storage
  • file_delete
  • file_append
  • network_dhcp_expiry
  • storage_lvm_vg_rename
  • storage_lvm_thinpool_rename
  • network_vlan
  • image_create_aliases
  • container_stateless_copy
  • container_only_migration
  • storage_zfs_clone_copy
  • unix_device_rename
  • storage_lvm_use_thinpool
  • storage_rsync_bwlimit
  • network_vxlan_interface
  • storage_btrfs_mount_options
  • entity_description
  • image_force_refresh
  • storage_lvm_lv_resizing
  • id_map_base
  • file_symlinks
  • container_push_target
  • network_vlan_physical
  • storage_images_delete
  • container_edit_metadata
  • container_snapshot_stateful_migration
  • storage_driver_ceph
  • storage_ceph_user_name
  • resource_limits
  • storage_volatile_initial_source
  • storage_ceph_force_osd_reuse
  • storage_block_filesystem_btrfs
  • resources
  • kernel_limits
  • storage_api_volume_rename
  • macaroon_authentication
  • network_sriov
  • console
  • restrict_devlxd
  • migration_pre_copy
  • infiniband
  • maas_network
    api_status: stable
    api_version: “1.0”
    auth: trusted
    public: false
    auth_methods:
  • tls
    environment:
    addresses:
    architectures:
    • x86_64
    • i686
      certificate: |
      -----BEGIN CERTIFICATE-----
      Cert==
      -----END CERTIFICATE-----
      certificate_fingerprint: ars123
      driver: lxc
      driver_version: 2.1.1
      kernel: Linux
      kernel_architecture: x86_64
      kernel_version: 3.13.0-91-generic
      server: lxd
      server_pid: 2336
      server_version: “2.21”
      storage: dir
      storage_version: “1”

:~# lxc info --show-log mms-live-1
Name: mms-live-1
Remote: unix://
Architecture: x86_64
Created: 2018/01/16 09:25 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

        lxc 20180118124735.421 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124735.421 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124736.217 ERROR    lxc_start - start.c:start:1535 - No such file or directory - Failed to exec "/sbin/init".
        lxc 20180118124736.218 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 7)
        lxc 20180118124736.218 ERROR    lxc_start - start.c:__lxc_start:1459 - Failed to spawn container "mms-live-1".
        lxc 20180118124736.219 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124736.219 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124736.219 ERROR    lxc_container - lxccontainer.c:wait_on_daemonized_start:760 - Received container state "STOPPING" instead of "RUNNING"
        lxc 20180118124736.271 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/blkio/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.271 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/blkio/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.291 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/cpuacct/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.291 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/cpuacct/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.316 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/cpuset/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.317 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/cpuset/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.329 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/devices/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.329 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/devices/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.343 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/freezer/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.343 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/freezer/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.358 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/hugetlb/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.358 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/hugetlb/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.391 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/memory/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.391 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/memory/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.401 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/systemd/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.401 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/systemd/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.412 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1248 - Failed to delete "/sys/fs/cgroup/perf_event/lxc/salt-tester-1/lxc/mms-live-1": Invalid argument
        lxc 20180118124736.412 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1286 - Error destroying /sys/fs/cgroup/perf_event/lxc/salt-tester-1/lxc/mms-live-1
        lxc 20180118124736.412 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
        lxc 20180118124736.412 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
10

What storage backend is this using (lxc storage list) and can you post the content of /proc/self/mountinfo?

11

It was will based storage. Normal in a directory. I honestly think that the hard drives were failing simultaneously and causing this grief. I’ve never seen it on another host. There were bad blocks etc.