Instance stuck deleting forever

Incus
1

I was trying to perform GPU passthrough on a virtual machine and it never started… I tried deleting it by using the --force option but no luck. I set up the autostart on boot flag so it doesn’t matter if I reboot the server, the instance will always try to come back and I won’t be able to remove it :slightly_frowning_face:

It doesn’t matter if I let incus instance delete gaming --force forever, it won’t delete. The instance won’t start either

There’s no operations too

hypervisor@hypervisor:~$ incus operation ls
+----+------+-------------+-------+------------+---------+
| ID | TYPE | DESCRIPTION | STATE | CANCELABLE | CREATED |
+----+------+-------------+-------+------------+---------+

I also tried incus config set game-server boot.autostart=false but it is also stuck forever

journalctl doesn’t show anything useful

May 01 23:29:50 hypervisor systemd[1]: Starting incus.service - Incus - Main daemon...
░░ Subject: A start job for unit incus.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit incus.service has begun execution.
░░ 
░░ The job identifier is 643.
May 01 23:29:50 hypervisor startup[2780]: time="2025-05-01T23:29:50-05:00" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
May 01 23:29:50 hypervisor startup[2780]: time="2025-05-01T23:29:50-05:00" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
May 01 23:29:56 hypervisor startup[2780]: time="2025-05-01T23:29:56-05:00" level=warning msg="Unable to use virtio-fs for device, using 9p as a fallback" device=data driver=disk err="Virtiofsd missing" instance=gaming project=default
May 01 23:29:56 hypervisor startup[2780]: time="2025-05-01T23:29:56-05:00" level=warning msg="Unable to use virtio-fs for device, using 9p as a fallback" device=library driver=disk err="Virtiofsd missing" instance=gaming project=default

Here’s the instance config

architecture: x86_64
config:
  boot.autostart: "true"
  cloud-init.user-data: <hidden>
  limits.cpu: "2"
  limits.memory: 8GiB
  security.nesting: "true"
  volatile.cloud-init.instance-id: a3903009-f568-489a-8cd6-9791dfc00cf7
  volatile.eth0.host_name: tap4db9ef03
  volatile.eth0.hwaddr: 10:66:6a:53:66:02
  volatile.eth0.last_state.ip_addresses: 10.0.8.13,fd42:6111:b330:eb74:1266:6aff:fe53:6602
  volatile.gpu.last_state.pci.driver: amdgpu
  volatile.gpu.last_state.pci.slot.name: "0000:01:00.0"
  volatile.last_state.power: RUNNING
  volatile.uuid: 1c65d753-7ea8-4ad8-8890-1dd8e005cff1
  volatile.uuid.generation: 1c65d753-7ea8-4ad8-8890-1dd8e005cff1
  volatile.vm.definition: pc-q35-9.2
  volatile.vsock_id: "331660789"
devices:
  data:
    path: /home
    pool: virt-main
    source: gaming-data
    type: disk
  eth0:
    ipv4.address: 10.0.8.13
    network: untrusted
    security.acls: gaming-svc
    type: nic
  gpu:
    pci: "01:00.0"
    type: gpu
  library:
    path: /library
    pool: virt-spin0
    source: gaming-library
    type: disk
  root:
    path: /
    pool: virt-main
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

How can I force delete this instance?

2

I let the deleting task and it ended up failing because the API never responded… So I read the docs and found out you could access the incus database directly so this is how I killed the instance

Show all the instances on the table

hypervisor@hypervisor:~$ incus admin sql global "select * from instances"

Which shows all my instances

+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| ID  | NODE ID |      NAME      | ARCHITECTURE | TYPE | EPHEMERAL |         CREATION DATE          | STATEFUL |         LAST USE DATE          | DESCRIPTION | PROJECT ID |     EXPIRY DATE      |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 22  | 1       | torrent        | 2            | 0    | 0         | 2025-03-02T01:04:40.254453437Z | 0        | 2025-05-02T00:10:09.0664545Z   |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 23  | 1       | security       | 2            | 0    | 0         | 2025-03-02T01:04:40.264778213Z | 0        | 2025-05-02T00:10:02.040237571Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 24  | 1       | code           | 2            | 0    | 0         | 2025-03-02T01:04:40.274862082Z | 0        | 2025-05-02T11:56:19.863713461Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 25  | 1       | streaming      | 2            | 0    | 0         | 2025-03-02T01:04:40.284384905Z | 0        | 2025-05-02T00:10:08.520302082Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 26  | 1       | knowledge      | 2            | 0    | 0         | 2025-03-02T01:04:43.854132499Z | 0        | 2025-05-02T00:09:59.453681761Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 27  | 1       | reverse-proxy  | 2            | 0    | 0         | 2025-03-02T01:04:47.280707183Z | 0        | 2025-05-02T00:10:01.673490876Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 31  | 1       | servarr        | 2            | 0    | 0         | 2025-03-02T02:49:49.461970509Z | 0        | 2025-05-02T00:10:03.533386015Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 36  | 1       | game-server    | 2            | 0    | 0         | 2025-03-16T01:05:08.668635828Z | 0        | 2025-05-02T11:56:22.141023689Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 72  | 1       | container-host | 2            | 0    | 0         | 2025-03-29T04:03:47.136836271Z | 0        | 2025-05-02T11:56:20.370785977Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 73  | 1       | bastion        | 2            | 0    | 0         | 2025-03-30T00:24:14.742301069Z | 0        | 2025-05-02T11:56:19.590967591Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 79  | 1       | dns-vpn-0      | 2            | 0    | 0         | 2025-04-01T01:20:29.027593673Z | 0        | 2025-05-02T11:56:21.306810307Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 80  | 1       | dns-home-1     | 2            | 0    | 0         | 2025-04-01T01:20:29.051032116Z | 0        | 2025-05-02T11:56:20.948956777Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 81  | 1       | dns-home-0     | 2            | 0    | 0         | 2025-04-01T01:20:29.049843517Z | 0        | 2025-05-02T11:56:20.656935016Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 82  | 1       | dns-vpn-1      | 2            | 0    | 0         | 2025-04-01T01:20:29.082726555Z | 0        | 2025-05-02T11:56:21.734928218Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 103 | 1       | monitoring     | 2            | 0    | 0         | 2025-04-20T18:26:22.01545395Z  | 0        | 2025-05-02T00:10:00.393213012Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 109 | 1       | registry       | 2            | 0    | 0         | 2025-04-26T01:49:02.367692068Z | 0        | 2025-05-02T01:15:24.345924039Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 111 | 1       | automation     | 2            | 1    | 0         | 2025-04-30T03:13:44.74984352Z  | 0        | 2025-05-02T11:56:19.368732777Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 113 | 1       | vc-support     | 2            | 0    | 0         | 2025-05-01T01:40:06.486922456Z | 0        | 2025-05-02T00:10:10.024727324Z |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+
| 118 | 1       | gaming         | 2            | 1    | 0         | 2025-05-02T03:04:55.495210184Z | 0        | 2025-05-02T04:27:23.59442121Z  |             | 1          | 0001-01-01T00:00:00Z |
+-----+---------+----------------+--------------+------+-----------+--------------------------------+----------+--------------------------------+-------------+------------+----------------------+

Then I delete the one that gives me problems

hypervisor@hypervisor:~$ incus admin sql global "delete from instances where id=118;"
Rows affected: 1

And now all my instances start normally

hypervisor@hypervisor:~$ incus ls
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
|      NAME      |  STATE  |             IPV4             |                      IPV6                       |      TYPE       | SNAPSHOTS |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| automation     | RUNNING | 10.0.1.11 (enp5s0)           | fd42:b2d9:3b0a:cbac:216:3eff:fe61:ebb9 (enp5s0) | VIRTUAL-MACHINE | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| bastion        | RUNNING | 100.64.0.7 (tailscale0)      | fd7a:115c:a1e0::7 (tailscale0)                  | CONTAINER       | 0         |
|                |         | 10.0.1.10 (eth0)             | fd42:b2d9:3b0a:cbac:216:3eff:fed6:a1ec (eth0)   |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| code           | RUNNING | 10.0.7.10 (eth0)             | fd42:1cb6:8b2e:5ef1:216:3eff:febc:76f6 (eth0)   | CONTAINER       | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| container-host | RUNNING | 172.18.0.1 (br-163dcba5aa76) | fd42:6111:b330:eb74:216:3eff:fe05:b7f5 (eth0)   | CONTAINER       | 0         |
|                |         | 172.17.0.1 (docker0)         |                                                 |                 |           |
|                |         | 10.0.8.12 (eth0)             |                                                 |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| dns-home-0     | RUNNING | 10.0.2.2 (eth0)              | fd42:c48c:9352:d791:216:3eff:fe62:166 (eth0)    | CONTAINER       | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| dns-home-1     | RUNNING | 10.0.2.3 (eth0)              | fd42:c48c:9352:d791:216:3eff:fe8f:8030 (eth0)   | CONTAINER       | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| dns-vpn-0      | RUNNING | 100.64.0.4 (tailscale0)      | fd7a:115c:a1e0::4 (tailscale0)                  | CONTAINER       | 0         |
|                |         | 10.0.2.4 (eth0)              | fd42:c48c:9352:d791:216:3eff:fef5:4c90 (eth0)   |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| dns-vpn-1      | RUNNING | 100.64.0.10 (tailscale0)     | fd7a:115c:a1e0::5 (tailscale0)                  | CONTAINER       | 0         |
|                |         | 10.0.2.5 (eth0)              | fd42:c48c:9352:d791:216:3eff:fe4b:9421 (eth0)   |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| game-server    | RUNNING | 10.0.8.11 (eth0)             | fd42:6111:b330:eb74:216:3eff:fef6:101e (eth0)   | CONTAINER       | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| knowledge      | RUNNING | 10.0.0.10 (eth0)             | fd42:624c:8834:e36c:216:3eff:fe60:544b (eth0)   | CONTAINER       | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| monitoring     | RUNNING | 100.64.0.8 (tailscale0)      | fd7a:115c:a1e0::8 (tailscale0)                  | CONTAINER       | 0         |
|                |         | 10.0.9.10 (eth0)             | fd42:30bf:f5c9:a7d5:216:3eff:fe7d:4c1a (eth0)   |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| registry       | RUNNING | 172.18.0.1 (br-0159f2199e6f) | fd42:624c:8834:e36c:216:3eff:fe57:90c8 (eth0)   | CONTAINER       | 0         |
|                |         | 172.17.0.1 (docker0)         |                                                 |                 |           |
|                |         | 10.0.0.14 (eth0)             |                                                 |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| reverse-proxy  | RUNNING | 100.64.0.3 (tailscale0)      | fd7a:115c:a1e0::e (tailscale0)                  | CONTAINER       | 0         |
|                |         | 10.0.5.10 (eth0)             | fd42:6799:5704:55d7:216:3eff:fee3:1fb7 (eth0)   |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| security       | RUNNING | 100.64.0.5 (tailscale0)      | fd7a:115c:a1e0::3 (tailscale0)                  | CONTAINER       | 0         |
|                |         | 10.0.0.11 (eth0)             | fd42:624c:8834:e36c:216:3eff:fe99:e5ed (eth0)   |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| servarr        | RUNNING | 10.0.0.12 (eth0)             | fd42:624c:8834:e36c:216:3eff:fe3e:1de8 (eth0)   | CONTAINER       | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| streaming      | RUNNING | 10.0.0.13 (eth0)             | fd42:624c:8834:e36c:216:3eff:fea0:fc2d (eth0)   | CONTAINER       | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| torrent        | RUNNING | 10.0.6.10 (eth0)             | fd42:4538:d42d:1a2f:216:3eff:feef:88b3 (eth0)   | CONTAINER       | 0         |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+
| vc-support     | RUNNING | 100.68.0.6 (tailscale0)      | fd7a:115c:a1e0::6 (tailscale0)                  | CONTAINER       | 0         |
|                |         | 10.0.10.10 (eth0)            | fd42:fb00:cf51:daae:216:3eff:fea5:1132 (eth0)   |                 |           |
+----------------+---------+------------------------------+-------------------------------------------------+-----------------+-----------+

So I guess it is a bad idea to set an instance to start on boot before making sure it works properly :slightly_smiling_face:

3

Coming back at this and in case it helps anybody… I have managed to get it working! Although not with a system container but with a virtual machine instead

My first mistake was not giving GRUB enough parameters and not blacklisting my GPU on my host before trying to pass it through. Nothing wrong on the Incus side since I also tried libvirt and it hanged the same way… So I added this to my grub configuration at /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=downstream,multifunction"

Which to my understanding allows the ACS patch to work (I’m using Liquorix kernel)

Then I add this to /etc/modprobe.d/amdgpu-blocklist.conf

blacklist amdgpu
blacklist radeon

Which kills my AMD GPU on the host… I have an Nvidia one to be used on the hypervisor so it doesn’t affect its display output

After that… I added this option to my gaming machine configuration

raw.qemu.conf: '[device "qemu_gpu"]'

It killed the default QEMU GPU so that my dedicated one that is being passed through works… Finally, the GPU is passed like this

devices:
 gpu:
    pci: "01:00.0"
    type: gpu

After that… I log in to the instance… Add the non-free-firmware repository and install firmware-amd-graphics… I reboot and now I can see the TTY on my monitor when booting and the only GPU available is the dedicated one

gaming@gaming:~$ lshw -c video
WARNING: you should run this program as super-user.
  *-display                 
       description: VGA compatible controller
       product: Ellesmere [Radeon RX 470/480/570/570X/580/580X/590]
       vendor: Advanced Micro Devices, Inc. [AMD/ATI]
       physical id: 0
       bus info: pci@0000:06:00.0
       logical name: /dev/fb0
       version: e7
       width: 64 bits
       clock: 33MHz
       capabilities: vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=amdgpu latency=0 resolution=3840,2160
       resources: iomemory:70100-700ff iomemory:70100-700ff irq:67 memory:701000000000-70100fffffff memory:701010000000-7010101fffff ioport:e000(size=256) memory:c1000000-c103ffff memory:c1040000-c105ffff
WARNING: output may be incomplete or inaccurate, you should run this program as super-user.

In case it helps anyone, here’s my Terraform/Opentofu instance configuration

resource "incus_instance" "gaming" {
  name  = "gaming"
  image = "images:debian/bookworm/cloud"
  type= "virtual-machine"

  config = {
    "boot.autostart" = false
    "limits.cpu"     = 2
    "limits.memory" = "8GiB"
    "cloud-init.user-data" = templatefile(
      "${path.root}/cloud-init/generic.yml.tftpl",
      {
        ssh_public_key           = var.ssh_public_key
        lxc_username             = "gaming"
        lxc_username_description = "Game Server Linux System Container"
        lxc_user_password        = var.lxc_user_password
        ansible_password         = var.ansible_password
      }
    )
    "security.nesting" = true
    "raw.qemu.conf": "[device \"qemu_gpu\"]"
  }
  device {
    name = "eth0"
    type = "nic"
    properties = {
      network        = var.untrusted_net.name
      "ipv4.address" = "10.0.8.14"
      "security.acls" = join(",",
        [
          incus_network_acl.gaming_svc.name
      ])
    }
  }
  device {
    name = "root"
    type = "disk"
    properties = {
      path = "/"
      pool = var.virt_main_pool
    }
  }

  device {
    name = "data"
    type = "disk"
    properties = {
      path   = "/home"
      source = incus_storage_volume.gaming_data.name
      pool   = var.virt_main_pool
    }
  }

  device {
    name = "library"
    type = "disk"
    properties = {
      path   = "/library"
      source = incus_storage_volume.gaming_library.name
      pool   = var.virt_spin0_pool
    }
  }

  device {
    name = "gpu"
    type = "gpu"
    properties = {
      pci = "01:00.0"
    }
  }

}
1 Like