Workflow for deleting/replacing a VM root device/disk

Incus
, ,
1

Hi all,

tl;dr:

  • what’s the expected workflow to remove/replace an instance/VM root device?
  • why incus config device override $instance root && incus device remove $instance root doesn’t lead to a removed root disk?

Context:

I’m on IncusOS. I’ve created a VM from an Incus-provided template (/rockylinux/10/default) whose full config is:

% incus config show INSTANCE --expanded
architecture: x86_64
config:
  boot.autostart: "false"
  image.architecture: amd64
  image.description: Rockylinux 10 amd64 (20260404_02:06)
  image.os: Rockylinux
  image.release: "10"
  image.requirements.cdrom_agent: "true"
  image.serial: "20260404_02:06"
  image.type: disk-kvm.img
  image.variant: default
  limits.cpu: "2"
  limits.memory: 2GiB
  security.secureboot: "false"
  volatile.base_image: 8bd4c4658d804dd7bef50ae845a406fddc5246ab5d929b5018bd258148371aaa
  volatile.cloud-init.instance-id: 2ae74b12-6589-438d-9dc1-ac03f7c422fb
  volatile.eth0.hwaddr: 10:66:6a:1b:f3:40
  volatile.eth1.hwaddr: 10:66:6a:fa:f1:d2
  volatile.last_state.power: STOPPED
  volatile.uuid: 69925ca6-f0b5-4d85-8545-f31ce8c8bb31
  volatile.uuid.generation: 69925ca6-f0b5-4d85-8545-f31ce8c8bb31
  volatile.vm.definition: pc-q35-10.2
  volatile.vm.needs_reset: "true"
  volatile.vm.rtc_adjustment: "-1"
  volatile.vm.rtc_offset: "-1"
  volatile.vsock_id: "409498921"
devices:
  agent:
    source: agent:config
    type: disk
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  eth1:
    network: incusbr202
    type: nic
  root:
    path: /
    pool: local
    size: 15GiB
    type: disk
ephemeral: false
profiles:
- default
- vm-agent
- vm-default-no-sb
stateful: false
description: ""

I’d like to swap this VM’s root device with another disk image (upstream’s official cloud image) without destroying/recreating the instance.

(I know it would be trivial to do it this way, but I often have the need to swap VMs’ root disk for work. I do it with Proxmox all the time.)

I didn’t find a way to do it in the webUI, so I went to the CLI but incus config device remove INSTANCE root gave:

Device from profile(s) cannot be removed from individual instance. Override device “root” or modify profile instead

Researching this error message mostly gave results for resizing the root device, not removing/replacing it (e.g. Replace or decrease vm's root device, which despite its title it is about shrinking not deleting/replacing).

I’ve looked in the docs and the forums but haven’t found something relevant.

I tried:

% incus config device override INSTANCE root
Device root overridden for INSTANCE
% incus config device remove INSTANCE root
Device root removed from INSTANCE

But that didn’t actually remove the root device:

% incus config show INSTANCE --expanded
...
devices:
  agent:
    source: agent:config
    type: disk
  eth0:
    name: eth0
    network: incusbr0
    type: nic
  eth1:
    network: incusbr202
    type: nic
  root:
    path: /
    pool: local
    size: 15GiB
    type: disk
...

Does it mean an instance cannot be without a root device? A search for “incus diskless instance” doesn’t return anything useful for me. Could it be a leftover of Always set a root disk entry · lxc/incus@0ee94b5 · GitHub? That’s from 2016 though…

I’m not sure I understand the logic behind having instances’ devices “locked” to the profile they were created with. Is it so that all related instances can be updated by updating their attached profiles?

Thanks for any help and/or clarifications!

2

What you’re referring to is probably an incus rebuild

Wipe the instance root disk and re-initialize with a new image (or empty volume).

1 Like
3

Removing the root disk or creating an instance without a root disk is impossible:

incus create --no-profiles --empty test
Creating test
Error: Failed creating instance record: Failed initializing instance: Failed getting root disk: No root device could be found

Replacing the root volume directly isn’t possible AFAIK, but you can use the publish command to export your instance as an image, and the rebuild command to wipe your instance clean and replace its root disk with an image.

The override command creates an override, which you remove with the remove command. You’re essentially left with the same device defined in the profile.

1 Like
4

Thanks for the explanations @bensmrs, it does make things clearer. Do you know the rationale for this limitation?

I’ll look into publish/rebuild to adapt my worflow to it, thanks a lot @hi-ko and @bensmrs.

5

I don’t, and to be fair, I’m quite unhappy about it :slight_smile: (not particularly interested in swapping disks, but I have a use case for diskless VMs).
I’m sure @stgraber has the big picture in mind.

1 Like
6

Thanks for your input, diskless VMs have quite a number of use cases indeed (we do a lot of PXE-booted RAM-only volatile clients for one thing).

I can confirm incus rebuild --empty INSTANCE works for having a root disk device only in name, with a size of 88 KiB :smile: Now I’m looking into importing Rocky upstream cloud-init QCOW2 images as a custom volume/image and rebuild my instance from that.