USB emulation is a bit of a can of worms security wise which is why we’ve generally been avoiding fully emulated USB devices (or really any fully emulated device).
I think it’d be interesting to understand exactly why it is that those particular images require being put on the USB bus. We have special flagging for anything in .iso to be attached as a cdrom drive, which tends to help with those cases so long as the file is named appropriately.
Hi Scott,
Thanks for sharing about LXDWare. I totally new to both incus and LXDware 
But I managed to install incus 0.1 in one embedded board (radxa’s rockpi4c running dietpi) , built lxd-dashboard docker image on another embedded board (orangepi5 running armbian) following Building your own Docker image for the LXD dashboard using source code – LXDWARE and I was able to connect them both following Adding remote hosts in the LXD dashboard – LXDWARE
The only difference is that I use these commands instead of “lxc”
root@rockpi4c:~# incus config set core.https_address [::]
root@rockpi4c:~# incus config trust add-certificate lxdware.crt
root@rockpi4c:~# incus config trust list
Is there anyway to reach out to Mathew Penning ? (I couldn’t find his emailid). Would be happy to donate whatever little I can I hope his project is still live and kicking. LXD-Dashboard is too good to be true. Planning to play with it more and will be happy to share.
Thanks again.
@Student I have already reached out to Matthew Penning and he is working the LXD Dashboard issue. Here is what Matthew had to say on the issue:
Blockquote
Yes, I do plan on supporting Incus as well. There have been some breaking changes with the REST API in their first release 0.1. (Incus 0.1 has been released). The dashboard connects to the LXD and Incus servers using their REST API. I do plan on updating my python port of the software (GitHub - PenningLabs/lxconsole: LXD Graphical Web Console) to work with Incus. At the moment it is having issues as a result of the removal of the /1.0/containers and /1.0/virtual-machines endpoints but I am planning on changing my code to work with the /1.0/instances endpoint here in my next version 0.4.0. I should hopefully have this out in the next few weeks.
You might want to drop by my channel ScottiBYTE Channel and check out some of my LXD content. I have at this point at least 90 videos on LXD and I will be tracking incus on the channel as well.
@stgraber Agreed Stéphane. However, since LXD VM’s really leverage KVM, does it not make sense to have parity with KVM features. Interestingly, in my tests I was unable to successfully boot both TailsOS and any of the Android X86 distros by simply attaching their ISO images. Both do boot on bare metal, but when it comes to a VM they very much want to execute the boot process through USB disk bus emulation for whatever reason. I am struggling to understand exactly what that is. It makes sense that the Android OS bits would want it since Android likes USB disk devices and not SCSI emulated devices.
–Scott
Our goal is to provide a solid and secure VM experience, not to expose everything that QEMU supports. That also lets us control the attack surface by narrowing what’s exposed to the guest and providing tight security profiles on the host.
You can bypass most of those by using raw.qemu or raw.qemu.conf which requires an unrestricted project and also makes it clear, should bugs be filed (especially security issues) that this isn’t a supported environment.
I am not aware of raw.qemu.conf. Do you happen to have a use case example and an explanation of what it does exactly? I agree about a solid and secure virtual container environment as a priority. I didn’t mean to be critical. I was really just trying to understand the differences.
This has some basic examples. In your case, you’d most likely want to add a new device and drive section to add a disk.
Alternatively you can also use raw.qemu to directly pass addition arguments to the qemu process, basically passing anything that’s otherwise listen in qemu’s help/manpage.
Thanks for an interesting project @stgraber and others. 
I just decided to give it a try, because we’re debugging an issue we’re seeing with LXD on particular machines where the host is not running an Ubuntu kernel (Ubuntu 22.04 and 20.04 containers stuck in "System is booting" up state - #6 by perlun - Support - Ubuntu Community Hub). We have nailed down the problem to probably be related to Canonical’s AppArmor patches not being present on the machines where this is happening.
Anyway, to try and debug this I decided to install Incus to see if it exhibits the same issue. I am using the 0.1-202310131918-debian12 package installed using the instructions at the Zabbly repo.
With the current default Ubuntu 22.04 image, I can run incus launch images:ubuntu/22.04 without the above issue, but this obviously using a different image than the official one:
$ incus image info images:ubuntu/22.04
Fingerprint: b9769f7eef4aab2b9710c7fa7ae34cc002277b935ba04ae426dd6f8cf28b07f0
Size: 117.57MiB
Architecture: x86_64
Type: container
Public: yes
Timestamps:
Created: 2023/10/18 00:00 UTC
Uploaded: 2023/10/18 00:00 UTC
Expires: never
Last used: 0001/01/01 00:00 UTC
Properties:
os: Ubuntu
release: jammy
serial: 20231018_07:42
variant: default
type: squashfs
architecture: amd64
description: Ubuntu jammy amd64 (20231018_07:42)
Aliases:
- ubuntu/jammy/default
- ubuntu/jammy/default/amd64
- ubuntu/22.04/default
- ubuntu/22.04/default/amd64
- ubuntu/jammy
- ubuntu/jammy/amd64
- ubuntu/22.04
- ubuntu/22.04/amd64
Cached: no
Auto update: disabled
Profiles: []
I tried adding Canonical’s official source like this:
$ incus remote add ubuntu https://cloud-images.ubuntu.com/releases --protocol=simplestreams`
…but trying to launch an image with incus launch ubuntu:ubuntu/22.04 fails with a The requested image couldn't be found error. 
And listing the images with Incus seems to return an empty list. Any ideas of what I might be doing wrong?
incus remote ls vs lxc remote ls$ incus remote ls
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| NAME | URL | PROTOCOL | AUTH TYPE | PUBLIC | STATIC | GLOBAL |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| images | https://images.linuxcontainers.org | simplestreams | none | YES | NO | NO |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix:// | incus | file access | NO | YES | NO |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu | https://cloud-images.ubuntu.com/releases | simplestreams | none | YES | NO | NO |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
$ lxc remote ls
+-----------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| NAME | URL | PROTOCOL | AUTH TYPE | PUBLIC | STATIC | GLOBAL |
+-----------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| images | https://images.linuxcontainers.org | simplestreams | none | YES | NO | NO |
+-----------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix:// | lxd | file access | NO | YES | NO |
+-----------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu | https://cloud-images.ubuntu.com/releases | simplestreams | none | YES | YES | NO |
+-----------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu-daily | https://cloud-images.ubuntu.com/daily | simplestreams | none | YES | YES | NO |
+-----------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu-minimal | https://cloud-images.ubuntu.com/minimal/releases/ | simplestreams | none | YES | NO | NO |
+-----------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
incus image ls vs lxc image ls$ incus image ls ubuntu:
+-------+-------------+--------+-------------+--------------+------+------+-------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+-------+-------------+--------+-------------+--------------+------+------+-------------+
$ lxc image ls ubuntu: | head -n10
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| a (5 more) | 2d53824fdf89 | yes | ubuntu 17.10 amd64 (release) (20180706) | x86_64 | CONTAINER | 169.51MB | Jul 6, 2018 at 12:00am (UTC) |
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+-----------+-------------------------------+
| a (5 more) | 34bae4293007 | yes | ubuntu 17.10 amd64 (release) (20180706) | x86_64 | VIRTUAL-MACHINE | 307.06MB | Jul 6, 2018 at 12:00am (UTC) |
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+-----
[...]
That would be because Ubuntu publishes their images with only the lxd.tar.xz filetype, not the incus.tar.xz, they are therefore considered as LXD-only images.
Canonical could obviously mark their images as being supported on Incus by adding the incus.tar.xz filetype, but whether they’d actually be willing to do the testing and validation work needed for that seems doubtful given that they produce LXD.
Ah, that makes sense. 
Obviously, I don’t have any insight into the the actual format of the images used here, it’s pretty much a “black box” to me.
I completely agree, this is unlikely given that Incus is a bit of a “competing product” to so speak. In practice, this means that we’ll have to rely on unofficial images for Incus as far as Ubuntu is concerned.
There’s a note at https://images.linuxcontainers.org/ saying “Note that the images found on this image server are unofficial images. Whenever possible, you should try to use official images from your Linux distribution of choice.” - I wonder if it would make sense to mention something there about this. I don’t have any clear idea how it could be worded though…
Yeah, given that distros other than Ubuntu have generally not been providing their own images but instead just been helping us with our own, that message can probably go away.
It was also there as a pretty strong disclaimer that we didn’t really test the images, but that has changed a few months ago with all images going through automated testing prior to publishing now.
Sounds great! Yeah, removing that disclaimer probably makes sense then, to not discourage/scare people away from these images unnecessarily.
All in all, thanks for an interesting project. Will keep an eye on it, and also noted that it’s on its way to be packaged for inclusion in Debian, which is obviously a nice thing as the project matures: Incus - Debian Wiki
Yeah, @gibmat and @freeekanayaka have been working on the Debian packaging side of things.
@ganto has ganto/lxc4 Copr for Fedora.
Gentoo has app-containers/incus – Gentoo Packages from @juippis
There’s also an AUR for ArchLinux here: https://aur.archlinux.org/packages/incus
So slowly seeing some momentum around packaging. It will obviously take time before it starts hitting the main stable repos and some distros will most likely wait for the first LTS release as having to update every few weeks isn’t very practical.
True that, and for Debian that’s pretty much a no-no anyway because of the project’s very strict approach to updating the stable distro (they typically backport security fixes manually rather than importing new versions as-is). But even getting it into sid is a nice step in the right direction from my POV.
Unsnapping lxd was truly a jail break!
It finally can breathe 
If the current version of incus is based on mature 5.x version of lxd why not start using 1.x?
Is the current 0.2 version unstable compared to the LXD?
Now need to unsnap Distrobuilder 
We want Incus to fit into the same release cadence as LXC and LXCFS. Doing a 1.0 would be a bit odd given that it wouldn’t be related to LXC or LXCFS 1.0.
So the current plan is to keep the 0.x releases until we tag all the 6.0 LTS releases in March/April 2024.
Incus 0.x is at least as stable and reliable as your LXD 5.x.
I mentioned that somewhere else recently but the snap just builds itself without any work on our part so we don’t have a good reason to kill it off.
But you also don’t need to use the snap, distributions can package distrobuilder directly (though I’m not aware of any that does) or you can very easily build it from source locally with a simple go install which is how we install it on our own image build infrastructure.
distributions can package distrobuilder directly (though I’m not aware of any that does)
There are some, repology can list them:
At least in Gentoo we’ve had some interest in trying out the more experimental available stages (like musl) via a container or vm first, and distrobuilder makes it pretty safe to test.