VM KVM console connect, disconnect, cannot reconnect

Incus
1

When I start a VM with incus 6.4 on an Arch Linux x86_64 host, I can run incus console VyOS the first time, and the console works well, but if I disconnect with the Ctrl-A q hotkeys, I cannot connect again to the vm console until I stop and restart the vm. The incus console VyOS command just hangs until I hit Ctrl-A q again. If I try it a few times, I get an error message:

$ sudo incus console VyOS
To detach from the console, press: <ctrl>+a q
Error: Connect to console socket "/run/incus/VyOS/qemu.console": dial unix /run/incus/VyOS/qemu.console: connect: resource temporarily unavailable

The VM is VyOS, configured to use a KVM console. Is there something about the KVM console in this guest which is not compatible with the QEMU console?

2

Welcome!

I see that the nightly builds (rolling release) of VyOS are free, therefore one can download and reproduce what you are getting with such a build.

Can you verify that you are also getting this issue with the nightly builds?

3

Yes, I am seeing this behavior with vyos-1.5-rolling-202408230022-generic-amd64.iso.

I booted this ISO, logged in with the vyos:vyos default username and password, and ran install image taking the defaults. One question in the installer is about KVM or serial console, and I chose KVM. Once the installer was completed, I ran poweroff, removed the ISO disk from the instance, started it again, and used the incus console command to connect.

4

That KVM is not about virtualization but about the Keyboard, Video, Mouse hardware device switch.

Since the issue is related to the keyboard access of the VM, can you try again but select serial console instead of KVM?

5

I deleted the VyOS instance and created a new one, selecting the serial console instead of KVM. This time, I still get the same issue with incus console: the first connection after starting the instance works fine, but if I disconnect, there is nothing displayed when I connect to the console again. If I disconnect and reconnect a few times, I get the error message about resource temporarily unavailable.

I tested a different VM using an Alpine image from the default images remote, and with the Alpine image, I can disconnect and reconnect to the console without any problems.

incus launch images:alpine/3.20 --config security.secureboot=false --vm --console  # works OK
6

Can you try using incus console --type vga to connect to the VM?
You should be able to reconnect to the session and continue working exactly where you left off.

incus console with the --type=vga is more suitable to containers.

There’s no VyOS image for Incus at https://images.linuxcontainers.org/ while it should be quite possible to have one.

7

Using a VGA console is not straightforward—the incus server I’m using is headless and I’m connecting to it over ssh.

8

You could connect to an Incus VM with incus exec and get a shell (this is somewhat similar to incus console). However with VMs, the VM needs to have the Incus agent installed.

At the moment, we are using the ISO image of VyOS to boot it into an empty Incus VM. There’s a manual process to install the Incus agent, however it’s not straightforward because the Linux kernel of the VyOS ISO does not support the 9p filesystem (it’s a special filesystem used to send configuration to cloud VMs).

It should be possible to copy the Incus agent files from a proper Incus VM (i.e. images:debian/12) and I spent the last 30 minutes on that. It’s not straightforward to manually install the agent. Of course you can give it a go.

Indeed, for the time being, I suggest to setup SSH on your first access to the VyOS console.

The next step would be to create a VyOS image for Incus and I posted a link with such work.

1 Like
9

I’m curious why the serial console works OK with other VMs but not with the VyOS one—maybe it sets some kernel command line that doesn’t work well with QEMU.

10

The VyOS is a customized Linux distribution.
The Linux kernel in VyOS should have enabled the 9P filesystem, which would make it easy to create Incus VM images.

vyos@vyos:~$ grep 9P /boot/config-6.6.47-amd64-vyos 
# CONFIG_NET_9P is not set
vyos@vyos:~$

Here is an attempt to create some suitable image. However it has now stalled. Develop build VyOS on LXD - Development - VyOS Forums

How do you use VyOS? Isn’t it necessary to use with NIC passthrough? Do you do that?

11

I don’t use VyOS now, but I would like to try it in a homelab setup with NIC passthrough. I found that I need to rebuild the server image so that it enables intel_iommu, so that will have to be done before I can implement NIC passthrough. A few years ago, I ran VyOS in a VM under QEMU, but I may not have tried to use the serial console then. By the way, Distrobuilder added VyOS about a year ago.

12

It turns out to be relatively easy to build a customized VyOS installation ISO image from the rolling release and use a custom kernel config. I was able to add these config items to the kernel and also include the incus-agent Debian package.

CONFIG_NET_9P=m
CONFIG_NET_9P_FD=m
CONFIG_NET_9P_VIRTIO=m
CONFIG_NET_9P_XEN=m
CONFIG_NET_9P_RDMA=m
CONFIG_9P_FS=m
CONFIG_9P_FSCACHE=y
CONFIG_9P_FS_POSIX_ACL=y
CONFIG_9P_FS_SECURITY=y

Now, incus shell my-vyos works OK.