Running virtual machines with LXD 4.0

markm · February 17, 2021, 5:30am

If you open the link on a Windows machine it will give you the MS Tool. You can trick it by using Chrome so you get to download the ISO directly. Here is a good example of doing that:

mkg20001 · February 24, 2021, 3:19am

For anyone that didn’t know yet:

If you’re installing this on a server with no good access to display, you can get VNC access by adding the -vnc 0.0.0.0:0 option to raw.qemu like so:

echo -n ' -vnc 0.0.0.0:0 -drive file=/home/USER/Downloads/Win10_1909_English_x64.iso,index=0,media=cdrom,if=ide -drive file=/home/USER/Downloads/virtio-win-0.1.173.iso,index=1,media=cdrom,if=ide' | lxc config set win10 raw.qemu -

NOTE: VNC will only work after you do the part with pressing ESC and booting from the disk (and don’t forget the server firewall, VNC is TCP 5900)

stgraber · February 24, 2021, 4:48am

Right, that still works on some distro but it’s dependent on whether VNC support is built into QEMU.

In general the recommendation is to use lxc console from a remote system which then uses the LXD API and SPICE to attach to the console. This can be done from any Linux, Windows or macOS client after adding the remote server as a remote in the client.

Maihym · February 28, 2021, 4:57am

Take a look here, stgraber provides another way to connect to the virtual machine.

simos · February 28, 2021, 8:22am

I summarized the part from this thread on launching Windows VM at the following post,

In my case,

I could not access the Windows VM IP from the host or other containers. I used ping to check, and also portscanning. The Windows VM got an IP from the lxdbr0 interface as expected, though.
The VM did not have soundcard support, so no audio. I suppose this is a distrobuilder issue?
I tried to setup macvlan (apply macvlan profile after the Windows VM was created but was obviously stopped). The Windows VM could not start due to an error. Could start if I removed the macvlan profile.

Having said that, it was an easy and painless process to setup a Windows VM through LXD.

stgraber · February 28, 2021, 4:33pm

Windows firewall usually prevents pinging and incoming access by default.
There is no virtio-snd is still in its early stages and isn’t part of LXD VMs yet. Given that there isn’t a good VGA driver either, it’s recommended that folks connect over RDP post installation, RDP allows for sound passthrough.
That’s odd, what was the error?

simos · February 28, 2021, 7:05pm

It’s the following.

$ lxc start win10 --console=vga
Error: Failed to start device "eth0": Failed to run: ip link add dev maca2b0d635 link enp3s0 type macvtap mode bridge: RTNETLINK answers: Device or resource busy
Try `lxc info --show-log win10` for more info
$ lxc info --show-log win10
Name: win10
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/27 17:50 UTC
Status: Stopped
Type: virtual-machine
Profiles: default, macvlan
Error: open /var/snap/lxd/common/lxd/logs/win10/qemu.log: no such file or directory

stgraber · February 28, 2021, 7:25pm

Ah, okay, so it’s looking like some kind of compatibility issue with macvlan/macvtap and your network driver.

muslu · March 1, 2021, 2:55pm

Hi all;

I tested Windows vm in LXC on Ubuntu 18.04 and its worked.

Note: remmina (for slice) or virt-viewer can connect vga.

apt install -yqq remmina-plugin-spice virt-viewer

snap install distrobuilder --classic --edge
apt install -y libguestfs-tools wimtools
distrobuilder repack-windows /home/muslu/win10.iso /home/muslu/win10_distrobuilder.iso
lxc init win10 --empty --vm -c security.secureboot=false
lxc config device override win10 root size=50GiB
lxc config device add win10 iso disk source=/home/muslu/win10_distrobuilder.iso boot.priority=10
lxc start win10 --console=vga

If install virt-viewer than vga windows will be open automaticly after lxc start.

makdosdogancan · March 4, 2021, 12:51pm

thanks Muslu. It’s very helpfull.

simos · March 4, 2021, 4:53pm

@muslu, which executable is provided from the wimtools package that is needed by distrobuilder?

muslu · March 5, 2021, 10:39am

Hi all;

Sorry for delay.

I wanted to test all the options.

When i test first time with my own pc (Ubuntu 18.04), i did get error for win parse and i installed wimtools.

i tested same code on my other pc (Ubuntu 18.04) but i did get more errors.

distrobuilder repack-windows /home/muslu/win10.iso /home/muslu/win10_distrobuilder.iso
2021/03/05 13:26:29 Mounting Windows ISO ...
2021/03/05 13:26:29 Mounting driver ISO ...
2021/03/05 13:26:29 Injecting drivers into boot.wim (index 2)...
2021/03/05 13:26:36 Injecting drivers into install.wim (index 1)...
Error: Failed to modify index 1 of "install.wim": Failed to inject drivers: Failed to edit Windows DRIVERS registry for driver "NetKVM": Failed to run: hivexregedit --merge --prefix='HKEY_LOCAL_MACHINE\DRIVERS' /var/cache/distrobuilder.507518749/wim/Windows/System32/config/DRIVERS: reg_import: cannot create \DriverDatabase\DeviceIds\pci\VEN_1AF4&DEV_1000 since parent \DriverDatabase\DeviceIds\pci\ does not exist at /usr/lib/x86_64-linux-gnu/perl5/5.26/Win/Hivex/Regedit.pm line 341, <FILE> line 5.
    Win::Hivex::Regedit::_merge_node(CODE(0x557773bab150), HASH(0x557773ca8f90), "\DriverDatabase\DeviceIds\pci\VEN_1AF4\x{26}DEV_1000", ARRAY(0x557773d2d190), ARRAY(0x557773d2d148)) called at /usr/lib/x86_64-linux-gnu/perl5/5.26/Win/Hivex/Regedit.pm line 196
    Win::Hivex::Regedit::reg_import(GLOB(0x557773d5d960), CODE(0x557773bab150)) called at /usr/bin/hivexregedit line 331

I will try again and report to you.

Thansk for everyting about lxc/lxd/lxcfs/distrobuilder and etc.

stgraber · March 5, 2021, 2:19pm

@monstermunchkin

TomvB · March 11, 2021, 4:58pm

@stgraber How can i limit the bandwidth?

like: lxc config device set VM eth0 limits.ingress 100Mbit

stgraber · March 11, 2021, 5:00pm

limits.ingress and limits.egress should work as expected, though note that they only work with bridges and cannot work with macvlan, physical or sr-iov.

TomvB · March 11, 2021, 8:23pm

Thanks! Using macvlan.

nvm: works now… using root partition only.

TomvB · March 12, 2021, 1:36pm

I have a new test server available for LXD.
CPU: Intel Xeon W3520

egrep -c ‘(vmx|svm)’ /proc/cpuinfo
8

kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Server OS: 20.04
2 disks in RAID1 with LVM
The VM starts after the launch (ipv6 only)

lxc launch ubuntu:20.04 test1 --vm
Then the VM stops (after a few seconds) and I can’t start the VM

Error:
Error: open /var/snap/lxd/common/lxd/logs/test1/qemu.log: no such file or directory

lxc info
driver: lxc | qemu
driver_version: 4.0.6 | 5.2.0
firewall: xtables
kernel: Linux
kernel_architecture: x86_64
kernel_features:

Any idea @stgraber

Update: I cant delete or start the VM after this. Reboot required to delete it.

After a long wait…
user@node:~$ lxc delete test1
Error: Error deleting storage volume: Error removing LVM logical volume: Failed to run: lvremove -f /dev/LXD/virtual-machines_test1.block: Logical volume LXD/virtual-machines_test1.block in use.

lxc start test1
Error: virtiofsd failed to bind socket within 10s
Try lxc info --show-log test1 for more info

stgraber · March 12, 2021, 8:23pm

Anything scary looking in dmesg? Sounds like that system isn’t having a good day.

TomvB · March 13, 2021, 2:42pm

Nope, nothing. Reinstallation doesn’t help either

Container:
t=2021-03-12T20:47:26+0000 lvl=info msg="Started container" action=start created=2021-03-12T11:04:26+0000 ephemeral=false instance=test instanceType=container project=default stateful=false used=2021-03-12T11:20:09+0000

VM:
t=2021-03-12T20:53:31+0000 lvl=warn msg="Error getting disk usage" err="strconv.ParseFloat: parsing \"\": invalid syntax" instance=test1 instanceType=virtual-machine project=default

QEMU log:

KVM internal error. Suberror: 3
extra data[0]: 800000ec
extra data[1]: 31
extra data[2]: 81
extra data[3]: 30db0
RAX=0000000000000000 RBX=0000000000000001 RCX=0000000000000001 RDX=00000000000001ca
RSI=ffff9823b5d1cba8 RDI=0000000000000001 RBP=ffffa46d80077e90 RSP=ffffa46d80077e78
R8 =000000001327186b R9 =0000000000000000 R10=0000000000002000 R11=0000000000002000
R12=0000000000000001 R13=ffff9823401c8c80 R14=0000000000000000 R15=0000000000000000
RIP=ffffffffadf54c14 RFL=00000246 [—Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 ffffffff 00c00000
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0000 0000000000000000 ffffffff 00c00000
DS =0000 0000000000000000 ffffffff 00c00000
FS =0000 0000000000000000 ffffffff 00c00000
GS =0000 ffff9823b5d00000 ffffffff 00c00000
LDT=0000 0000000000000000 ffffffff 00c00000
TR =0040 fffffe0000036000 0000206f 00008b00 DPL=0 TSS64-busy
GDT= fffffe0000034000 0000007f
IDT= fffffe0000000000 00000fff
CR0=80050033 CR2=00000000ffffffff CR3=0000000029a0a000 CR4=000006a0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=00 85 c0 7e 07 0f 00 2d b6 c1 4b 00 fb f4 8b 05 74 8f 78 00 <65> 44 8b 25 14 b5 0b 52 85 c0 0f 8f 85 00 00 00 5b 41 5c 41 5d 5d c3 65 8b 05 fe b4 0b 52

Tested with HWE (without success)
Tested with Proxmox (VM/LXC) instead of Ubuntu 20.04 with LXD and it works.

@stgraber I’ll use proxmox for now. I don’t know why it is going wrong on ubuntu 20.04.

muslu · March 19, 2021, 8:29am

OK!.

distrobuilder is working like a magic.