Permissions issues with creating a VM through LXD, but not `qemu-system`

I am having trouble with the permssions needed to create a VM with lxc.

Using qemu-system-x86_64 I am able to create a VM

sudo qemu-system-x86_64 \
-name ubuntu2,process=ubuntu2 \
-machine type=pc,accel=kvm \
-cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-m 8G \
-vga qxl \
-spice port=5930,disable-ticketing=on \
-rtc clock=host,base=localtime \
-serial none \
-parallel none \
-drive  id=disk0,if=virtio,cache=none,format=raw,file=/home/admin/VMs/ubuntu.img \
-drive file=/home/admin/ISOs/ubuntu-20.04.3-desktop-amd64.iso,index=1,media=cdrom \
-net tap \
-net nic

And view it with remote-viewer:

sudo remote-viewer spice://127.0.0.1:5930

However, using the same set of commands to create a VM through LXD does not work:

echo "-name ubuntu2,process=ubuntu2 \
-machine type=pc,accel=kvm \
-cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-m 8G \
-vga qxl \
-spice port=5930,disable-ticketing=on \
-rtc clock=host,base=localtime \
-serial none \
-parallel none \
-drive  id=disk0,if=virtio,cache=none,format=raw,file=/home/admin/VMs/ubuntu.img \
-drive file=/home/admin/ISOs/ubuntu-20.04.3-desktop-amd64.iso,index=1,media=cdrom \
-net tap \
-net nic" | lxc config set ubuntu2 raw.qemu -

However starting the VM after setting those permissions results in Permission denied:

~$ sudo lxc start ubuntu2
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited -- /snap/lxd/21902/bin/qemu-system-x86_64 -S -name ubuntu2 -uuid 63f34fd9-8cb5-4e82-9a69-13f47ae43954 -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/ubuntu2/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/ubuntu2/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/ubuntu2/qemu.pid -D /var/snap/lxd/common/lxd/logs/ubuntu2/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd -name ubuntu2,process=ubuntu2 -machine type=pc,accel=kvm -cpu host -smp 4,sockets=1,cores=4,threads=1 -m 8G -vga qxl -spice port=5930,disable-ticketing=on -rtc clock=host,base=localtime -serial none -parallel none -drive id=disk0,if=virtio,cache=none,format=raw,file=/home/admin/VMs/ubuntu.img -drive file=/home/admin/ISOs/ubuntu-20.04.3-desktop-amd64.iso,index=1,media=cdrom -net tap -net nic: char device redirected to /dev/pts/0 (label console)
: Process exited with non-zero value 1
Try `lxc info --show-log ubuntu2` for more info
admin@virtland:~$ lxc info --show-log ubuntu2
Name: ubuntu2
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2021/11/27 18:50 EST
Last Used: 2021/11/27 18:51 EST

Log:

qemu-system-x86_64: -drive id=disk0,if=virtio,cache=none,format=raw,file=/home/admin/VMs/ubuntu.img: Could not open '/home/admin/VMs/ubuntu.img': Permission denied

I tried messing with the permissions in groups in ways that seemed likely, but it did not help:

sudo setfacl -m g:kvm:rx /home/admin
sudo setfacl -R --modify group:kvm:rwx /home/admin/VMs
sudo setfacl -R --modify group:lxd:rwx /home/admin/VMs
sudo setfacl -R --modify user:lxd:rwx /home/admin/VMs


sudo usermod -aG kvm admin
sudo usermod -aG kvm lxd

LXC info:

~$ lxc list
+-------------+---------+------+------+-----------------+-----------+
|    NAME     |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+-------------+---------+------+------+-----------------+-----------+
| lubuntu-1   | STOPPED |      |      | CONTAINER       | 0         |
+-------------+---------+------+------+-----------------+-----------+
| mycontainer | STOPPED |      |      | CONTAINER       | 0         |
+-------------+---------+------+------+-----------------+-----------+
| steam       | STOPPED |      |      | CONTAINER       | 0         |
+-------------+---------+------+------+-----------------+-----------+
| ubuntu2     | STOPPED |      |      | VIRTUAL-MACHINE | 0         |
+-------------+---------+------+------+-----------------+-----------+
| ubuntu-gui  | STOPPED |      |      | CONTAINER       | 0         |
+-------------+---------+------+------+-----------------+-----------+
| x2go-test   | STOPPED |      |      | CONTAINER       | 0         |
+-------------+---------+------+------+-----------------+-----------+

ACL permissions on the file in question:

~$ sudo getfacl -e /home/admin/VMs/ubuntu.img
getfacl: Removing leading '/' from absolute path names
# file: home/admin/VMs/ubuntu.img
# owner: admin
# group: admin
user::rw-
user:lxd:rwx                    #effective:rwx
group::r--                      #effective:r--
group:kvm:rwx                   #effective:rwx
group:lxd:rwx                   #effective:rwx
mask::rwx
other::r--

I tried stopping and disabling apparmor too, but the error persisted.

This could be AppArmor that LXD uses to secure the QEMU process.

Do you see any DENIED messages in sudo dmesg when you start the VM?

1 Like

You are correct. Although I disabled it and it is listed as inactive(dead), somehow it still blocks things.

~$ sudo systemctl status apparmor
● apparmor.service - Load AppArmor profiles
     Loaded: loaded (/lib/systemd/system/apparmor.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:apparmor(7)
             https://gitlab.com/apparmor/apparmor/wikis/home/
~$ lxc start ubuntu3
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited -- /snap/lxd/21902/bin/qemu-system-x86_64 -S -name ubuntu3 -uuid bbd0b463-9cdd-45ce-b46e-abc7b055f504 -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/ubuntu3/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/ubuntu3/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/ubuntu3/qemu.pid -D /var/snap/lxd/common/lxd/logs/ubuntu3/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd -name ubuntu31,process=ubuntu31 -machine type=pc,accel=kvm -cpu host -smp 4,sockets=1,cores=4,threads=1 -m 8G -vga qxl -spice port=5930,disable-ticketing=on -rtc clock=host,base=localtime -serial none -parallel none -drive id=disk0,if=virtio,cache=none,format=raw,file=/home/boss/VMs/ubuntu-new.img -drive file=/home/boss/ISOs/ubuntu-20.04.3-desktop-amd64.iso,index=1,media=cdrom -nic tap,model=virtio-net-pci,script=no: char device redirected to /dev/pts/0 (label console)
: Process exited with non-zero value 1
Try `lxc info --show-log ubuntu3` for more info
~$ sudo dmesg | grep DENIED
[   27.870200] audit: type=1400 audit(1638190592.552:65): apparmor="DENIED" operation="file_mmap" namespace="root//lxd-TORBOX_<var-snap-lxd-common-lxd>" profile="/usr/sbin/cups-browsed" name="/usr/sbin/cups-browsed" pid=7147 comm="cups-browsed" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
[   87.529363] audit: type=1400 audit(1638190652.182:129): apparmor="DENIED" operation="open" profile="lxd-ubuntu3_</var/snap/lxd/common/lxd>" name="/home/boss/VMs/ubuntu-new.img" pid=11616 comm="ubuntu31" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000
[   87.529373] audit: type=1400 audit(1638190652.182:130): apparmor="DENIED" operation="open" profile="lxd-ubuntu3_</var/snap/lxd/common/lxd>" name="/home/boss/VMs/ubuntu-new.img" pid=11616 comm="ubuntu31" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000
~$ lxc start ubuntu3
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited -- /snap/lxd/21902/bin/qemu-system-x86_64 -S -name ubuntu3 -uuid bbd0b463-9cdd-45ce-b46e-abc7b055f504 -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/ubuntu3/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/ubuntu3/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/ubuntu3/qemu.pid -D /var/snap/lxd/common/lxd/logs/ubuntu3/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd -name ubuntu31,process=ubuntu31 -machine type=pc,accel=kvm -cpu host -smp 4,sockets=1,cores=4,threads=1 -m 8G -vga qxl -spice port=5930,disable-ticketing=on -rtc clock=host,base=localtime -serial none -parallel none -drive id=disk0,if=virtio,cache=none,format=raw,file=/home/boss/VMs/ubuntu-new.img -drive file=/home/boss/ISOs/ubuntu-20.04.3-desktop-amd64.iso,index=1,media=cdrom -nic tap,model=virtio-net-pci,script=no: char device redirected to /dev/pts/0 (label console)
: Process exited with non-zero value 1
Try `lxc info --show-log ubuntu3` for more info
~$ sudo dmesg | grep DENIED
[   27.870200] audit: type=1400 audit(1638190592.552:65): apparmor="DENIED" operation="file_mmap" namespace="root//lxd-TORBOX_<var-snap-lxd-common-lxd>" profile="/usr/sbin/cups-browsed" name="/usr/sbin/cups-browsed" pid=7147 comm="cups-browsed" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
[   87.529363] audit: type=1400 audit(1638190652.182:129): apparmor="DENIED" operation="open" profile="lxd-ubuntu3_</var/snap/lxd/common/lxd>" name="/home/boss/VMs/ubuntu-new.img" pid=11616 comm="ubuntu31" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000
[   87.529373] audit: type=1400 audit(1638190652.182:130): apparmor="DENIED" operation="open" profile="lxd-ubuntu3_</var/snap/lxd/common/lxd>" name="/home/boss/VMs/ubuntu-new.img" pid=11616 comm="ubuntu31" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000
[  111.803597] audit: type=1400 audit(1638190676.457:132): apparmor="DENIED" operation="open" profile="lxd-ubuntu3_</var/snap/lxd/common/lxd>" name="/home/boss/VMs/ubuntu-new.img" pid=12799 comm="ubuntu31" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000
[  111.803606] audit: type=1400 audit(1638190676.457:133): apparmor="DENIED" operation="open" profile="lxd-ubuntu3_</var/snap/lxd/common/lxd>" name="/home/boss/VMs/ubuntu-new.img" pid=12799 comm="ubuntu31" requested_mask="r" denied_mask="r" fsuid=0 ouid=1000

Why are you adding a disk via raw.qemu rather than using the built-in LXD disk device type (Instances | LXD). You should try and avoid using raw.qemu for as much as possible, rather than doing lots of things through it.

Because it seems more straightforward to just drop in a qemu-create command than to try and learn LXD equivalents, especially as my understanding is that only some qemu options are implemented.

LXD will add snap-aware AppArmor rules if you use the supported disk device approach, it will also ensure that paths passed to QEMU are snap-aware too.

The raw.* keys are generally unsupported and do not guarantee that “drop-in” config actually works.

If you want to go with that approach then you’ll need to also add apparmor rules using raw.apparmor.

1 Like