GPU + volume (security.shifted true)

I would like to use a GPU in a container, so

incus config set <container-name> security.privileged true
incus config device add <container-name> gpu gpu

then I use a volume

incus storage volume set tank static-website security.shifted true

I’m not sure if I’m doing things correctly, the only way it works is security.shifted to true

if I’m set to false, I get this error

Failed to start device "static-website": Failed mounting volume: Failed shifting custom storage volume "static-website" on storage pool "tank": Idmaps of container and storage volume are not identical

I asked questions to LLMs, especially the latest one, Grok4, and it told me that having both values (shifted & privileged) set to true is not correct. Is that a hallucination?

I see that Scottibyte also uses shifted true when a container is in privileged mode.

my conf

zfs-2.3.3-1
zfs-kmod-2.3.3-1

PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling

security.shifted doesn’t really make sense when security.privileged it used as security.privileged makes it so the container doesn’t use a uid/gid map at all, so by definition, there’s no need to shift anything.

To give a bit more context, I simply want to add an Intel GPU to a container. Is it possible to do this without setting the container to security.privileged=true? Because for now, I can’t use my GPU inside the container.

Incus 6.14

# incus info --resources

GPU:
  NUMA node: 0
  Vendor: Intel Corporation (8086)
  Product: Alder Lake-N [Intel Graphics] (46d4)
  PCI address: 0000:00:02.0
  Driver: i915 (6.12.38-1-lts)
  DRM:
    ID: 0
    Card: card0 (226:0)
    Control: controlD64 (226:0)
    Render: renderD128 (226:128)
  SR-IOV information:
    Current number of VFs: 0
    Maximum number of VFs: 7
root@c1:~# ls -lasht /dev/dri
total 0
0 drwxr-xr-x 9 root root      540 Jul 19 10:32 ..
0 drwxr-xr-x 2 root root       80 Jul 19 10:32 .
0 crw-rw---- 1 root root 226,   0 Jul 19 10:32 card0
0 crw-rw-rw- 1 root root 226, 128 Jul 19 10:32 renderD128

Errors

1

root@c1:~# intel_gpu_top
Failed to initialize PMU! (Permission denied)

2

ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i big_buck_bunny_720p_30mb.mp4 -c:v h264_vaapi output.mp4


[AVHWDeviceContext @ 0x600c42b9e040] libva: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
[AVHWDeviceContext @ 0x600c42b9e040] libva: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed
[AVHWDeviceContext @ 0x600c42b9e040] Failed to initialise VAAPI connection: -1 (unknown libva error).
Device creation failed: -5.
[h264 @ 0x600c42b65840] No device available for decoder: device type vaapi needed for codec h264.
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_vaapi))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Device setup failed for decoder on input stream #0:0 : Input/output error

The only way for me to mount a volume and have access to the GPU is by setting both of the following values to true

container ->  privileged true
volume    ->  shifted true

Is adding a GPU to a container only possible with a privileged container?

incus config device add <container-name> gpu gpu

You want to use a GPU in a container. See here, Sharing a GPU with multiple containers - #2 by qkiel

hello Simos

I followed the instructions in this thread, but I still can’t get my Intel GPU to work inside the container. The only way it works is by setting the container to privileged.

Steps to reproduce

(host) # incus launch images:debian/12/amd64 test-gpu (unprivileged )
(host) # incus config device add test-gpu gpu gpu

(container) # apt install intel-gpu-tools
(container) # intel_gpu_top
(container) # Failed to initialize PMU! (Permission denied)
(container) # vainfo

libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva error: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
libva info: va_openDriver() returns 1
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: va_openDriver() returns -1
vaInitialize failed with error code -1 (unknown libva error),exit

(container) # ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i big_buck_bunny_720p_30mb.mp4 -c:v h264_vaapi -b:v 2M output_vaapi.mp4

libva: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
libva: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed
No device available for decoder: device type vaapi needed for codec h264.
Device setup failed for decoder on input stream #0:0 : Input/output error

I’m using a minimal Arch Linux setup with an Intel N150 GPU. I’ve updated the drivers

(host) # sudo pacman -Syu intel-media-driver

You’re adding GPU without proper GID, for example gid=44 (where 44 is the GID of video group inside your container). Also, make sure that the default user inside container belong to the video and renderer groups.

This is explained in the link @simos gave you.

After that, the ls -al /dev/dri/ command should give you something like this (notice video group):

crw-rw---- 1 root video 226,   0 Nov 15 08:41 card0
crw-rw---- 1 root video 226, 128 Nov 15 08:41 renderD128

Here’s me doing this on an AMD GPU (instead of Intel GPU).
As @qkiel already replied, you missed the gid=44 part.

$ incus launch images:debian/12/cloud mycontainer
Launching mycontainer
$ incus config device add mycontainer mygpu gpu pci=07:00.0 gid=44
Device mygpu added to mycontainer
$ incus file push big_buck_bunny_1080p_h264.mov mycontainer/root/
$ incus shell mycontainer
root@mycontainer:~# apt install -y ffmpeg
...
root@mycontainer:~# ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i big_buck_bunny_1080p_h264.mov -c:v h264_vaapi -b:v 2M output_vaapi.mp4 
...
frame= 1394 fps=157 q=-0.0 Lsize=   16573kB time=00:00:59.52 bitrate=2281.0kbits/s speed=6.72x    
^C
root@mycontainer:~# ffmpeg -i big_buck_bunny_1080p_h264.mov -c:v h264_vaapi -b:v 2M output.mp4
...
frame= 1547 fps=109 q=-1.0 Lsize=   17468kB time=00:01:05.51 bitrate=2184.2kbits/s speed=4.63x    
^C
root@mycontainer:~#

I have added group 44 and the PCI of my GPU. It’s still not working.

root@test-gpu:~# cat /etc/group
utmp:x:43:
video:x:44:
sasl:x:45
Container

[root@incus-arch ~]# lspci
00:00.0 Host bridge: Intel Corporation Alder Lake-N Processor Host Bridge/DRAM Registers
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-N [Intel Graphics]
00:0a.0 Signal processing controller: Intel Corporation Platform Monitoring Technology (rev 01)
Host

[root@incus-arch ~]# incus config show test-gpu
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian bookworm amd64 (20250715_05:24)
  image.os: Debian
  image.release: bookworm
  image.serial: "20250715_05:24"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 26f79e878ed59bbdc524068b66044556490de77772e325c05b7f99e0eec8483d
  volatile.cloud-init.instance-id: 2866a60b-12e9-41f2-87e3-082d0a046763
  volatile.eth0.host_name: veth47224d40
  volatile.eth0.hwaddr: 10:66:6a:e4:a5:44
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 4551d37a-8b6f-4d48-9ace-b0ea0068f135
  volatile.uuid.generation: 4551d37a-8b6f-4d48-9ace-b0ea0068f135
devices:
  mygpu:
    gid: "44"   <----------------
    pci: "00:02.0"  <----------------
    type: gpu
ephemeral: false
profiles:
- default
stateful: false
description: ""
Container

# ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i big_buck_bunny_720p_30mb.mp4 -c:v h264_vaapi -b:v 2M output_vaapi.mp4

[AVHWDeviceContext @ 0x5d0fdb4e13c0] libva: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
[AVHWDeviceContext @ 0x5d0fdb4e13c0] libva: /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so init failed

Driver

Host (archlinux vanilla minimal)

[root@incus-arch ~]# pacman -Qs intel-media-driver
local/intel-media-driver 25.2.5-1
    Intel Media Driver for VAAPI — Broadwell+ iGPUs
Container debian 12

root@test-gpu:~# dpkg -l | grep intel-media-va-driver
ii  intel-media-va-driver:amd64     23.1.1+dfsg1-1            amd64        VAAPI driver for the Intel GEN8+ Graphics family

Can you use HW acceleration on the host?

See this thread on the hardware aspect, [Solved] VAINFO:libva error: /usr/lib/dri/iHD_drv_video.so init failed / Newbie Corner / Arch Linux Forums

Okay, by sharing the logs—especially the one about the driver version—I just realized that the drivers weren’t the same version between my Arch host and my Debian container. I just tested with an Arch container, and everything works now.

I don’t even need to add the group ID. It work.

Host (arch)

[root@incus-arch ~]# pacman -Qs intel-media-driver
local/intel-media-driver 25.2.5-1
    Intel Media Driver for VAAPI — Broadwell+ iGPUs
Container (arch)

# pacman -Qs intel-media-driver
local/intel-media-driver 25.2.5-1
    Intel Media Driver for VAAPI — Broadwell+ iGPUs
Container (arch)

# ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i big_buck_bunny_720p_30mb.mp4 -c:v h264_vaapi -b:v 2M output_vaapi.mp4

(encode-work) [out#0/mp4 @ 0x57b75e46f880] video:41022KiB audio:7125KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 0.244896%
frame= 4271 fps=377 q=-0.0 Lsize=   48264KiB time=00:02:50.80 bitrate=2314.9kbits/s speed=15.1x
env intel-media-driver
Arch Linux Host 25.2.5-1
Arch Linux Container 25.2.5-1
Debian Container 23.1.1+dfsg1-1 (!!)

Debian 12 / GPU N150 :

https://www.reddit.com/r/MiniPCs/comments/1htxoi6/beelink_mini_s13_intel_n150_linux_driver_issues/

2 Likes

The gid is needed if you want to run the commands as non-root in the container.

It helps to provide a full example with the set of commands to make it visible what’s going on. It helps because

  1. it shows exactly what is being tried, so it can be reproduced
  2. it replicates the full set of commands and new users can learn from your experience.

In your case, I would demonstrate that ffmpeg is running successfully on the host, then show that I create the container and set it up, and then try to demonstrate the thing that is not working. When I do that, I often find the solution to my question by myself just because I wrote down the steps in a straightforward way.

1 Like