Incus / LXD profile for GUI apps: Wayland, X11 and Pulseaudio

:warning: Update: new profile using VFS idmap shifting instead of raw.idmap is in the post below in this thread. There’s also a bonus profile for Arch containers.

Old post:
I made a profile for running GUI apps in an Incus / LXD container. It supports wayland, X11 and pulseaudio. This profile is based on work of Justin Ludwig, thank you!
https://blog.swwomm.com/2022/08/lxd-containers-for-wayland-gui-apps.html

Profile has been tested using:

  • both Incus and LXD
  • Ubuntu 22.04 as a host
  • Ubuntu 22.04 container images, both from community repository images:ubuntu/jammy/cloud and official Canonical repository ubuntu:22.04

This post is divided into four parts:

  1. Profile
  2. Explanation
  3. Testing
  4. Troubleshooting

1. Profile

Note 1: profile assumes that your user on the host has a UID and GID 1000, as well as the user in the container has a UID and GID 1000. You can check this using the id -u and id -g commands.

raw.idmap: |-
  uid 1000 1000
  gid 1000 1000

means:

raw.idmap: |-
  uid <host_user_uid> <container_user_uid>
  gid <host_user_gid> <container_user_gid>

Note 2: those Ubuntu images have a default ubuntu user, which is hard-coded in the profile.
Note 3: profile adds some environment variables to .profile file inside container. Every time you change a profile using command incus profile edit <profile_name> it will be applied once again to all containers using it and therefore those environment variables will be duplicated in .profile files. This doesn’t break anything, just be aware of that.

Profile:

config:
  raw.idmap: |-
    uid 1000 1000
    gid 1000 1000
  security.nesting: "true"
  user.user-data: |
    #cloud-config
    package_update: true
    package_upgrade: true
    package_reboot_if_required: true
    packages:
      - pulseaudio-utils
    write_files:
    - path: /usr/local/bin/mystartup.sh
      permissions: 0755
      content: |
        #!/bin/sh
        uid=$(id -u)
        run_dir=/run/user/$uid
        mkdir -p $run_dir && chmod 700 $run_dir && chown $uid:$uid $run_dir
        ln -sf /mnt/.container_wayland_socket $run_dir/wayland-0
        mkdir -p $run_dir/pulse && chmod 700 $run_dir/pulse && chown $uid:$uid $run_dir/pulse
        ln -sf /mnt/.container_pulseaudio_socket $run_dir/pulse/native
        tmp_dir=/tmp/.X11-unix
        mkdir -p $tmp_dir
        ln -sf /mnt/.container_x11_socket $tmp_dir/X0
    - path: /usr/local/etc/mystartup.service
      content: |
        [Unit]
        After=local-fs.target
        [Service]
        Type=oneshot
        ExecStart=/usr/local/bin/mystartup.sh
        [Install]
        WantedBy=default.target
    runcmd:
    - mkdir -p /home/ubuntu/.config/systemd/user/default.target.wants
    - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/default.target.wants/mystartup.service
    - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/mystartup.service
    - chown -R ubuntu:ubuntu /home/ubuntu
    - echo 'export WAYLAND_DISPLAY=wayland-0' >> /home/ubuntu/.profile
    - echo 'export XDG_SESSION_TYPE=wayland' >> /home/ubuntu/.profile
    - echo 'export QT_QPA_PLATFORM=wayland' >> /home/ubuntu/.profile
    - echo 'export DISPLAY=:0' >> /home/ubuntu/.profile
description: GUI Wayland and X11 profile with pulseaudio
devices:
  gpu:
    type: gpu
    gid: 44
  wayland_socket:
    source: /run/user/1000/wayland-0
    path: /mnt/.container_wayland_socket
    type: disk
  x11_socket:
    source: /tmp/.X11-unix/X0
    path: /mnt/.container_x11_socket
    type: disk
  pulseaudio_socket:
    source: /run/user/1000/pulse/native
    path: /mnt/.container_pulseaudio_socket
    type: disk

The easiest way to use a profile like that is to copy it into a text file, then create an empty profile in Incus:

incus profile create <profile_name>

and update that profile with the file’s content:

incus profile edit <profile_name> < /<path>/<file_name>

2. Explanation

For an in depth explanation of how this profile creates a script and startup systemd service that links all sockets to their usual location inside the container, please read Justin’s post at his blog. Here I’ll explain only tweaks I made:

  • Installing pulseaudio-utils package will create pulseaudio cookie inside container.
  • I added X11 socket for apps that don’t use Wayland yet.
  • All sockets are shared as type: disk, not type: proxy device.
  • Adding GPU with gid: 44 enables GPU hardware video acceleration in containers. See Testing section below.
  • Key security.nesting: "true" is for Steam, Docker, etc.

3. Testing

Launch container using this profile:

incus launch images:ubuntu/jammy/cloud -p default -p <profile_name> <container_name>

Now login into your container:

incus exec <container_name> -- sudo --user ubuntu --login

Pulseaudio socket is called native and you can find it at /run/user/1000/pulse/. Wayland socket wayland-0 is in /run/user/1000/ folder, and X11 socket X0 is in /tmp/.X11-unix/ folder. On host in /tmp/.X11-unix/ folder you will find also Xwayland socket as X1.

All those sockets inside container should be visible at /mnt/ and linked into proper folders. Run those command to check if that’s true:

ll /mnt/.container*
ll /tmp/.X11-unix/X?
ll /run/user/*/wayland-?
ll /run/user/*/pulse/native

Check if most important environment variables are set, mainly WAYLAND_DISPLAY=wayland-0 and DISPLAY=:0:

printenv | grep -i display

Check if your user inside container is part of the video group using groups command. Then see if video render is owned by this group using ll /dev/dri/ command. Output should show root video (without gid: 44 it would be root root):

crw-rw---- 1 root video 226,   0 Nov 15 08:41 card0
crw-rw---- 1 root video 226, 128 Nov 15 08:41 renderD128

To test pulseaudio run pactl info command couple of times. If it shows Connection failure: Access denied at least once, then see Troubleshooting section.

For a final test, install Chrome, then run it in Wayland or X11 mode and watch any Youtube video with sound:

sudo apt update && sudo apt upgrade -y
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install ~/google-chrome-stable_current_amd64.deb
sudo apt install libegl1

Without libegl1 package, Chrome will complain. Depending on which ubuntu image you used to create the container, Chrome will spill out some errors, but should work perfectly fine. To run Chrome in Wayland mode, use this command:

google-chrome --enable-features=UseOzonePlatform --ozone-platform=wayland

To run it in X11 mode, simply use this command:

google-chrome

Now on the host you can run xlsclients and see if Chrome shows up when run in X11 mode and if it’s absent when run in Wayland mode.

4. Troubleshooting

If pulseaudio test pactl info showed Connection failure: Access denied, then try copying pulseaudio cookie from host ~/.config/pulse/cookie to container:

incus file push -p --mode=600 --gid=1000 --uid=1000 ~/.config/pulse/cookie <container_name>/home/ubuntu/.config/pulse/

If you still have problems with pulseaudio you may try to disable shared memory inside container in /etc/pulse/client.conf config file manually or with this command:

sed -i "s/; enable-shm = yes/enable-shm = no/g" /etc/pulse/client.conf

If Chrome doesn’t start in X11 mode, you can try changing in profile socket X0 to X1 and corresponding environment variable from DISPLAY=:0 to DISPLAY=:1 .

4 Likes

Ubuntu 22.04 has a new kernel 6.5.0 and now VFS idmap shifting is working for some sockets (wayland and pulseaudio, but not X11), so I updated my profile for GUI apps to take advantage of this.

What’s new

Scripts do exactly what they did before, but are rewritten and are placed in /var/lib/cloud/scripts/ folder, which means they’ll be executed by cloud-init automatically. Subfolder /per-once means that the script will be run once on first boot of an instance, and it won’t be run again even if you clone an instance or create a new instance from a saved image. Subfolder /per-boot means that the script will be run on every boot.

Also, scripts don’t assume the default user in a container is named ubuntu anymore, instead they assume the user’s UID is 1000. This way they don’t have to be tweaked to works for other distros.

  • Script set_up_sockets.sh puts wayland and pulseaudio sockets in proper locations inside container.
  • Script set_up_env_vars.sh sets up required environment variables for the user inside container.
  • Script change_gid.sh makes sure the default user in container with UID 1000 has his GID also 1000. For example, in containers images:ubuntu/jammy/cloud default user ubuntu has GID 1002, which makes sockets and shared folders in container belong to ubuntu:lxd instead of ubuntu:ubuntu. In short, if there is a mismatch this script moves a group that has GID 1000 to the first free GID and assigns 1000 to the group of default user, then fixes ownership of user’s home folder.

I switched from X11 to xWayland socket, and it works perfectly well. You can always change it back by switching socket X1 to X0 and corresponding environment variable from DISPLAY=:1 to DISPLAY=:0.

When using VFS idmap shifting for pulseuadio socket, you have to copy pulse cookie from host into container (--uid=1000 --gid=1000 are the UID and GID of the user inside container):

incus file push -p --mode=600 --uid=1000 --gid=1000 ~/.config/pulse/cookie <container_name>/home/<username>/.config/pulse/

Profile

As before, this profile is for:

  • both Incus and LXD
  • Ubuntu 22.04 container images, both from community repository images:ubuntu/jammy/cloud and official Canonical repository ubuntu:22.04
  • but should work for other distros with very little modifications
config:
  security.nesting: "true"
  cloud-init.user-data: |
    #cloud-config
    package_update: true
    package_upgrade: true
    package_reboot_if_required: true
    packages:
      - pulseaudio-utils
    write_files:
    - path: /var/lib/cloud/scripts/per-boot/set_up_sockets.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        user_name=$( getent passwd ${user_uid} | cut -d: -f1 )
        user_gid=$( getent passwd ${user_name} | cut -d: -f4 )
        if [[ -n ${user_name} ]]; then
          mnt_dir=/mnt/.container_sockets
          run_dir=/run/user/${user_uid}
          [[ ! -d "${run_dir}" ]] && mkdir -p "${run_dir}" && chmod 700 "${run_dir}" && chown ${user_uid}:${user_gid} "${run_dir}"
          [[ ! -d "${run_dir}/pulse" ]] && mkdir -p "${run_dir}/pulse" && chmod 700 "${run_dir}/pulse" && chown ${user_uid}:${user_gid} "${run_dir}/pulse"
          [[ -S "${mnt_dir}/wayland-0" ]] && [[ -d "${run_dir}" ]] && [[ ! -e "${run_dir}/wayland-0" ]] && touch "${run_dir}/wayland-0" && sudo mount --bind "${mnt_dir}/wayland-0" "${run_dir}/wayland-0"
          [[ -S "${mnt_dir}/native" ]]  && [[ -d "${run_dir}/pulse" ]] && [[ ! -e "${run_dir}/pulse/native" ]] && touch "${run_dir}/pulse/native" && sudo mount --bind "${mnt_dir}/native" "${run_dir}/pulse/native"
        fi
    - path: /var/lib/cloud/scripts/per-once/set_up_env_vars.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        home_dir="$( getent passwd ${user_uid} | cut -d: -f6 )"
        profile="${home_dir}/.profile"
        if [[ -f "${profile}" ]]; then
          echo "export WAYLAND_DISPLAY=wayland-0" >> "${profile}"
          echo "export XDG_SESSION_TYPE=wayland" >> "${profile}"
          echo "export QT_QPA_PLATFORM=wayland" >> "${profile}"
          echo "export DISPLAY=:1" >> "${profile}"
        fi
    - path: /var/lib/cloud/scripts/per-once/change_gid.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        user_name=$( getent passwd ${user_uid} | cut -d: -f1 )
        user_gid=$( getent passwd ${user_name} | cut -d: -f4 )
        home_dir=$( getent passwd ${user_name} | cut -d: -f6 )
        if [[ -n ${user_name} && ! ${user_uid} == ${user_gid} ]]; then
          group_to_move=$( getent group ${user_uid} | cut -d: -f1 )
          if [[ -n ${group_to_move} ]]; then
            for gid in {1000..6000}; do
              return_value=$( getent group ${gid} )
              if [[ -z ${return_value} ]]; then
                groupmod -g ${gid} ${group_to_move}
                break
              fi
            done
          fi
          users_group=$( getent group ${user_gid} | cut -d: -f1 )
          groupmod -g ${user_uid} ${users_group}
          chown -R ${user_uid}:${user_uid} "${home_dir}"
        fi
description: GUI Wayland and xWayland profile with pulseaudio, shifting enabled
devices:
  gpu:
    type: gpu
    gid: 44
  pulseaudio_socket:
    type: disk
    shift: true
    source: /run/user/1000/pulse/native
    path: /mnt/.container_sockets/native
  wayland_socket:
    type: disk
    shift: true
    source: /run/user/1000/wayland-0
    path: /mnt/.container_sockets/wayland-0
  xwayland_socket:
    type: proxy
    bind: container
    security.gid: "1000"
    security.uid: "1000"
    connect: unix:@/tmp/.X11-unix/X1
    listen: unix:@/tmp/.X11-unix/X1

Note: profile assumes your user on host (not in container) has UID and GID 1000. You can check it using id -u and id -g commands. If that’s not true, change the following lines to match your UID and GID on host:

  pulseaudio_socket:
    source: /run/user/1000/pulse/native
  wayland_socket:
    source: /run/user/1000/wayland-0
  xwayland_socket:
    security.gid: "1000"
    security.uid: "1000"

Where:

  • /run/user/1000/... means /run/user/<host_user_uid>/...
  • security.gid: "1000" means security.gid: "<host_user_uid>"
  • security.uid: "1000" means security.uid: "<host_user_gid>"

Troubleshooting

See the first post.

If you notice problems with snap packages installed in container, you can try an intermediate profile:

Profile for snaps troubleshooting

This profile uses new scripts to set up sockets, but keeps the raw.dmap setting. Like the old profile, it assumes that your user on the host has a UID and GID 1000, as well as the user in the container has a UID and GID 1000. You can check this using the id -u and id -g commands.

raw.idmap: |-
  uid 1000 1000
  gid 1000 1000

means:

raw.idmap: |-
  uid <host_user_uid> <container_user_uid>
  gid <host_user_gid> <container_user_gid>

Profile for snaps troubleshooting:

config:
  raw.idmap: |-
    uid 1000 1000
    gid 1000 1000
  security.nesting: "true"
  cloud-init.user-data: |
    #cloud-config
    package_update: true
    package_upgrade: true
    package_reboot_if_required: true
    packages:
      - pulseaudio-utils
    write_files:
    - path: /var/lib/cloud/scripts/per-boot/set_up_sockets.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        user_name=$( getent passwd ${user_uid} | cut -d: -f1 )
        user_gid=$( getent passwd ${user_name} | cut -d: -f4 )
        if [[ -n ${user_name} ]]; then
          mnt_dir=/mnt/.container_sockets
          run_dir=/run/user/${user_uid}
          tmp_dir=/tmp/.X11-unix
          [[ ! -d "${tmp_dir}" ]] && mkdir -p "${tmp_dir}" && chmod 777 "${tmp_dir}"
          [[ ! -d "${run_dir}" ]] && mkdir -p "${run_dir}" && chmod 700 "${run_dir}" && chown ${user_uid}:${user_gid} "${run_dir}"
          [[ ! -d "${run_dir}/pulse" ]] && mkdir -p "${run_dir}/pulse" && chmod 700 "${run_dir}/pulse" && chown ${user_uid}:${user_gid} "${run_dir}/pulse"
          [[ -S "${mnt_dir}/X0" ]] && [[ -d "${tmp_dir}" ]] && [[ ! -e "${tmp_dir}/X0" ]] && touch "${tmp_dir}/X0" && sudo mount --bind "${mnt_dir}/X0" "${tmp_dir}/X0"
          [[ -S "${mnt_dir}/wayland-0" ]] && [[ -d "${run_dir}" ]] && [[ ! -e "${run_dir}/wayland-0" ]] && touch "${run_dir}/wayland-0" && sudo mount --bind "${mnt_dir}/wayland-0" "${run_dir}/wayland-0"
          [[ -S "${mnt_dir}/native" ]]  && [[ -d "${run_dir}/pulse" ]] && [[ ! -e "${run_dir}/pulse/native" ]] && touch "${run_dir}/pulse/native" && sudo mount --bind "${mnt_dir}/native" "${run_dir}/pulse/native"
        fi
    - path: /var/lib/cloud/scripts/per-once/set_up_env_vars.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        home_dir="$( getent passwd ${user_uid} | cut -d: -f6 )"
        profile="${home_dir}/.profile"
        if [[ -f "${profile}" ]]; then
          echo "export WAYLAND_DISPLAY=wayland-0" >> "${profile}"
          echo "export XDG_SESSION_TYPE=wayland" >> "${profile}"
          echo "export QT_QPA_PLATFORM=wayland" >> "${profile}"
          echo "export DISPLAY=:0" >> "${profile}"
        fi
    - path: /var/lib/cloud/scripts/per-once/change_gid.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        user_name=$( getent passwd ${user_uid} | cut -d: -f1 )
        user_gid=$( getent passwd ${user_name} | cut -d: -f4 )
        home_dir=$( getent passwd ${user_name} | cut -d: -f6 )
        if [[ -n ${user_name} && ! ${user_uid} == ${user_gid} ]]; then
          group_to_move=$( getent group ${user_uid} | cut -d: -f1 )
          if [[ -n ${group_to_move} ]]; then
            for gid in {1000..6000}; do
              return_value=$( getent group ${gid} )
              if [[ -z ${return_value} ]]; then
                groupmod -g ${gid} ${group_to_move}
                break
              fi
            done
          fi
          users_group=$( getent group ${user_gid} | cut -d: -f1 )
          groupmod -g ${user_uid} ${users_group}
          chown -R ${user_uid}:${user_uid} "${home_dir}"
        fi
description: GUI Wayland and X11 profile with pulseaudio, raw.idmap
devices:
  gpu:
    type: gpu
    gid: 44
  pulseaudio_socket:
    type: disk
    source: /run/user/1000/pulse/native
    path: /mnt/.container_sockets/native
  wayland_socket:
    type: disk
    source: /run/user/1000/wayland-0
    path: /mnt/.container_sockets/wayland-0
  x11_socket:
    source: /tmp/.X11-unix/X0
    path: /mnt/.container_sockets/X0
    type: disk

Bonus profile for Arch containers

The only difference is that in Arch video group has GID 985 and default user is not added to it. So GPU device has this GID and script set_up_env_vars.sh has a new line usermod -a -G video ${user_name}. It also sets missing environment variable XDG_RUNTIME_DIR.

Remember to copy pulse cookie from host into container as shown before, or audio won’t work.

config:
  security.nesting: "true"
  cloud-init.user-data: |
    #cloud-config
    package_update: true
    package_upgrade: true
    package_reboot_if_required: true
    packages:
      - libpulse
    write_files:
    - path: /var/lib/cloud/scripts/per-boot/set_up_sockets.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        user_name=$( getent passwd ${user_uid} | cut -d: -f1 )
        user_gid=$( getent passwd ${user_name} | cut -d: -f4 )
        if [[ -n ${user_name} ]]; then
          mnt_dir=/mnt/.container_sockets
          run_dir=/run/user/${user_uid}
          [[ ! -d "${run_dir}" ]] && mkdir -p "${run_dir}" && chmod 700 "${run_dir}" && chown ${user_uid}:${user_gid} "${run_dir}"
          [[ ! -d "${run_dir}/pulse" ]] && mkdir -p "${run_dir}/pulse" && chmod 700 "${run_dir}/pulse" && chown ${user_uid}:${user_gid} "${run_dir}/pulse"
          [[ -S "${mnt_dir}/wayland-0" ]] && [[ -d "${run_dir}" ]] && [[ ! -e "${run_dir}/wayland-0" ]] && touch "${run_dir}/wayland-0" && sudo mount --bind "${mnt_dir}/wayland-0" "${run_dir}/wayland-0"
          [[ -S "${mnt_dir}/native" ]]  && [[ -d "${run_dir}/pulse" ]] && [[ ! -e "${run_dir}/pulse/native" ]] && touch "${run_dir}/pulse/native" && sudo mount --bind "${mnt_dir}/native" "${run_dir}/pulse/native"
        fi
    - path: /var/lib/cloud/scripts/per-once/set_up_env_vars.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        user_name=$( getent passwd ${user_uid} | cut -d: -f1 )
        home_dir="$( getent passwd ${user_name} | cut -d: -f6 )"
        profile="${home_dir}/.bash_profile"
        usermod -a -G video ${user_name}
        if [[ -f "${profile}" ]]; then
          echo "export WAYLAND_DISPLAY=wayland-0" >> "${profile}"
          echo "export XDG_SESSION_TYPE=wayland" >> "${profile}"
          echo "export QT_QPA_PLATFORM=wayland" >> "${profile}"
          echo "export DISPLAY=:1" >> "${profile}"
          echo "export XDG_RUNTIME_DIR=/run/user/${user_uid}" >> "${profile}"
        fi
    - path: /var/lib/cloud/scripts/per-once/change_gid.sh
      permissions: 0755
      content: |
        #!/bin/bash
        user_uid=1000
        user_name=$( getent passwd ${user_uid} | cut -d: -f1 )
        user_gid=$( getent passwd ${user_name} | cut -d: -f4 )
        home_dir=$( getent passwd ${user_name} | cut -d: -f6 )
        if [[ -n ${user_name} && ! ${user_uid} == ${user_gid} ]]; then
          group_to_move=$( getent group ${user_uid} | cut -d: -f1 )
          if [[ -n ${group_to_move} ]]; then
            for gid in {1000..6000}; do
              return_value=$( getent group ${gid} )
              if [[ -z ${return_value} ]]; then
                groupmod -g ${gid} ${group_to_move}
                break
              fi
            done
          fi
          users_group=$( getent group ${user_gid} | cut -d: -f1 )
          groupmod -g ${user_uid} ${users_group}
          chown -R ${user_uid}:${user_uid} "${home_dir}"
        fi
description: GUI Wayland and xWayland profile with pulseaudio, shifting enabled
devices:
  gpu:
    type: gpu
    gid: 985
  pulseaudio_socket:
    type: disk
    shift: true
    source: /run/user/1000/pulse/native
    path: /mnt/.container_sockets/native
  wayland_socket:
    type: disk
    shift: true
    source: /run/user/1000/wayland-0
    path: /mnt/.container_sockets/wayland-0
  xwayland_socket:
    type: proxy
    bind: container
    security.gid: "1000"
    security.uid: "1000"
    connect: unix:@/tmp/.X11-unix/X1
    listen: unix:@/tmp/.X11-unix/X1
2 Likes

Thanks for this. The instructions on how to run GUI apps in system containers have gone a long way.

It was thanks to your blog posts that I learned about lxd and containers. Isolating things is tremendously useful when playing with software like Webots, etc. So thank you :kissing_heart:

1 Like

How difficult might it be to get this profile working on a Debian 12 host and Debian 12 container?

Would it not be possible to create a container image that is specifically designed to run X11 applications?

This profile should work as is on a Debian 12 host and Debian 12 container.

What do you mean by “specifically designed to run X11 applications”? Without Wayland and Pulseaudio bits?

I mean a container image that includes an X Window server and PulseAudio or other sound libraries that would support a graphical desktop environment. In effect, it would already include the profile that you created or a similar one.

Locally, there are two ways to do that:

  • You can create an image from an instance or a snapshot that uses this profile, and use it as a base for new instances.
  • If you’re familiar with distrobuilder, you can build your own images from scratch with anything you want.
1 Like