LXD fails to mount tmpfs into container on start: Operation Not Permitted

LXD
1

I have a container that I mount a tmpfs into to act as a cache directory:

devices:
  cache:
    path: /var/cache/application_name
    source: /mnt/hot/lxd/cache/application_name
    type: disk

The tmpfs looks like:

$ stat /mnt/hot/lxd/cache/application_name
  File: /mnt/hot/lxd/cache/application_name
  Size: 40        	Blocks: 0          IO Block: 4096   directory
Device: 0,39	Inode: 1           Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1001/lxdusers)   Gid: (1000985/ UNKNOWN)
Access: 2022-07-24 20:09:57.170526652 -0700
Modify: 2022-07-24 19:54:07.383827363 -0700
Change: 2022-07-24 19:54:07.383827363 -0700
 Birth: 2022-07-24 19:54:07.383827363 -0700

$ mount | grep application
applicationcache on /mnt/hot/lxd/cache/application_name type tmpfs (rw,nosuid,nodev,relatime,size=10485760k,nr_inodes=10000,mode=755,uid=1001,gid=1000985,inode64)

However when I start the container I get the following error in the logs:

lxc info --show-log application_name       
Name: application_name
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2021/07/16 17:37 PDT
Last Used: 2022/07/24 20:08 PDT

Log:

lxc application_name 20220725030817.785 ERROR    conf - ../src/lxc/conf.c:mount_entry:2459 - Operation not permitted - Failed to mount "/var/lib/lxd/devices/application_name/disk.cache--config.var-cache-application_name" on "/usr/lib/lxc/rootfs/var/cache/application_name"
lxc application_name 20220725030817.785 ERROR    conf - ../src/lxc/conf.c:lxc_setup:4375 - Failed to setup mount entries
lxc application_name 20220725030817.785 ERROR    start - ../src/lxc/start.c:do_start:1275 - Failed to setup container "application_name"
lxc application_name 20220725030817.785 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc application_name 20220725030817.793 WARN     network - ../src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from "eth0" to its initial name "veth7c14d1f2"
lxc application_name 20220725030817.794 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING"
lxc application_name 20220725030817.794 ERROR    start - ../src/lxc/start.c:__lxc_start:2074 - Failed to spawn container "application_name"
lxc application_name 20220725030817.794 WARN     start - ../src/lxc/start.c:lxc_abort:1039 - No such process - Failed to send SIGKILL via pidfd 17 for process 1663
lxc 20220725030822.944 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220725030822.944 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"

The lxd daemon runs as root and the tmpfs permissions are setup to match the uid/gid of the processes in the container, so I don’t know why I’m getting permissions errors.

2

I removed the tmpfs device configuration from the container and it starts up fine without it. I can then add the device configuration back while the container is running and it works as well.

So somehow the permissions are only wrong on starting the container, but not later on.

$ lxc config device add application_name cache disk source=/mnt/hot/lxd/cache/application_name path=/var/cache/application_name

$ sudo touch /mnt/hot/lxd/cache/application_name/asdf

# Inside the container
$ ls /var/cache/application_name/
asdf
3

This sounds like a duplicate of Cannot start lxc containers with gui profile - #11 by stgraber

What version of LXD are you running?

4

I’m on lxd 5.4 and lxc 5.0.0 from the Arch linux repositories. It does sound like the same symptoms as that thread, but that issue was on 5.3. Maybe there’s another edge that fix didn’t address.

5

The issue is with LXC, not with LXD. The snap includes the needed bugfix on top of LXC, the package in the Arch repo does not.

Until either Arch cherry-picks the fix that’s needed or they include the soon to be released lxc 5.0.1, this issue will persist.