Share folder errors using shiftfs

[mtm (Mike)]

Setup:
Host:

mike@ford:~$ uname -a
Linux ford 5.4.0-128-generic #144-Ubuntu SMP Tue Sep 20 11:00:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

mike@ford:~$ lxc version
Client version: 5.6
Server version: 5.6

mike@ford:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.5 LTS
Release:	20.04
Codename:	focal

• One main user - uid 1000 gid 1000
• NFS mounted folders in /etc/fstab
• NFS folders are shared onto other Ubuntu 20.04 desktop environments where each user is either the same uid and gid or they are under the same gid to have rw access to the files and folders on the file server

Container:

• Ubuntu Server 20.04 running Nextcloud 23.05 with LAMP
• Default user (uid 1000 gid 1000), www-data (uid 33 gid 33) for Nextcloud + one I created when I attempted to share the NFS mounted folders from the host to the container (uid 1001 gid 1001)

Goal:

• Have the container user www-data rw access to the NFS mounted folders
• Eventually do the same in other, future containers
• Have the existing nfs mounts outside of this server still continue to maintain their current access and permissions

Reproducible Issues:

• After attempting to use shiftfs I run into the error

Error: Failed to handle idmapped storage: invalid argument - Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/default/containers/nextcloud/rootfs/var/log/journal
Try `lxc info --show-log nextcloud` for more info

Let’s see if I can figure out how to explain this… This is the steps, in the order I applied them, and their results:

mike@ford:~$ modinfo shiftfs

filename:       /lib/modules/5.4.0-128-generic/kernel/fs/shiftfs.ko
license:        GPL v2
description:    id shifting filesystem
author:         Christian Brauner <christian.brauner@ubuntu.com>
author:         Seth Forshee <seth.forshee@canonical.com>
author:         James Bottomley
alias:          fs-shiftfs
srcversion:     225AF9C817280FFD72CB9A8
depends:        
retpoline:      Y
intree:         Y
name:           shiftfs
vermagic:       5.4.0-128-generic SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        78:F3:A0:F8:41:58:7F:86:D7:73:F7:84:CC:EE:03:B7:33:B2:EC:6C
sig_hashalgo:   sha512
signature:      

mike@ford:~$ sudo snap set lxd shiftfs.enable=true

mike@ford:~$ sudo systemctl reload snap.lxd.daemon

mike@ford:~$ lxc info
    shiftfs: "true"

mike@ford:~$ lxc config set nextcloud security.privileged true

mike@ford:~$ lxc restart nextcloud

Error: Failed to handle idmapped storage: invalid argument - Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/default/containers/nextcloud/rootfs/var/log/journal
Try `lxc info --show-log nextcloud` for more info

mike@ford:~$ lxc info --show-log nextcloud

Name: nextcloud
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/10/08 20:36 UTC
Last Used: 2022/10/15 00:09 UTC

Log:

Yes, the log is empty.

mike@ford:~$ lxc config show nextcloud -e

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20221003)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20221003"
  image.type: squashfs
  image.version: "20.04"
  security.privileged: "true"
  volatile.base_image: 0c3a7ed4f4133852ca2b91d9157082a2cf53498f3c6cec97fb4a48661c01f1c8
  volatile.cloud-init.instance-id: ec30b506-0a8e-4f3f-a826-5c3b1249e001
  volatile.eth0.hwaddr: 00:16:3e:3c:96:5f
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: b6ed334f-bbee-4e90-bd26-54a1ce9b8e54
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

mike@ford:~$ lxc config set nextcloud security.privileged false

mike@ford:~$ lxc start nextcloud

The container starts up successfully now.

mike@ford:~$ lxc config show nextcloud

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20221003)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20221003"
  image.type: squashfs
  image.version: "20.04"
  security.privileged: "false"
  volatile.base_image: 0c3a7ed4f4133852ca2b91d9157082a2cf53498f3c6cec97fb4a48661c01f1c8
  volatile.cloud-init.instance-id: ec30b506-0a8e-4f3f-a826-5c3b1249e001
  volatile.eth0.host_name: vethe8f20c9d
  volatile.eth0.hwaddr: 00:16:3e:3c:96:5f
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: b6ed334f-bbee-4e90-bd26-54a1ce9b8e54
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

I used (primarily) the following pages to implement shiftfs:

Share folders and volumes between host and containers

Note: In recent kernels (>5.12) a new approach called “idmapped mounts” is directly included in the Linux kernel. It acts as a replacement/sucessor of shiftfs and uses all the below mentioned commands as well. It only supports ext4, xfs, vfat and btrfs (>5.15) for the underlying filesystems of LXD for now, but ZFS and cephfs are planned to be included as well. If you use unsupported filesystems or older kernels, stick with shiftfs for now. Usecases: General Advantages: faster startup o…

Cleanest way to mount folder from host to container rw

Hi all, how can we mount a folder from the host to the container rw mode? in a way i dont have to change user id’s and stuff like that?

Use shiftfs in LXD

Sooooo, what did I miss? Where is my mistake, as I’m sure it’s something I can’t quite see, now working on this yesterday and today, and recreating the results each time. Thanks!

[tomp (Thomas Parrott)]

@mtm

mike@ford:~$ lxc config set nextcloud security.privileged true

Why are you setting this?

[tomp (Thomas Parrott)]

Please can you show lxc config show <instance> --expanded for the config with the directory share added?

[mtm (Mike)]

@tomp

@mtm

mike@ford:~$ lxc config set nextcloud security.privileged true

Why are you setting this?

I set mike@ford:~$ lxc config set nextcloud security.privileged true as the container ‘nextcloud’ is an existing container.

From the post https://discuss.linuxcontainers.org/t/trying-out-shiftfs/5155:

Blockquote
To convert an existing container, the easiest is to temporarily convert it to privileged:

lxc config set NAME security.privileged true
lxc restart NAME
lxc config unset NAME security.privileged
lxc restart NAME

[mtm (Mike)]

@tomp

Please can you show lxc config show <instance> --expanded for the config with the directory share added?

@tomp Oh I thought I did, but perhaps I pasted the incorrect config from my notes.

Here it goes… The config below is is after I enter

lxc config set nextcloud security.privileged true

then

lxc restart nextcloud

where the error message

Error: Failed to handle idmapped storage: invalid argument - Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/default/containers/nextcloud/rootfs/var/log/journal Try lxc info --show-log nextcloud for more info

returns.

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 20.04 LTS amd64 (release) (20221003)
  image.label: release
  image.os: ubuntu
  image.release: focal
  image.serial: "20221003"
  image.type: squashfs
  image.version: "20.04"
  security.privileged: "true"
  volatile.base_image: 0c3a7ed4f4133852ca2b91d9157082a2cf53498f3c6cec97fb4a48661c01f1c8
  volatile.cloud-init.instance-id: ec30b506-0a8e-4f3f-a826-5c3b1249e001
  volatile.eth0.hwaddr: 00:16:3e:3c:96:5f
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: b6ed334f-bbee-4e90-bd26-54a1ce9b8e54
devices:
  Weir:
    path: /home/ubuntu/server/Weir
    shift: "true"
    source: /home/mike/server/Weir/
    type: disk
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

When I unset the security.privileged, then start the container, it starts fine.

Funny thing is, I have another issue that I originally thought was unrelated.

In crafting the replies above, I went through these steps again to recreate it and copy directly from terminal.

When it starts again, there is no error or warning that comes up. When I try to enter the container using

lxc exec nextcloud bash

it replies with

bash: /root/.bashrc: Permission denied
root@nextcloud:~#

and the container, and Nextcloud, ends up being useless and inaccessible, which appeares to be a user, group, and ownership issue, not allowing www-data, or root, access to anything.

This happened prior to the original post, and I thought it was about an id shifting issue caused by my previous attempt to connect my host mounted folders into the containter, and before I attempted shiftfs.

I cleared this error up following a few hints from this post

Container mount with user 'nobody' and group 'nogroup'

Looks like the uid/gid map isn’t applied or the filesystem is wrongly shifted. Can you show (need to run as root): lxc config show --expanded NAME lxc exec NAME – grep shiftfs /proc/mounts ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers/NAME/rootfs/ That may give us some clues as to what’s going on here.

And this one

User "nobody" and group "nogroup" after the migration of the containers

Basically you should just add the lines from the original host to the new one: However I notice that some of your existing maps on the new host overlap with the ID ranges from the old host, albeit with different users, so it may be fine, but I’m not sure. @stgraber can you have multiple users with overlapping ID ranges?

The latter post helped me correct the uid/gid for the file system, and I used another container to compare.

There are much more details involved in these steps, and I don’t want to flood this post with more detail than may be necessary, or was requested in your replies. I appreciate your time.

[tomp (Thomas Parrott)]

Hi,

Did you manage to resolve the issue then?

Thanks

[mtm (Mike)]

No. Sorry, I wasn’t that clear in my response to your original questions.

The original post explains my current issue. While I was attempting to resolve this issue, the second issue appeared. I was able to resolve the second issue and get the container back to square one, so to speak. Then, when I reattempt shiftfs, the first issue returns.

So the error (first issue, original post):

@mtm

Error: Failed to handle idmapped storage: invalid argument - Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/default/containers/nextcloud/rootfs/var/log/journal Try lxc info --show-log nextcloud for more info

still exists.

The second issue

@mtm

bash: /root/.bashrc: Permission denied
root@nextcloud:~#

was resolved.