Security.idmap.isolated true -> Common start logic: Failed to change ACLs

I am trying to replicate per container maps following https://ubuntu.com/blog/custom-user-mappings-in-lxd-containers

This is what I get:

david@nnwh:~$ lxc launch ubuntu-minimal_20.04_LTS_DA dev
Creating dev
Starting dev

david@nnwh:~$ lxc config set dev security.idmap.isolated true

david@nnwh:~$ lxc restart dev
Error: Common start logic: Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/default/containers/dev/rootfs/var/log/journal/6740eb7bc3c04fcb84447b57fa8f63ee/system@3d0273812d8d449cb48a67c4450a2bfc-0000000000000001-0005aa4fee4819d5.journal
Try lxc info --show-log dev for more info

david@nnwh:~$ lxc info --show-log dev
Name: dev
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/09/13 19:00 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc dev 20200913190053.265 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory “/sys/fs/cgroup/cpuset//lxc.monitor.dev”
lxc dev 20200913190053.267 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory “/sys/fs/cgroup/cpuset//lxc.payload.dev”
lxc dev 20200913190053.274 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1573 - No such file or directory - Failed to fchownat(17, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )

Have you restarted LXD or your computer?

Let’s see. Ubuntu 18.04 for host, running LXD 4.5 from the snap package.

$ lxc launch ubuntu:18.04 mycontainer -c security.idmap.isolated=true
Creating mycontainer
Starting mycontainer  
$ 

Or, add isolation later,

$ lxc launch ubuntu:18.04 mycontainer 
Creating mycontainer
Starting mycontainer
$ lxc stop mycontainer
$ lxc config set mycontainer security.idmap.isolated=true
$ lxc start mycontainer
$ 

Shall we do it live?

$ lxc launch ubuntu:18.04 mycontainer
Creating mycontainer
Starting mycontainer
$ lxc config set mycontainer security.idmap.isolated=true
$ lxc restart mycontainer
$

So, it works, but something very specific to your setup is causing an issue.
Can you test the other two alternatives on your system? What’s your host?

@brauner would you mind spending a bit of time figuring out what’s going on with shifting of /var/log/journal? We’ve seen this happen over and over again even long after we fixed the ACL shifting bug that would have potentially run us out of xattr space.

So it feels like we may still have a bug somewhere but we’re a bit short of a reproducer for this so far…

So far not. But now I did and see this:

root@nnwh:~# systemctl status lxd-agent.service
● lxd-agent.service - LXD - agent
Loaded: loaded (/lib/systemd/system/lxd-agent.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Mon 2020-09-14 07:32:14 UTC; 10s ago
└─ ConditionPathExists=/dev/virtio-ports/org.linuxcontainers.lxd was not met
Docs: https://linuxcontainers.org/lxd

Aug 14 15:19:34 nnwh systemd[1]: Condition check resulted in LXD - agent being skipped.
Sep 14 07:32:14 nnwh systemd[1]: Condition check resulted in LXD - agent being skipped.
root@nnwh:~# systemctl status lxd-agent-9p.service
● lxd-agent-9p.service - LXD - agent - 9p mount
Loaded: loaded (/lib/systemd/system/lxd-agent-9p.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Mon 2020-09-14 07:32:21 UTC; 13s ago
└─ ConditionPathExists=/dev/virtio-ports/org.linuxcontainers.lxd was not met
Docs: https://linuxcontainers.org/lxd

Aug 14 15:19:34 nnwh systemd[1]: Condition check resulted in LXD - agent - 9p mount being skipped.
Sep 14 07:32:14 nnwh systemd[1]: Condition check resulted in LXD - agent - 9p mount being skipped.
Sep 14 07:32:21 nnwh systemd[1]: Condition check resulted in LXD - agent - 9p mount being skipped.

That was just the agent. I found that I need the snap service:
systemctl status snap.lxd.daemon.service
This seems fine.
And after restart the security.idmap.isolated also works.

Thanks!

I need to reopen this issue. For some reason I cannot remove the tick mark for “Solution”.

The issue seems to be my own image.

david@nnwh:~$ lxc delete dev
david@nnwh:~$ lxc launch ubuntu-minimal:focal dev -c security.idmap.isolated=true
Creating dev
Starting dev
david@nnwh:~$ lxc stop dev
david@nnwh:~$ lxc delete dev
david@nnwh:~$ lxc launch ubuntu-minimal_20.04_LTS_DA dev -c security.idmap.isolated=true
Creating dev
Starting dev
Error: Common start logic: Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/default/containers/dev/rootfs/var/log/journal
Try `lxc info --show-log local:dev` for more info
david@nnwh:~$ lxc info --show-log local:dev
Name: dev
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/09/14 09:10 UTC
Status: Stopped
Type: container
Profiles: default

Log:

As soon as one publishes an image the error occurs:

david@nnwh:~$ lxc stop dev
david@nnwh:~$ lxc publish dev --alias ubuntu-minimal_20.04_LTS_isolated
Instance published with fingerprint: 885fc25b25cb06cb9344c2a91240b57f2da8c36ee7af4ddac0eb7c84c63d1a3e
david@nnwh:~$ lxc launch ubuntu-minimal_20.04_LTS_isolated dev2 -c security.idmap.isolated=true
Creating dev2
Starting dev2                              
Error: Common start logic: Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/default/containers/dev2/rootfs/var/log/journal
Try `lxc info --show-log local:dev2` for more info

should hopefully fix it.
1 Like

If this helps:
The same happens when using “copy” on a container with “security.idmap.isolated=true”.
While copying a container without this set copies just fine either to a container with or without it.