Unprivileged LXC on Yocto

pericdario · March 25, 2021, 11:49am

Hello,

I have a problem with running unprivileged LXC on Yocto. LXC version is 3.1.0 and privileged LXC works fine.

Log file : https://pastebin.com/zYJFZs2Y

/etc/subuid and /etc/subgid is set to :

root:1000000:65536

lxc config is :

lxc.idmap = u 0 1000000 65536
lxc.idmap = g 0 1000000 65536
lxc.log.file = /home/user/martini/fedora/logfile.log
lxc.console.logfile = /home/user/martini/fedora/console.log
lxc.rootfs.path = /home/user/fedora/rootfs

Output of cat /proc/self/mounts : https://pastebin.com/Umwd3U5r

lxc-checkconfig shows that everything is enabled.

Thanks in advance.

stgraber · March 25, 2021, 1:25pm

If you can, would be nice to switch to LXC 4.0.x so it’s a bit easier to figure this kind of stuff out. Also bumping logging with lxc.log.level=trace would help too.

williamblair · March 25, 2021, 1:37pm

@pericdario

Are you by chance running LVM volumes? Also, what do you get when you run this:

/sbin/sysctl -a | grep kernel.unprivileged

pericdario · March 25, 2021, 2:23pm

Hi @williamblair,

Output of /sbin/sysctl -a | grep kernel.unprivileged is :

sysctl: error reading key 'net.ipv6.conf.all.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.default.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.eth0.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.ip6_vti0.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.ip6tnl0.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.ip_vti0.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.lo.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.lxcbr0.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.sit0.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.tunl0.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.usb0.stable_secret': Input/output error
sysctl: error reading key 'net.ipv6.conf.vethL8SA7O.stable_secret': Input/output error

I changed owner for /dev/mmcblk0p10 to 1000000:1000000 and mount error for /proc disappeared. Rootfs is located at /dev/mmcblk0p10 so maybe it was one of the causes for the problem.

The new error I’m getting is :

conf - conf.c:lxc_mount_rootfs:1342 - Failed to mount rootfs “/home/user/fedora/rootfs” onto “/usr/lib64/lxc/rootfs” with options “(null)”.

Full log : https://pastebin.com/4H84TSwS

williamblair · March 25, 2021, 2:28pm

@pericdario I was expecting something like this:

root@lxc-host-machine:/home/lxc# /sbin/sysctl -a | grep kernel.unprivileged
kernel.unprivileged_bpf_disabled = 0
kernel.unprivileged_userns_apparmor_policy = 1
kernel.unprivileged_userns_clone = 0

I don’t know why you are getting input/output errors. su to root and type this directly:

/sbin/sysctl -n kernel.unprivileged_userns_clone

Do you get a 0 or a 1 returned?

Also, what does your container config look like? Or did you put in pastebin and I missed it. It should look something like this:

`lxc.group = onboot
lxc.start.autostart = 1

lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = linux64

lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.rootfs.path = btrfs:/home/lxc/.local/share/lxc/wt-relay-01/rootfs
lxc.uts.name = wt-relay-01

lxc.net.0.type = veth
lxc.net.0.link = br0

pericdario · March 25, 2021, 2:38pm

Hi @stgraber,

I have tried switching to LXC 4, however I faced Namespace error:
Failed to clone a new set of namespaces,
that was issued on this post: https://github.com/lxc/lxd/issues/4720
(even though lxc-checkconifg showed that everything was enabled)

You suggested that it was probably a kernel issue but I was able to use LXC 3 with no problems on the same kernel, so I had to go with LXC 3.

pericdario · March 25, 2021, 3:10pm

@williamblair i missed out to copy :

kernel.unprivileged_bpf_disabled = 0

So the output looks like :

kernel.unprivileged_bpf_disabled = 0
sysctl: error reading key ‘net.ipv6.conf.all.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.default.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.eth0.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.ip6_vti0.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.ip6tnl0.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.ip_vti0.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.lo.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.lxcbr0.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.sit0.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.tunl0.stable_secret’: Input/output error
sysctl: error reading key ‘net.ipv6.conf.usb0.stable_secret’: Input/output error

I ran the /sbin/sysctl -a | grep kernel.unprivileged command as root and the output was the same as in my prevous post

I posted LXC config in my first post:

lxc.idmap = u 0 1000000 65536
lxc.idmap = g 0 1000000 65536
lxc.log.file = /home/user/martini/fedora/logfile.log
lxc.console.logfile = /home/user/martini/fedora/console.log
lxc.rootfs.path = /home/user/fedora/rootfs

The output of /sbin/sysctl -n kernel.unprivileged_userns_clone :
sysctl: error: 'kernel.unprivileged_userns_clone' is an unknown key

Maybe kernel.unprivileged_userns_clone is causing the problem?

williamblair · March 25, 2021, 4:10pm

@pericdario type this in and then attempt to bring your container back up:

/sbin/sysctl -w kernel.unprivileged_userns_clone=1

If the container then works, yah!! Then, see if this will make it permanent through reboots:

echo 'kernel.unprivileged_userns_clone=1' > /etc/sysctl.d/userns.conf

I don’t know much about Yoco so it could be different.

pericdario · March 25, 2021, 4:19pm

@williamblair i already tried that. Output is :

sysctl: error: 'kernel.unprivileged_userns_clone' is an unknown key

Maybe this is a kernel problem?

williamblair · March 25, 2021, 4:25pm

Ok , what I was referring was regarding Debian use namespaces specifically (I just found this out) My apologies. So, in light of that, I would take a look and see if user namespaces are enabled on your distro build.

My apologies.

pericdario · March 25, 2021, 4:42pm

Yes, user namespaces are enabled. This patch will probably solve a problem.

williamblair · March 25, 2021, 4:53pm

Please keep us posted and hopefully we’ve helped save you some time at least.

pericdario · March 25, 2021, 7:01pm

After applying patch i got back to problem with proc.

williamblair · March 25, 2021, 9:10pm

What is happening now?

pericdario · March 26, 2021, 10:44am

@williamblair

This is a new log : https://pastebin.com/uL4anB2M
This is a old one without /proc problem : https://pastebin.com/4H84TSwS

I can’t figured out how “/proc error” disappeared before this patch for user namespace.
The good new is that this patch solved a problem :

 ERROR    conf - conf.c:lxc_mount_rootfs:1342 - Failed to mount rootfs "/home/user/fedora/rootfs" onto "/usr/lib64/lxc/rootfs" with options "(null)"

I tried again with changing owner of /dev/mmcblk0p10 but probably that didn’t solve a problem.

Is there possibility that lxc. log didn’t write problem, while the problem is still there?

For example i can mount proc like :

    mkdir /mnt/proc
    mount -t proc proc /mnt/pro

williamblair · March 26, 2021, 4:05pm

@pericdario I don’t think I’m capable of assisting you. My apologies.

@stgraber any ideas?

stgraber · March 26, 2021, 4:10pm

@brauner tends to be the goto for weird environments

pericdario · March 29, 2021, 9:01am

@stgraber

This is a log from LXC 4 : https://pastebin.com/ssCHbEtB

Any idea?

stgraber · March 29, 2021, 2:14pm

Ok, so still unable to mount proc.
Can you try mounting a clean copy of proc somewhere (like you did above) and keep it around, then see if LXC behaves better?

There is a kernel feature which prevents an unprivileged user from mounting a new copy of proc unless a non-overmounted copy of proc is available on the host somewhere accessible.

pericdario · March 29, 2021, 2:44pm

Nothing changed