Unprivileged container - tar: can't create node ./dev/urandom: Operation not permitted

LXC
1

I am trying to run unprivileged lxc container in openwrt (lxc 2.1.1). I am getting Operation not permitted error…
Any idea? Thanks

root@OpenWrt:~# lxc-create --name ubuntu -t download -- --server images.linuxcontainers.org -d ubuntu -r bionic -a amd64
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs
tar: can't create node ./dev/ptmx: Operation not permitted
tar: can't create node ./dev/tty: Operation not permitted
tar: can't create node ./dev/urandom: Operation not permitted
tar: can't create node ./dev/random: Operation not permitted
tar: can't create node ./dev/full: Operation not permitted
tar: can't create node ./dev/zero: Operation not permitted
tar: can't create node ./dev/null: Operation not permitted
 
---
You just created an Ubuntu bionic amd64 (20180719_08:51) container.
 
To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

Host mounts:

root@ubuntu:~# cat /proc/self/mounts
/dev/root / ext4 rw,noatime,block_validity,delalloc,barrier,user_xattr 0 0
none /dev tmpfs rw,relatime,size=492k,mode=755,uid=100000,gid=100000 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
sysfs /sys/devices/virtual/net sysfs rw,relatime 0 0
sysfs /sys/devices/virtual/net sysfs rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys/fs/fuse/connections sysfs rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,noatime 0 0
sysfs /sys/kernel/security sysfs rw,nosuid,nodev,noexec,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
proc /proc/sys/fs/binfmt_misc proc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/null tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
tmpfs /dev/zero tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
tmpfs /dev/full tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
tmpfs /dev/urandom tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
tmpfs /dev/random tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
tmpfs /dev/tty tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
devpts /dev/console devpts rw,nosuid,noexec,relatime,mode=600,ptmxmode=000 0 0
devpts /dev/pts devpts rw,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/ptmx devpts rw,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty1 devpts rw,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty2 devpts rw,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty3 devpts rw,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/tty4 devpts rw,relatime,gid=100005,mode=620,ptmxmode=666,max=1024 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,uid=100000,gid=100000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755,uid=100000,gid=100000 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,uid=100000,gid=100000 0 0
tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,mode=755,uid=100000,gid=100000 0 0

From container:

root@ubuntu:~# ls -lha /dev/
total 4.0K
drwxr-xr-x  5 root   root       420 Jul 19 19:15 .
drwxr-xr-x 21 root   root      4.0K Jul 19 17:28 ..
c--x--x--x  1 root   root    136, 1 Jul 19 19:15 console
lrwxrwxrwx  1 root   root        13 Jul 19 19:15 fd -> /proc/self/fd
crw-rw-rw-  1 nobody nogroup   1, 7 Jul 19 16:37 full
drwxrwxrwt  2 nobody nogroup     40 Jul 19 19:15 mqueue
crw-rw-rw-  1 nobody nogroup   1, 3 Jul 19 16:37 null
crw-rw-rw-  1 root   root      5, 2 Jul 19 19:15 ptmx
drwxr-xr-x  2 root   root         0 Jul 19 19:15 pts
crw-rw-rw-  1 nobody nogroup   1, 8 Jul 19 16:37 random
drwxrwxrwt  2 root   root        40 Jul 19 19:15 shm
lrwxrwxrwx  1 root   root        15 Jul 19 19:15 stderr -> /proc/self/fd/2
lrwxrwxrwx  1 root   root        15 Jul 19 19:15 stdin -> /proc/self/fd/0
lrwxrwxrwx  1 root   root        15 Jul 19 19:15 stdout -> /proc/self/fd/1
crw-rw-rw-  1 nobody nogroup   5, 0 Jul 19 19:16 tty
crw--w----  1 root   tty     136, 0 Jul 19 19:15 tty1
crw--w----  1 root   tty     136, 1 Jul 19 19:15 tty2
crw--w----  1 root   tty     136, 2 Jul 19 19:15 tty3
crw--w----  1 root   tty     136, 3 Jul 19 19:15 tty4
crw-rw-rw-  1 nobody nogroup   1, 9 Jul 19 16:37 urandom
crw-rw-rw-  1 nobody nogroup   1, 5 Jul 19 16:37 zero

Can be this related to https://github.com/fgrehm/vagrant-lxc/issues/339#issuecomment-108150622 ?

2

There’s a bunch of stuff that could cause this:

  • the filesystem where the container is supposed to be unpacked is mount with nodev
  • the environment you’re running this in somehow doesn’t allow for creating device nodes (the circle-ci example illustrates this)
    In any case it doesn’t look like the errors you’re seeing are fatal. Are you able to run the container afterwards?
3

I can create device nodes on host OS:

root@OpenWrt:~# mknod -m 666 /tmp/tty c 4 0 && ls -lha /tmp/tty 
crw-rw-rw-    1 root     root        4,   0 Jul 22 11:18 /tmp/tty

Also / , where is saved lxc.lxcpath is mounted without nodev

root@OpenWrt:~# mount
/dev/root on / type ext4 (rw,noatime,block_validity,delalloc,barrier,user_xattr)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,pids,clone_children)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime)
/dev/sda1 on /boot type ext4 (ro,noatime,block_validity,delalloc,barrier,user_xattr)
/dev/sda1 on /boot type ext4 (ro,noatime,block_validity,delalloc,barrier,user_xattr)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)


root@OpenWrt:~# cat /etc/lxc/lxc.conf 
lxc.lxcpath = /srv/lxc

Yes I can run container with this error, but I want to know, where is a probelm.
Any other ideas?
Thanks

4

If you’re creating unprivileged containers you won’t be able to mknod() prior to 4.18 kernels. Could be that the template is not correctly being informed that it is run in a user namespace. You can try and see whether you see the same issue with one of the 3.*.* releases.

5

I am using 4.14.54 kernel. So thats my case.
LXC 3.x.x is not available yet in openwrt. Just 2.1.1
So should I ignore this message unitl 3.x.x or kernel >= 4.18 ?

Or possible workaround (to hide error messages) should be add
–exclude=./rootfs/dev --exclude=./rootfs/var/spool/postfix/dev to tar parameters in lxc-download template ,right?

6

You can ignore them for now, yes.

7

Hi Christian,

Sorry for re-opening this old thread. I have a similar problem, but I cannot even create a container. Same setting (LXC 2.1.1 on OpenWrt). My kernel is Linux OpenWrt 4.19.62

Privileged containers work but I get the following error for unprivileged containers:

lxc-create -n template -t download -- --dist devuan --release ascii --arch amd64
Using image from local cache
Unpacking the rootfs
tar: ./dev/ptmx: Cannot mknod: Operation not permitted
tar: ./dev/tty: Cannot mknod: Operation not permitted
tar: ./dev/urandom: Cannot mknod: Operation not permitted
tar: ./dev/random: Cannot mknod: Operation not permitted
tar: ./dev/full: Cannot mknod: Operation not permitted
tar: ./dev/zero: Cannot mknod: Operation not permitted
tar: ./dev/null: Cannot mknod: Operation not permitted
tar: Exiting with failure status due to previous errors
lxc-create: template: lxccontainer.c: create_run_template: 1473 container creation template for template failed
lxc-create: template: tools/lxc_create.c: main: 329 Error creating container template

Any idea what I can do/investigate?

8

That would likely be a bug with the devuan image. The rootfs tarballs/squashfs should never have files inside the /dev directory, this is causing this issue.

@monstermunchkin can you fix this? We’d want to confirm that all our production distrobuilder yaml files properly wipe /dev to avoid this kind of problem.

9

Hi Stéphane,

Thank you for your quick reply! Unfortunately I get the same error with Debian containers:

lxc-create -n template -t download -- --dist debian --release buster --arch amd64
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs
tar: ./dev/ptmx: Cannot mknod: Operation not permitted
tar: ./dev/tty: Cannot mknod: Operation not permitted
tar: ./dev/urandom: Cannot mknod: Operation not permitted
tar: ./dev/random: Cannot mknod: Operation not permitted
tar: ./dev/full: Cannot mknod: Operation not permitted
tar: ./dev/zero: Cannot mknod: Operation not permitted
tar: ./dev/null: Cannot mknod: Operation not permitted
tar: Exiting with failure status due to previous errors
lxc-create: template: lxccontainer.c: create_run_template: 1473 container creation template for template failed
lxc-create: template: tools/lxc_create.c: main: 329 Error creating container template

Centos on the other hand does not have the same problem:

lxc-create -n template -t download -- --dist centos --release 7 --arch amd64
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created a Centos 7 x86_64 (20190802_07:08) container.

Is there anything I can do myself to fix the debian/devuan container? Is there any way I can be of any help in fixing the image on the server?

Thank you,
alex

10

Here’s the PR that makes sure that images have a clean /dev.