Run openwrt inside lxd

1

Hi

I’m trying to run openwrt inside lxd.
There has been work made here:
https://github.com/melato/openwrt-lxd
It almost works.
The problem is that there are missing devices in /dev, which is empty.
The lxd documentation is not clear. It says it creates a /dev empty tmpfs then it configures several devices, including zero, random, full, etc…
the problem is that the container has nothing in /dev except pts and shm

How come /dev is empty? Isn’t it supposed to be created by lxc?

Another problem I have, is that I tried to add some lines into the container configuration. It works once, and when I stop the container, the configuration is reset. Is it normal?

Here is the configuration I have:

architecture: x86_64
config:
  image.architecture: x86_64
  image.description: openwrt 17.01.4 x86_64 (dhcp) (20180329_21:23)
  image.name: openwrt-17.01.4-x86_64-dhcp-20180329_21:23
  image.os: openwrt
  image.release: 17.01.4
  image.variant: dhcp
  security.privileged: "true"
  volatile.base_image: c88d4b2b56e0241c7f64dbcd995ff0cff2aabe9162360b11199d8b564a8f2ee5
  volatile.eth0.hwaddr: 00:16:3e:af:cd:5c
  volatile.idmap.base: "0"
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

thanks

2

LXD will indeed setup /dev for you, the fact that it’s empty suggests that your image either over-mounted it with another layer of tmpfs/devtmpfs, or somehow cleaned it up at boot time, possibly hoping to re-populate it cleanly, only to fail because it’s inside a container.

3

Thanks for your response.

it seems indeed that there are 2 mounts on /dev :
# mount
tank/lxd/containers/openwrt on / type zfs (rw,relatime,xattr,noacl)
tank/lxd/containers/openwrt on / type zfs (rw,relatime,xattr,noacl)
none on /dev type tmpfs (rw,relatime,size=492k,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
udev on /dev/fuse type devtmpfs (rw,nosuid,relatime,size=4032220k,nr_inodes=1008055,mode=755)
udev on /dev/net/tun type devtmpfs (rw,nosuid,relatime,size=4032220k,nr_inodes=1008055,mode=755)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/lxd type tmpfs (rw,relatime,size=100k,mode=755)
tmpfs on /dev/.lxd-mounts type tmpfs (rw,relatime,size=100k,mode=711)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666,max=1024)
devpts on /dev/ptmx type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666,max=1024)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)

looks like it is /sbin/init that did this mount. For some reason, it can’t polpulate it.
When I do the mknods by hand, and launch /sbin/init from the shell, everything works.

Would you know how I could prevent this mount? or at least umount the second layer?

4

umount -l /dev should let you remove that second layer.

5

I just found out that openwrt doesn’t start with /sbin/init (which is busybox init), but with /etc/preinit.
Now I can’t find where is the lxc.init_cmd configuration documented.
with lxc launch, I have error messages.

error: Unknown configuration key: lxc.init.cmd

error: Unknown configuration key: lxc.init_cmd

how is lxc low level configuration managed from lxd? and where is it documented?

6

On liblxc version starting from 2.1 it’s called lxc.init.cmd and you can set it with:

lxc config set <container-name> raw.lxc lxc.init.cmd=/sbin/init
7

Hello,

I have published my own scripts which builds lxd images from openwrt rootfs tar balls. The images can be used in unprivileged containers which I think is an advantage.

The scripts build a patched version of procd which doesn’t remount /dev and other file systems to allow it to run in unprivileged mode.

Two architectures are supported, x86_64 and i686. Currently 17.04.1 is supported. Snapshot is not fully supported since an older version of procd is used in the created image.

Usage: ./build.sh [-a|--arch <x86_64|i686>] [-v|--version <version>] [-p|--packages <packages>] [-f|--files] [--help]

https://github.com/mikma/lxd-openwrt

1 Like
8

Cool, I’m sure it’ll be quite useful to some of our users!