Hi there,
sorry for the late reply, i have strict filter rules or emails. 
But I got now everything working!!! 
FYI: âopenwrt-02â is privileged, the others are standard container, only explicitly with âsecurity.privileged: falseâ and trace logging.
I have read on github about rootless containers in âruncâ two weeks ago:
I already tried then mounting of proc elsewhere, but it didnât work. Then I though about kernel hacking (âSB_I_USERNS_VISIBLEâ). Before that I tried it once more, after getting everything else as it should be (cgroup hierarchy, etc). And read the trace logs once more carefully. Then I saw that âsysfsâ was causing this time the problem. So I finally tried mounting âsysfsâ also elsewhere, and since then it works like a charme.
Starting and stopping containers, importing images, moving them around and running them UNPRIVILEGED.
I had one of the routers (mipsel), where I ran LXD, up for more than one week handling containers. I had no crash or anything else. The only thing, which needs to be done is to stop the containers with the âforceâ flag. I have not investigated why until now. The container start logs (trace level) look all clean as should be.
OpenWRT images can be created easily, like normal system images, as they include ârootfsâ already. I used the âmetadata.yamlâ from debian, changed it a little bit and use, and use two simple templates (one for hosts as within debian and one for â/etc/config/systemâ for the hostname).
Without modifications after OpenWRT container start, an âunmountâ (umount -A -f -l -R (-v) /dev) needs to be performed in the container, because â/dev/urandomâ is not available. âubusdâ needs it to start.
I have a patch for procd, solving this. In normal operation âprocdâ performs in the process step âearlyâ also so called âearly mountsâ, mounting e.g. â/procâ, â/sysâ, â/devâ, /dev/pts", etc) needed for start. In a following step âcoldplugâ â/devâ and â/dev/ptsâ get unmounted, and fresh tmpsfs and devpts get mounted again.
So I added a check if OpenWRT is starting in LXC environment (/proc/1/environ) and if so, then skip the mounts for /dev and /dev/pts allowing access to the LXD created â/devâ system. And OpenWRT starts in containers without a problem. 
You can do everything else as in normal OpenWRT, with the restrictions of container enviornments. 
I have also created a patch for mounting the configured cgroups v1 in OpenWRT in hierarchies. Normally they are mounted flat (process step âearlyâ in procd / init), performing a hierarchy mount with checks. To get rid of shell scripts necessary for this. LXD start script mounts /proc and /sys under /srv/proc and /srv/sys, LXD itself has /srv/LXD as LXD_DIR.
If you tell me, where I can provide you the necessary patches for LXD, please let me know. I do not want a github account currently and always avoided it until know. I have one google email and shop at amazon and electronic parts at aliexpress, thatâs enough.
Getting LXD to work on mips(el) was easier than to figure out what needs to be done for OpenWRT to be used in container environments (because of the a little bit crud start process of the system).
Debian images also work, but for them you need to activate FPU simulation for mips, and this should be done (I only tried it for fun, altough it works). For LXD to get to work on OpenWRT for arm no modifications are necessary (I run LXD also on an netgear R7800). Memory usage is ok i think, itâs about 8 MB in real memory, an additional base container is about 4 to 5 MB (only checked it with âhtopâ).
MIPSEL platform i am using is a âD-Link 860L B1â and two âubiquiti edge router xâ. On the D-Link I use a btrfs raid 1 as âextrootâ (patches, I created, are necessary for OpenWRT to do so), to have enough space (two small SSDâs in a small usb-case for little bucks, I also use this for the R7800).
The modifications for OpenWRT (patches to the build system, go toolchain integration, with more or less no modifications to the existing system) hopefully i get to submit soon. You know, real life also wants a little bit of me (family, friends, etc).
As with the patches: Please consider I am no real GO hacker and have not done real programming for more than 10 years.
But getting LXD working in OpenWRT is a further step to container everywhere. Next I will try to get âruncâ to work, altough I am a fan of LXD (LXC) and use it on other systems quite extensively.
Have a fun sunday and kind greetings,
Manfred
PS: To get a glimpse of a Makefile for OpenWRT and GO build for a package (LXD Makefile - quite simple if you know them, especially without âpostinstallâ and âpostrmâ scripts, which are special to LXD in this case):
#
# This software is licensed under the Public Domain.
#
include $(TOPDIR)/rules.mk
PKG_NAME:=lxd
PKG_VERSION:=3.0.0
PKG_RELEASE:=1
PKG_MAINTAINER:=xxxl <xxx@gmail.com>
PKG_LICENSE_FILES:=LICENSE
PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL:=https://github.com/lxc/lxd.git
PKG_SOURCE_VERSION:=lxd-3.0.0
PKG_SOURCE_DATE:=2018-05-02
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
PKG_MIRROR_HASH:=285fb38aed9445a7a5759480e26e389a31bbcd1694690a703030b7cb15cab138
PKG_BUILD_DEPENDS:=acl dnsmasq git lxc make pkg-config rsync \
squashfs-tools tar xz \
libseccomp libcap \
lvm2 btrfs-progs
PKG_USE_GO:=1
PKG_GO_IMPORT_PATH:=github.com/lxc/lxd
include $(INCLUDE_DIR)/package.mk
TAR_CMD:=$(HOST_TAR) -C $(PKG_BUILD_DIR) --strip-components 1 $(TAR_OPTIONS)
define Package/lxd/Default
SECTION:=lang
CATEGORY:=Languages
SUBMENU:=Golang
TITLE:=LXD v$(PKG_VERSION) - System container manager
URL:=https://linuxcontainers.org/lxd/
MAINTAINER:=xxx <xxx@gmail.com>
endef
define Package/lxd-server
$(call Package/lxd/Default)
TITLE+= server
DEPENDS=+libacl +acl +lxc +liblxc \
+squashfs-tools-unsquashfs +xz +xz-utils \
+shadow-newuidmap +shadow-newgidmap
endef
define Package/lxd-server/description
System container manager using linux containers
LXD daemon (server) package
NOTE: LXD requires dnsmasq with ipv6 support for dhcp.
dnsmasq-dhcpv6 or dnsmasq-full
endef
define Package/lxd-client
$(call Package/lxd/Default)
TITLE+= client
endef
define Package/lxd-client/description
System container manager using linux containers
LXD client tools
endef
define Build/Compile
$(call copy_go_source)
(cd $(GOPATH)/src/$(PKG_GO_IMPORT_PATH)/ && make)
endef
define Build/InstallDev
echo "LXD InstallDev"
$(call Build/GO/InstallDev/Default,$(1))
endef
#define Build/Install
# # nothing to do
#endef
define Package/lxd-server/install
echo "LXD server"
$(call Package/GO/Install/Default,$(1),lxd lxd-benchmark,none,none)
$(INSTALL_DIR) $(1)/etc/init.d
$(INSTALL_BIN) ./files/lxd.init $(1)/etc/init.d/lxd
endef
define Package/lxd-client/install
echo "LXD client"
$(call Package/GO/Install/Default,$(1),lxc lxd-p2c,none,none)
endef
#### Install / Uninstall scripts
# server postinstall
define Package/lxd-server/postinst
#!/bin/sh
echo "Create symlinks for server binaries..."
ln -sf $(GO_ADDITIONAL_PACKAGES)/bin/lxd /usr/sbin/lxd
ln -sf $(GO_ADDITIONAL_PACKAGES)/bin/lxd-benchmark /usr/bin/lxd-benchmark
echo "Create 'sub{u,g}ids' for 'root' (LXD can run unprivileged containers)..."
echo "root:1000000:1000000000" | tee -a /etc/subuid /etc/subgid
echo "HINT: the '--group' sudo bit allows everyone to talk to LXD in the 'sudo' group"
echo "IMPORTANT: LXD requires dnsmasq with dhcp ipv6 support for lxd-bridge"
echo " ('dnsmasq-dhcpv6' or 'dnsmasq-full')"
endef
# client postinstall
define Package/lxd-client/postinst
#!/bin/sh
echo "Create symlinks for user binaries..."
ln -sf $(GO_ADDITIONAL_PACKAGES)/bin/lxc /usr/bin/lxc
ln -sf $(GO_ADDITIONAL_PACKAGES)/bin/lxd-p2c /usr/bin/lxd-p2c
endef
# server postrm
define Package/lxd-server/postrm
#!/bin/sh
echo "Remove symlinks for server binaries..."
rm -f /usr/sbin/lxd
rm -f /usr/bin/lxd-benchmark
echo "IMPORTANT: 'sub{u,g}ids' are not removed for 'root' (can be done manually)"
endef
# client postrm
define Package/lxd-client/postrm
#!/bin/sh
echo "Remove symlinks for client binaries..."
rm -f /usr/bin/lxc
rm -f /usr/bin/lxd-p2c
endef
$(eval $(call BuildPackage,lxd-server))
$(eval $(call BuildPackage,lxd-client))
âŠ
