LXD - success on OpenWRT (privileged containers) - but problems with unprivileged

lxd

#1

Hi there,

I have done a GO toolchain integration (with package management) for OpenWRT / LEDE in my spare time mainly with the intention to get LXD up and running on any of my routers and using the existing tooling and package management of the build system.

Now everything works fine, the complete cross compile toolchain with support for GO gets compiled with ease and compiled GO sources run so far on mips, mipsel and arm (also e.g. aptly, skydive).

But with LXD I have come down to a last problem.
Privileged containers can be started and are running with no problem so far.
But unprivileged containers always seem to have a permission problem with the mounting of “proc”.

lxc 20180507004945.947 ERROR lxc_utils - utils.c:safe_mount:1707 - Operation not permitted - Failed to mount proc onto /usr/lib/lxc/rootfs/proc
_ lxc 20180507004945.947 ERROR lxc_conf - conf.c:lxc_mount_auto_mounts:712 - Operation not permitted - error mounting proc on /usr/lib/lxc/rootfs/proc flags 14_

I am currently using LXD 3.0 and LXC (liblxc 2.1.1) if this is important in some way.

I have read a lot about that, but did not find any solution for it until now.
The only thing what I need to change is “security.privileged” to true and the containers start without a problem.

So i would like to kindly ask, if someone has hints for me what i can try, to get it going. As unprivileged containers on OpenWRT are preferable. I have liblxc compiled with and without “seccomp” but it makes no difference, apparmor, etc. are not used by me or not available.
I have taken a look into the sources of LXC, but do not habe any clue, what I could try.

Maybe someone of you has some tips for me, how to get around this problem. :wink:

Note: I have done the first version of the GO toolchain integration with GO version 1.8.5 (with available patches for mips softfloat) for LEDE 17.01 last november for myself and now ported it to master with availabilty of GO 1.10 (and official support for mips with softfloat). Now I also cleaned up the the needed patches and the build system for packages (as example for Makefile defintion for GO-packages within OpenWRT my already defined packages can be used). Both GO versions work for me also with CGO.

If anyone interested: As soon as I have solved this last issue :smiley: I will provide patches for easy integration of the GO in the toolchain and the package management of OpenWRT in the project forum.
I have also written some other patches, e.g. using BTRFS raid on startup as “extroot” (use small cheap external cases with support for two SSDs). As said: If anyone interested, i will provide the prepared patches for current master in the OpenWRT forum than. Hopefully within the next two days when LXD problem is solved.

Thanks for any help in advance before I take a deeper dive in the sources. :slight_smile:

Kind regards,

Manfred


#2

PS: I have done the necessary setup for uid-mapping. here i will provide the relevant parts of container startup of a successful and an unsuccessful one. Extension “log” is not permitted as attachements, so I cannot provide the complete logs if the startup process.

success (container name ‘success’):
lxc 20180507072828.428 TRACE lxc_conf - conf.c:make_anonymous_mount_file:2256 - Created anonymous mount file
lxc 20180507072828.435 DEBUG lxc_conf - conf.c:mount_entry:1863 - Remounting “/var/lib/lxd/devlxd” on “/usr/lib/lxc/rootfs/dev/lxd” to respect bind or remount options
lxc 20180507072828.436 DEBUG lxc_conf - conf.c:mount_entry:1884 - Flags for “/var/lib/lxd/devlxd” were 4128, required extra flags are 0
lxc 20180507072828.436 DEBUG lxc_conf - conf.c:mount_entry:1894 - Mountflags already were 4096, skipping remount
lxc 20180507072828.437 DEBUG lxc_conf - conf.c:mount_entry:1921 - Mounted “/var/lib/lxd/devlxd” on “/usr/lib/lxc/rootfs/dev/lxd” with filesystem type "none"
lxc 20180507072828.441 DEBUG lxc_conf - conf.c:mount_entry:1863 - Remounting “/var/lib/lxd/shmounts/success” on “/usr/lib/lxc/rootfs/dev/.lxd-mounts” to respect bind or remount options
lxc 20180507072828.441 DEBUG lxc_conf - conf.c:mount_entry:1884 - Flags for “/var/lib/lxd/shmounts/success” were 4128, required extra flags are 0
lxc 20180507072828.442 DEBUG lxc_conf - conf.c:mount_entry:1894 - Mountflags already were 4096, skipping remount
lxc 20180507072828.442 DEBUG lxc_conf - conf.c:mount_entry:1921 - Mounted “/var/lib/lxd/shmounts/success” on “/usr/lib/lxc/rootfs/dev/.lxd-mounts” with filesystem type "none"
lxc 20180507072828.442 INFO lxc_conf - conf.c:mount_file_entries:2212 - Set up mount entries
lxc 20180507072828.444 INFO lxc_conf - conf.c:lxc_fill_autodev:1224 - Populating "/dev"
lxc 20180507072828.444 DEBUG lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/lxc/rootfs/dev/null"
lxc 20180507072828.445 DEBUG lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/lxc/rootfs/dev/zero"
lxc 20180507072828.446 DEBUG lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/lxc/rootfs/dev/full"
lxc 20180507072828.446 DEBUG lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/lxc/rootfs/dev/urandom"
lxc 20180507072828.447 DEBUG lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/lxc/rootfs/dev/random"
lxc 20180507072828.448 DEBUG lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/lxc/rootfs/dev/tty"
lxc 20180507072828.448 INFO lxc_conf - conf.c:lxc_fill_autodev:1275 - Populated "/dev"
lxc 20180507072828.451 DEBUG lxc_conf - conf.c:lxc_setup_dev_console:1603 - mounted pts device “/dev/pts/1” onto "/usr/lib/lxc/rootfs/dev/console"
lxc 20180507072828.455 INFO lxc_utils - utils.c:lxc_mount_proc_if_needed:1758 - I am 1, /proc/self points to "1"
lxc 20180507072828.800 DEBUG lxc_conf - conf.c:setup_rootfs_pivot_root:1127 - pivot_root syscall to ‘/usr/lib/lxc/rootfs’ successful
lxc 20180507072828.800 DEBUG lxc_conf - conf.c:setup_pivot_root:1436 - finished pivot root
lxc 20180507072828.813 DEBUG lxc_conf - conf.c:lxc_setup_devpts:1481 - mount new devpts instance with options "newinstance,ptmxmode=0666,mode=0620,gid=5,max=1024"
lxc 20180507072828.814 DEBUG lxc_conf - conf.c:lxc_setup_devpts:1501 - created dummy “/dev/ptmx” file as bind mount target
lxc 20180507072828.815 DEBUG lxc_conf - conf.c:lxc_setup_devpts:1506 - bind mounted “/dev/pts/ptmx” to "/dev/ptmx"
lxc 20180507072828.815 INFO lxc_conf - conf.c:lxc_setup_ttys:930 - Finished setting up 0 /dev/tty device(s)
lxc 20180507072828.816 INFO lxc_conf - conf.c:setup_personality:1542 - set personality to '0x8’
lxc 20180507072828.817 DEBUG lxc_conf - conf.c:setup_caps:2368 - drop capability ‘sys_time’ (25)
lxc 20180507072828.817 DEBUG lxc_conf - conf.c:setup_caps:2368 - drop capability ‘sys_module’ (16)
lxc 20180507072828.818 DEBUG lxc_conf - conf.c:setup_caps:2368 - drop capability ‘sys_rawio’ (17)
lxc 20180507072828.818 DEBUG lxc_conf - conf.c:setup_caps:2368 - drop capability ‘mac_admin’ (33)
lxc 20180507072828.818 DEBUG lxc_conf - conf.c:setup_caps:2368 - drop capability ‘mac_override’ (32)
lxc 20180507072828.820 DEBUG lxc_conf - conf.c:setup_caps:2377 - capabilities have been setup
lxc 20180507072828.821 NOTICE lxc_conf - conf.c:lxc_setup:3277 - Container “success” is set up
lxc 20180507072828.823 TRACE lxc_start - start.c:lxc_spawn:1348 - Set up cgroup device limits
lxc 20180507072828.824 NOTICE lxc_start - start.c:start:1532 - Exec’ing “/sbin/init”.

failed (container name ‘test-container’):
lxc 20180507073338.160 DEBUG lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary “/usr/bin/newuidmap” does have the setuid bit set.
lxc 20180507073338.160 DEBUG lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary “/usr/bin/newgidmap” does have the setuid bit set.
lxc 20180507073338.160 DEBUG lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
lxc 20180507073338.165 TRACE lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 30637 0 1000000 1000000000 1000000000 0 1"
lxc 20180507073338.171 TRACE lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 30637 0 1000000 1000000000 1000000000 0 1"
lxc 20180507073338.172 TRACE lxc_conf - conf.c:run_userns_fn:3576 - calling function "chown_cgroup_wrapper"
lxc 20180507073338.210 DEBUG lxc_network - network.c:lxc_network_move_created_netdev_priv:2445 - Moved network device “vethCKFHN2”/“eth0” to network namespace of 30626
lxc 20180507073338.210 NOTICE lxc_utils - utils.c:lxc_switch_uid_gid:2073 - Switched to gid 0.
lxc 20180507073338.210 NOTICE lxc_utils - utils.c:lxc_switch_uid_gid:2079 - Switched to uid 0.
lxc 20180507073338.210 NOTICE lxc_utils - utils.c:lxc_setgroups:2091 - Dropped additional groups.
lxc 20180507073338.211 INFO lxc_start - start.c:do_start:925 - Unshared CLONE_NEWCGROUP.
lxc 20180507073338.211 DEBUG storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "dir"
lxc 20180507073338.211 TRACE dir - storage/dir.c:dir_mount:184 - Mounted “/var/lib/lxd/containers/failed-test-container/rootfs” on "/usr/lib/lxc/rootfs"
lxc 20180507073338.211 DEBUG lxc_conf - conf.c:lxc_setup_rootfs:1319 - Mounted rootfs “dir:/var/lib/lxd/containers/failed-test-container/rootfs” onto “/usr/lib/lxc/rootfs” with options “(null)”.
lxc 20180507073338.211 INFO lxc_conf - conf.c:setup_utsname:768 - ‘failed-test-container’ hostname has been setup
lxc 20180507073338.250 DEBUG lxc_network - network.c:setup_hw_addr:2711 - Mac address “00:16:3e:b0:ea:92” on “eth0” has been setup
lxc 20180507073338.252 DEBUG lxc_network - network.c:lxc_setup_netdev_in_child_namespaces:2969 - Network device “eth0” has been setup
lxc 20180507073338.252 INFO lxc_network - network.c:lxc_setup_network_in_child_namespaces:2997 - network has been setup
lxc 20180507073338.252 TRACE lxc_network - network.c:lxc_network_send_name_and_ifindex_to_parent:3078 - Sent network device names and ifindeces to parent
lxc 20180507073338.252 INFO lxc_conf - conf.c:mount_autodev:1149 - Preparing "/dev"
lxc 20180507073338.254 INFO lxc_conf - conf.c:mount_autodev:1171 - Mounted tmpfs on "/usr/lib/lxc/rootfs/dev"
lxc 20180507073338.275 INFO lxc_conf - conf.c:mount_autodev:1188 - Prepared "/dev"
lxc 20180507073338.276 ERROR lxc_utils - utils.c:safe_mount:1707 - Operation not permitted - Failed to mount proc onto /usr/lib/lxc/rootfs/proc
lxc 20180507073338.276 ERROR lxc_conf - conf.c:lxc_mount_auto_mounts:712 - Operation not permitted - error mounting proc on /usr/lib/lxc/rootfs/proc flags 14
lxc 20180507073338.276 ERROR lxc_conf - conf.c:lxc_setup:3178 - failed to setup the automatic mounts for 'failed-test-container’
lxc 20180507073338.276 ERROR lxc_start - start.c:do_start:944 - Failed to setup container “failed-test-container”.
lxc 20180507073338.276 ERROR lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
lxc 20180507073338.276 INFO lxc_network - network.c:lxc_delete_network_priv:2529 - Interface “eth0” with index 102 already deleted or existing in different network namespace
lxc 20180507073338.276 INFO lxc_network - network.c:lxc_delete_network_priv:2539 - Removed interface “eth0” with index 102
lxc 20180507073338.410 INFO lxc_network - network.c:lxc_delete_network_priv:2560 - Removed interface “vethJTUYH6” from "lxdbr0"
lxc 20180507073338.410 DEBUG lxc_network - network.c:lxc_delete_network:3124 - Deleted network devices


(Stéphane Graber) #3

Failure to mount proc is usually a sign of the kernel’s overmounting protection preventing the mount.

Either find what’s hiding part of your host’s /proc and unmount it, or mount a new fresh copy of proc somewhere else (/mnt/proc for example), then try again.


#4

It works indeed but /proc and /sys are not uid/gid shifted. They are ‘nobody:nobody’. Is it intended to be like this? I mean they are 0:0 indeed but i can’t remember that LXD (on another system) did this. I’ve tested this with LXC on OpenWRT.

mount -t proc none /mnt/proc
mount -t sysfs none /mnt/sys

(Stéphane Graber) #5

That’s normal, yes, the kernel doesn’t really keep track of container ownership for those filesystems and it’s usually perfectly fine as the owner isn’t actually used in access checks for procfs.

For sysfs it’s a bit different and the lack of ownership tracking by the kernel is something that @tyhicks on our team is looking at (it’s especially an issue for tweaking some network interface settings through /sys).


#6

Hi there,

sorry for the late reply, i have strict filter rules or emails. :slight_smile:
But I got now everything working!!! :sunny:

FYI: “openwrt-02” is privileged, the others are standard container, only explicitly with “security.privileged: false” and trace logging.

I have read on github about rootless containers in “runc” two weeks ago:

I already tried then mounting of proc elsewhere, but it didn’t work. Then I though about kernel hacking (“SB_I_USERNS_VISIBLE”). Before that I tried it once more, after getting everything else as it should be (cgroup hierarchy, etc). And read the trace logs once more carefully. Then I saw that “sysfs” was causing this time the problem. So I finally tried mounting “sysfs” also elsewhere, and since then it works like a charme.
Starting and stopping containers, importing images, moving them around and running them UNPRIVILEGED. :slight_smile:

I had one of the routers (mipsel), where I ran LXD, up for more than one week handling containers. I had no crash or anything else. The only thing, which needs to be done is to stop the containers with the “force” flag. I have not investigated why until now. The container start logs (trace level) look all clean as should be.

OpenWRT images can be created easily, like normal system images, as they include “rootfs” already. I used the “metadata.yaml” from debian, changed it a little bit and use, and use two simple templates (one for hosts as within debian and one for “/etc/config/system” for the hostname).

Without modifications after OpenWRT container start, an “unmount” (umount -A -f -l -R (-v) /dev) needs to be performed in the container, because “/dev/urandom” is not available. “ubusd” needs it to start.
I have a patch for procd, solving this. In normal operation “procd” performs in the process step “early” also so called “early mounts”, mounting e.g. “/proc”, “/sys”, “/dev”, /dev/pts", etc) needed for start. In a following step “coldplug” “/dev” and “/dev/pts” get unmounted, and fresh tmpsfs and devpts get mounted again.
So I added a check if OpenWRT is starting in LXC environment (/proc/1/environ) and if so, then skip the mounts for /dev and /dev/pts allowing access to the LXD created “/dev” system. And OpenWRT starts in containers without a problem. :smiley: You can do everything else as in normal OpenWRT, with the restrictions of container enviornments. :wink:

I have also created a patch for mounting the configured cgroups v1 in OpenWRT in hierarchies. Normally they are mounted flat (process step “early” in procd / init), performing a hierarchy mount with checks. To get rid of shell scripts necessary for this. :slight_smile: LXD start script mounts /proc and /sys under /srv/proc and /srv/sys, LXD itself has /srv/LXD as LXD_DIR.

If you tell me, where I can provide you the necessary patches for LXD, please let me know. I do not want a github account currently and always avoided it until know. I have one google email and shop at amazon and electronic parts at aliexpress, that’s enough. :slight_smile:

Getting LXD to work on mips(el) was easier than to figure out what needs to be done for OpenWRT to be used in container environments (because of the a little bit crud start process of the system).

Debian images also work, but for them you need to activate FPU simulation for mips, and this should be done (I only tried it for fun, altough it works). For LXD to get to work on OpenWRT for arm no modifications are necessary (I run LXD also on an netgear R7800). Memory usage is ok i think, it’s about 8 MB in real memory, an additional base container is about 4 to 5 MB (only checked it with “htop”).
MIPSEL platform i am using is a “D-Link 860L B1” and two “ubiquiti edge router x”. On the D-Link I use a btrfs raid 1 as “extroot” (patches, I created, are necessary for OpenWRT to do so), to have enough space (two small SSD’s in a small usb-case for little bucks, I also use this for the R7800).

The modifications for OpenWRT (patches to the build system, go toolchain integration, with more or less no modifications to the existing system) hopefully i get to submit soon. You know, real life also wants a little bit of me (family, friends, etc).

As with the patches: Please consider I am no real GO hacker and have not done real programming for more than 10 years. :wink:

But getting LXD working in OpenWRT is a further step to container everywhere. :smiley: Next I will try to get “runc” to work, altough I am a fan of LXD (LXC) and use it on other systems quite extensively. :wink:

Have a fun sunday and kind greetings,
Manfred

PS: To get a glimpse of a Makefile for OpenWRT and GO build for a package (LXD Makefile - quite simple if you know them, especially without “postinstall” and “postrm” scripts, which are special to LXD in this case):

#                                                                                                                                                                                                       
# This software is licensed under the Public Domain.
#                          
                                                                           
include $(TOPDIR)/rules.mk                                   
                                                           
PKG_NAME:=lxd                                                
PKG_VERSION:=3.0.0 
PKG_RELEASE:=1                   
                         
PKG_MAINTAINER:=xxxl <xxx@gmail.com>            
PKG_LICENSE_FILES:=LICENSE 
         
PKG_SOURCE_PROTO:=git                        
PKG_SOURCE_URL:=https://github.com/lxc/lxd.git        
PKG_SOURCE_VERSION:=lxd-3.0.0                                            
PKG_SOURCE_DATE:=2018-05-02                                                    
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz             
PKG_MIRROR_HASH:=285fb38aed9445a7a5759480e26e389a31bbcd1694690a703030b7cb15cab138
     
PKG_BUILD_DEPENDS:=acl dnsmasq git lxc make pkg-config rsync \
        squashfs-tools tar xz \   
        libseccomp libcap \
        lvm2 btrfs-progs                   

PKG_USE_GO:=1                   
PKG_GO_IMPORT_PATH:=github.com/lxc/lxd  
                                             
include $(INCLUDE_DIR)/package.mk
TAR_CMD:=$(HOST_TAR) -C $(PKG_BUILD_DIR) --strip-components 1 $(TAR_OPTIONS)
                                       
define Package/lxd/Default             
        SECTION:=lang                                                                   
        CATEGORY:=Languages
        SUBMENU:=Golang
        TITLE:=LXD v$(PKG_VERSION) - System container manager
        URL:=https://linuxcontainers.org/lxd/
        MAINTAINER:=xxx <xxx@gmail.com>
endef

define Package/lxd-server
$(call Package/lxd/Default)
  TITLE+= server
  DEPENDS=+libacl +acl +lxc +liblxc \
          +squashfs-tools-unsquashfs +xz +xz-utils \
          +shadow-newuidmap +shadow-newgidmap
endef

define Package/lxd-server/description
        System container manager using linux containers
        LXD daemon (server) package

        NOTE: LXD requires dnsmasq with ipv6 support for dhcp.
        dnsmasq-dhcpv6 or dnsmasq-full
endef

define Package/lxd-client
$(call Package/lxd/Default)
  TITLE+= client
endef

define Package/lxd-client/description
        System container manager using linux containers
        LXD client tools
endef

define Build/Compile
        $(call copy_go_source)
        (cd $(GOPATH)/src/$(PKG_GO_IMPORT_PATH)/ && make)
endef

define Build/InstallDev
        echo "LXD InstallDev"
        $(call Build/GO/InstallDev/Default,$(1))
endef

#define Build/Install
#       # nothing to do
#endef

define Package/lxd-server/install
        echo "LXD server"
        $(call Package/GO/Install/Default,$(1),lxd lxd-benchmark,none,none)
        $(INSTALL_DIR) $(1)/etc/init.d
        $(INSTALL_BIN) ./files/lxd.init $(1)/etc/init.d/lxd
endef

define Package/lxd-client/install
        echo "LXD client"
        $(call Package/GO/Install/Default,$(1),lxc lxd-p2c,none,none)
endef

#### Install / Uninstall scripts
# server postinstall
define Package/lxd-server/postinst
#!/bin/sh
echo "Create symlinks for server binaries..."
ln -sf $(GO_ADDITIONAL_PACKAGES)/bin/lxd /usr/sbin/lxd
ln -sf $(GO_ADDITIONAL_PACKAGES)/bin/lxd-benchmark /usr/bin/lxd-benchmark
echo "Create 'sub{u,g}ids' for 'root' (LXD can run unprivileged containers)..."
echo "root:1000000:1000000000" | tee -a /etc/subuid /etc/subgid
echo "HINT: the '--group' sudo bit allows everyone to talk to LXD in the 'sudo' group"
echo "IMPORTANT: LXD requires dnsmasq with dhcp ipv6 support for lxd-bridge"
echo "           ('dnsmasq-dhcpv6' or 'dnsmasq-full')"
endef

# client postinstall
define Package/lxd-client/postinst
#!/bin/sh
echo "Create symlinks for user binaries..."
ln -sf $(GO_ADDITIONAL_PACKAGES)/bin/lxc /usr/bin/lxc
ln -sf $(GO_ADDITIONAL_PACKAGES)/bin/lxd-p2c /usr/bin/lxd-p2c
endef

# server postrm
define Package/lxd-server/postrm
#!/bin/sh
echo "Remove symlinks for server binaries..."
rm -f /usr/sbin/lxd
rm -f /usr/bin/lxd-benchmark
echo "IMPORTANT: 'sub{u,g}ids' are not removed for 'root' (can be done manually)"
endef

# client postrm
define Package/lxd-client/postrm
#!/bin/sh
echo "Remove symlinks for client binaries..."
rm -f /usr/bin/lxc
rm -f /usr/bin/lxd-p2c
endef

$(eval $(call BuildPackage,lxd-server))
$(eval $(call BuildPackage,lxd-client))

#7

Sorry for the formatting of the Makefile :frowning_face:
I didn’t check the outcome and though it would be formatted normally. :wink:


(Christian Brauner) #8

Note that we supported this way before any other runtime did. Nvidia is actually using this in HPC environments. We’ve gone so far as to be able to run unpriviliged containers without a mapping for the container root user:


#9

hello,

below i provide a google drive folder with patches for LXD…

but first, maybe this was a missunderstanding… i only mentioned the “rootless” in runc link, because there i first read about the linux kernel restrictions and how to solve this by mounting proc (sysfs) elsewhere…
not more not less… i did not want anyone to offend…

but thanks for the video… :slight_smile:

as technology should be easily adopted in first state, for me it was a priority to get started with LXD on openwrt without modifications of config files… now openwrt containers can be started out of the box with LXD… install LXD (and dependencies are handled by “opkg”), import image, start container…

i have created a google drive folder… there you can find my custom “make package” for lxd on openwrt (this includes also the patches folder - it’s tailored to be compiled within the packaging system of openwrt and a go toolchain integrated - but sure lxd can be cross compiled with the patches applied) and a separate patches only folder… should be accessible by this link (hopefully, as i have not done this before)…
https://drive.google.com/drive/folders/1ztxSteRojulv03KJmkVHEKercKjJ7Gus?usp=sharing

in the patches folder there is also a patch for “boltdb” (and therefore a necessary patch for the lxd makefile), as there is no official support for mips(le)… i created this one already last november, when playing around with skydive… but as said, i do not have a github account and trying still to avoid it… i like of projects provide other means for project participation, as i do not want to be tracked everywhere, etc…
maybe you find the patches useful, for me it works stable without issues on mipsle (d-link 860, ubiquiti edge router x - my mips archer c7 is crrently on another version for development) and arm (r7800)…

ah, ok, there is also a “rsa” patch… but this seems to be related to the “rsa” package in go… key length until 2048 work without a problem and are generated within one minute on mipsle… everything above stucks for ever in (rsa.GenerateKey)… the kernel randomness (or how it is called) is not the problem, as i have started several “haveged” instances and it then was always at maximum (4065 or so, i do not remember the exact value)…
but i did not investigate this further… there is also some code where i write a template for manual key generation, but this is not complete, as it was then no priority to me, since reduced key length 2048 is working…

one thing i noted is, that on container shutdown i need to issue the force flag… but i am not sure if it is related to the openwrt container itself… the trace logs look all clean, start and shutdown…
what i have noted is, when issuing the “stop” command i see nothing in the container log… with the force flag the container starts to shutdown as expected and it is shown in log file without errors…

and please consider, i am no go hacker (haven’t done much with it) and in the last 10 years i have not programmed really much :wink: … so be patient when maybe looking at the patches :slight_smile:

the patches for openwrt (go toolchain, changes in “procd”, etc) i’m currently preparing… little cleanups (log statements) and rebase… as i have done this the last time 3 weeks ago, it should not be a problem… besides that i am compiling different “go openwrt packages” :slight_smile: … e.g. runc works without problems using a openwrt rootfs, although i still prefer lxd :wink: … i hope i can release them in the openwrt / lede forum today evening…
i will also provide a complete archive with the patches included i think…

bye,

manfred


#10

thanks for the help and reply…
below i posted an answer including a link to a google drive folder for patches…

thanks once again and bye,

manfred

ps: to you provide other means for project participation than github?!? if i maybe have something for you in the future…


(Nitin Sharma) #11

Hi manfred,

I am also trying to build and run lxd on openwrt. My platform is based on ARMv7 32 bit processor.

Your post is really helpful, but I dont know how to integrate GO toolchain with openwrt.
My openwrt workspace dont have GOPATH or copy_go_source function exported.
Please help me to to integrate GO toolchain and compile lxd?

Thanks,

Nitin


#12

Any update on this? I’m very interested in the procd patch.


#13

hi there,

i have been quite busy the last weeks, besides LXD, but now i am finished with what i wanted, my last tests regarding “machine learning” even on mipsel concluded today…
besides LXD i got quite a lot of other GO packages for openwrt to compile (also on mips(el)), like terraform, kubernetes, prometheus, influx stack, syncthing, docker (altough i am not a docker fan), and so on… for all of them i have written openwrt package makefiles with necessary patches and so on, as they can easily be compiled by selecting the packages in the configuration (make menuconfig)…

what took me a little longer was the “find3” location service…
therefore i needed “openblas” (with lapack - required fortran), numpy, scipy and scikit… they now all compile on arm and mipsel with openwrt (netgear r7800 - and mt7621 based “D-Link DIR-860L rev B1” and “Ubiquiti Networks EdgeRouter X”)…
an btw LXD containers for testing were really helpful during that time… :slight_smile:

i will upload the repositories (different branches for each feature patch - e.g. separate ones for containers. btrfs, etc) to gitlab tomorrow or even later today… also with the custom repositories for GO packages, python, and some additional custom packages (tfshark from wireshark, openblas)…

after uploading the repos i will post the links…
and after that i will write some short documentation for it :wink:

bye,

manfred

ps: the time is near :smiley:


#14

@mikma
i have created a gitlab account and uploaded the repository.


it’s “openwrt-dev”
i have rebased all my branches on current master / 18.06 version from today morning.
the patches for procd are in the branches with the extension “container”.

normally every branch for itself should be usable by it’s own. although i always combined them when i was building images / packages.

the “golang” branches include the necessary patches to build the go toolchain as part of the normal toolchain. but it need’s to be selected within “Advanced configuration options (for developers)” -> “Toolchain Options” -> “*** Go compiler ***” and enable “Build go”
I have also two versions 1.10 and 1.8 as i started with the go compiler in openwrt with inoffical patches for mips (soft-float) with 1.8. all current packages from my gitlab repo “openwrt_custom_packages_golang” are only tested with the 1.10 version of go.
all other settings should be by default ok. binaries will be installed within “/usr/lib/golang-packages” by default, but could be changed by modifications of the compiler options.

the only package including more than the binary is LXD, which includes modifications (setup) and procd init script. so simple install of the package should be enough. then proceed with “lxd init” and so on… for LXD you need the patches from “container” branches for the version of your choice (18.06 or master).

the “btrfs” branch includes patches for using btrfs on raid as extroot on startup and patches for configuration of subvolumes to mount in “/etc/config/fstab”
example:
config ‘global’
option anon_swap ‘0’
option anon_mount ‘0’
option auto_swap ‘1’
option auto_mount ‘1’
option delay_root ‘5’
option check_fs ‘0’

config ‘mount’
option target ‘/overlay’
option uuid ‘b8c76bbb-e1e3-4b01-9a6a-608077d8c89a’
option fstype ‘btrfs’
option btrfs_raid ‘1’
option options ‘subvol=@r7800_overlay_current’
option enabled ‘1’

config ‘mount’
option target ‘/data’
option uuid ‘694f6730-682f-41b3-9e81-21b86b17ae43’
option fstype ‘btrfs’
option options ‘subvol=@r7800_data_current’
option enabled ‘1’

config ‘mount’
option target ‘/data/lxd_container’
option uuid ‘694f6730-682f-41b3-9e81-21b86b17ae43’
option fstype ‘btrfs’
option options ‘subvol=@lxd_container’
option enabled ‘1’

the “kernel” branch includes new kernel modules as some minor modifications.

the “container” branch includes patches for mouting of “cgroup hierarchy and cgroup v2” as container detection for lxd / lxc.

the “host-variants” brnach includes patches for building of “host packages” in variants, e.g. build a python host package only for python3 or python2. but as it is a simple implementation it could be used as well for other purposes. i was tired of compiling “numpy”, “scipy” or “scikit” always for both python versions on the host, as this always takes some time. :wink:

i thinkt that’s the most important informations. if you use my package feeds, add them as described in the openwrt documentation. IMPORTANT: install at first my feeds than “/scripts/install -p custom… -a” as otherwise i have encountered strange results. after that run the normal feeds package installation.

hopefully it works for you as well as for me. :slight_smile:
at the weekend we have a big birthday party within the family, so i will not be available for most of the time.

bye,

manfred

ps: i hope i have not forgotten something important which is “normal” to me?!? :wink:


#15

PPS: i have forgotten… if you need openwrt images for LXD, you have two options: you can generate an rootfs tarball from the openwrt configuration menu or you use binwalk, etc to extract it from the generated image file.
here is a simple “metadata.yaml” file and the tempaltes i used. it’s normal LXD stuff :slight_smile:
creation_date: 1525762351
expiry_date: 1528354351
properties:
architecture: mipsel
description: openwrt snapshot mipsel (20180516_06:19)
name: openwrt-snapshot-mipsel-default-20180516_06:19
os: openwrt
release: snapshot
serial: “20180516_06:19”
variant: default
templates:
/etc/config/system:
when:
- create
- copy
create_only: false
template: system.tpl
properties: {}
/etc/hosts:
when:
- create
- copy
create_only: false
template: hosts.tpl
properties: {}

hosts.tpl
127.0.1.1 {{ container.name }}
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

system.tpl
config system
option hostname {{ container.name }}
option zonename ‘Europe/Vienna’
option timezone ‘CET-1CEST,M3.5.0,M10.5.0/3’
option ttylogin ‘0’
option log_size ‘64’
option urandom_seed ‘0’


(Christian Brauner) #16

Please note, that I managed to track down the failure to mount /proc. The root cause of this is that the atime flags (MS_NOATIME, MS_RELATIME, …) are locked in user namespaces. By default /proc will be mounted with noatime on openWRT but LXC will mount /proc with default options which means that it will be mounted with relatime. The kernel will refuse this for aforementioned reasons.
LXC now carries a patch upstream that copies the atime flags which you can try and backport to your distro.
As a workaround for now you can set:

lxc config set <container-name> raw.lxc "lxc.mount.entry = proc proc proc rw,remount,nodev,nosuid,noexec,noatime 0 0"

(michael) #17

Hello,

I use LXC 2.1.1 on Openwrt. Without the patch, I have the /proc mount issue pictured in the thread. When using LXC 2.1.1 + the patch, I then have an issue with sysfs:

lxc-start mycontainer 20180829123326.162 INFO lxc_network - network.c:lxc_setup_network_in_child_namespaces:2997 - network has been setup
lxc-start mycontainer 20180829123326.165 INFO lxc_conf - conf.c:mount_autodev:1159 - Preparing “/dev”
lxc-start mycontainer 20180829123326.166 INFO lxc_conf - conf.c:mount_autodev:1181 - Mounted tmpfs on “/usr/lib/lxc/rootfs/dev”
lxc-start mycontainer 20180829123326.166 INFO lxc_conf - conf.c:mount_autodev:1198 - Prepared “/dev”
lxc-start mycontainer 20180829123326.167 INFO lxc_conf - conf.c:lxc_mount_auto_mounts:718 - Mount source or target for /usr/lib/lxc/rootfs/proc/sys/net on /usr/lib/lxc/rootfs/proc/tty doesn’t exist. Skipping.
lxc-start mycontainer 20180829123326.168 ERROR lxc_utils - utils.c:safe_mount:1707 - No such file or directory - Failed to mount /usr/lib/lxc/rootfs/proc/tty onto /usr/lib/lxc/rootfs/proc/sys/net
lxc-start mycontainer 20180829123326.168 INFO lxc_conf - conf.c:lxc_mount_auto_mounts:718 - Mount source or target for /usr/lib/lxc/rootfs/proc/tty on /usr/lib/lxc/rootfs/proc/sys/net doesn’t exist. Skipping.
lxc-start mycontainer 20180829123326.171 ERROR lxc_utils - utils.c:safe_mount:1707 - Operation not permitted - Failed to mount sysfs onto /usr/lib/lxc/rootfs/sys
lxc-start mycontainer 20180829123326.171 ERROR lxc_conf - conf.c:lxc_mount_auto_mounts:722 - Operation not permitted - error mounting sysfs on /usr/lib/lxc/rootfs/sys flags 14

Is there a solution for this issue as well ?

Thanks & regards