Need help with proper usage of lxc.id_map

LXC
1

Context:

  • I’m working on a somewhat dated embedded device (based on yocto 2.0 - jethro) and we recently added LXC version 1.1.4 via meta-virtualization layer

  • We currently use systemd-nspawn and we’re now attempting a migration to LXC, so it’d be great to achieve a similar way of working then the one we have today with systemd-nspawn for containerized apps (in terms of networking, etc)

  • The way we work today with containerized apps is that processes spawned with systemd-nspawn will:

      1. Have the network namespace shared between host and guest OS

        • I’m able to achieve this with LXC using lxc.network.type = none

      1. Be owned by a user with UID != 0 (non-root) and we have an iptables rule where traffic with UID != 0 will be redirected to one specific port where we have an ACL, etc…

        • This is where I’m having a hard time right now and would appreciate some clarification!

Question(s):

  • I’ve made progress already, but I still need help fine-tuning my container config files:

    • Should I specify lxc.id_map in container creation time or execution time? (I’ve tried both and each one has it’s issues)

    • Maybe I should substitute lxc.mount.entry by a different option that would make things persist after container creation?

My tests so far:

Test 1

  • Description: Specifying lxc.id_map options in creation time
  • Conclusion: rootfs is created with UID=1015 BUT I have permission issues in execution time
root@mydevice:~# cat conf_A.conf 
lxc.mount.entry = /etc/mydevice etc/mydevice none ro,bind,create=dir 0 0       
lxc.mount.entry = /dev/p4_time dev/p4_time none ro,bind,create=file 0 0
lxc.utsname = alpha
lxc.network.type = none
lxc.id_map = u 0 1015 1
lxc.id_map = g 0 1015 1
root@mydevice:~# cat /etc/subuid 
root:1015:1     
root@mydevice:~# cat /etc/subgid 
root:1015:1
root@mydevice:~# lxc-create -n foo -f conf_A.conf -t n2busybox
This is path: /var/lib/lxc/foo
This is rootfs: /var/lib/lxc/foo/rootfs
This is SSH:
This is in_userns: 1
root@mydevice:~# lxc-ls
foo
root@mydevice:~# ls -al /var/lib/lxc/foo/ 
drwxrwx---    3 1015     1015          4096 Jan  1 05:07 .
drwxr-xr-x    3 root     root          4096 Jan  1 05:07 ..    
-rw-r--r--    1 root     root          1529 Jan  1 05:07 config
drwxr-xr-x   15 1015     1015          4096 Jan  1 05:07 rootfs
root@mydevice:~# lxc-execute -n foo -- sh
lxc-execute: utils.c: safe_mount: 1642 Operation not permitted - Failed to mount sysfs onto /usr/lib/lxc/rootfs/sys
lxc-execute: conf.c: lxc_mount_auto_mounts: 828 Operation not permitted - error mounting sysfs on /usr/lib/lxc/rootfs/sys flags 14
lxc-execute: conf.c: lxc_setup: 3736 failed to setup the automatic mounts for 'foo'
lxc-execute: start.c: do_start: 702 failed to setup the container
lxc-execute: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-execute: start.c: __lxc_start: 1172 failed to spawn 'foo'

Test 2

  • Description: Specifying lxc.id_map options in execution time
  • Conclusion: rootfs is created with UID=0 and I can execute, BUT some lxc.mount.entry options are not mounted due to permission problems!
root@mydevice:~# cat conf_A.conf 
lxc.mount.entry = /etc/mydevice etc/mydevice none ro,bind,create=dir 0 0
lxc.mount.entry = /dev/p4_time dev/p4_time none ro,bind,create=file 0 0
lxc.utsname = alpha
lxc.network.type = none
root@mydevice:~# lxc-create -n bar -f conf_A.conf -t n2busybox
This is path: /var/lib/lxc/bar
This is rootfs: /var/lib/lxc/bar/rootfs
This is SSH:
This is in_userns: 0
root@mydevice:~# lxc-ls
bar  foo
root@mydevice:~# ls -al /var/lib/lxc/bar/
drwxrwx---    3 root     root          4096 Jan  1 05:13 .
drwxr-xr-x    4 root     root          4096 Jan  1 05:13 ..
-rw-r--r--    1 root     root           946 Jan  1 05:13 config
drwxr-xr-x   15 root     root          4096 Jan  1 05:13 rootfs
root@mydevice:~# cat conf_B.conf 
lxc.id_map = u 0 1015 1
lxc.id_map = g 0 1015 1
root@mydevice:~# lxc-execute -n bar -f conf_B.conf -- sh
init.lxc: log.c: log_open: 180 failed to open log file "/var/log/lxc/bar.log" : Permission denied
init.lxc: log.c: log_open: 180 failed to open log file "/var/lib/lxc/bar/bar.log" : Permission denied
init.lxc: log.c: log_open: 180 failed to open log file "/var/log/lxc/bar.log" : Permission denied
init.lxc: initutils.c: mount_fs: 36 failed to mount /dev/mqueue : No such device
sh-3.2# ls -al /dev/p4_time
ls: /dev/p4_time: No such file or directory
sh-3.2# ls -al /etc/mydevice/
drwxr-xr-x    2 nobody   nogroup       4096 Jan  1 00:03 .
drwxrwxr-x   35 nobody   nogroup       4096 Jan  1 00:11 ..
-rw-r--r--    1 nobody   nogroup       1559 Mar 15  2024 test.conf
  • Follow up on Test2: trying to specify the lxc.mount.entry option of /dev/p4_time in execution time
root@mydevice:~# cat conf_B.conf 
lxc.mount.entry = /dev/p4_time dev/p4_time none ro,bind,create=file 0 0
lxc.id_map = u 0 1015 1
lxc.id_map = g 0 1015 1
root@mydevice:~# lxc-execute -n bar -f conf_B.conf -- sh
lxc-execute: utils.c: mkdir_p: 248 Permission denied - failed to create directory 'dev'
lxc-execute: conf.c: lxc_setup: 3746 failed to setup the mount entries for 'bar'
lxc-execute: start.c: do_start: 702 failed to setup the container
lxc-execute: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-execute: start.c: __lxc_start: 1172 failed to spawn 'bar'