Hi,
I (and others) have asked related questions before, but got no answers 
lxc-device seems to be the way to add devices to a running container, but it’s a bit clunky if you want to always have that device available. Is there a way to do this in the config file? I tried with a hook, to run a shell script, but it doesn’t seem to work properly.
Thanks,
Richard
For what it’s worth, I’m using Debian bullseye for both host and guest, with debian-packaged LXC (no LXD).
Kernel is 5.10 provided by my VPS provider (on Xen infrastructure).
Package versions:
lxc, liblxc1 1:4.0.6-2+deb11u1
lxc-templates 3.0.4-5
lxcfs 4.0.7-1
python3-lxc 1:3.0.4-1+b4
Also, when I say the hook/shell script doesn’t work, that’s not strictly true - the device gets created, and is usable, but it reports that it (or something) didn’t work.
Cheers,
Richard
More concretely:
I’m trying to use bindfs to mount a website root under home directories of human users, so they can update the site with all the ownerships and permissions being correct.
To do that, I need fuse, which requires the fuse device in the container.
Currently, the only way I’ve worked out to do that (automatically) is via a hook in the config file which runs a shell script (on the host). Since that apparently runs too early, I’ve split it in two such that the first one runs an ‘at’ job to run the second.
Container config:
lxc.hook.start-host = /usr/local/bin/fuse.hook
/usr/local/bin/fuse.hook:
#!/bin/bash
at now + 1 minute <<END 2>>/var/lib/lxc/rh-wptest2/hook.error
/usr/local/bin/fuse.hook.s2
END
/usr/local/bin/fuse.hook.s2:
lxc-device -n ${LXC_NAME} add /dev/fuse
lxc-attach -n ${LXC_NAME} /usr/local/bin/bindfs_mount
(in container) /usr/local/bin/bindfs_mount:
#!/bin/bash
file='/usr/local/etc/bindfs_mounts'
while read line; do
mount "${line}"
done < "${file}"
(in container) /usr/local/etc/bindfs_mounts:
/home/richard/testsite/doc_root
(in container) /etc/fstab:
/srv/testsite/doc_root /home/richard/testsite/doc_root fuse.bindfs noauto,--force-user=richard,--force-group=richard,--create-for-user=testsite,--create-for-group=testsite 0 0
This seems to work, but does produce error messages from the ‘at’ job:
lxc-device: rh-wptest2: commands.c: lxc_cmd_add_bpf_device_cgroup: 1185 Message too long - Failed to add new bpf device cgroup rule
lxc-device: rh-wptest2: lxccontainer.c: add_remove_device_node: 4657 set_cgroup_item failed while adding the device node
lxc-device: rh-wptest2: tools/lxc_device.c: main: 153 Failed to add /dev/fuse to rh-wptest2
Those messages are shown if I comment out the contents of fuse.hook.s2 and run the commands manually as well. Previously (several months ago) I thought they weren’t, but either I was wrong, or something has changed (upgrades?) in the meantime.
I would like to get rid of the hook and at job mess; it seems there must be a cleaner way to add a device.
I would also like to not see the error messages - and/or ideally, understand what they mean.
Versions etc:
richard@akl-host6:~$ sudo lxc-start --version
4.0.6
richard@akl-host6:~$ sudo lxc-checkconfig
LXC version 4.0.6
— Namespaces —
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
— Control groups —
Cgroups: enabled
Cgroup v1 mount points:
Cgroup v2 mount points:
/sys/fs/cgroup
Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup namespace: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled
— Misc —
Veth pair device: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
Macvlan: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
Vlan: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
Bridges: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
Advanced netfilter: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
FUSE (for use with lxcfs): enabledlibkmod: ERROR …/libkmod/libkmod-module.c:1668 kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
Error: could not get list of modules: No such file or directory
, not loaded
— Checkpoint/Restore —
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:
Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig
richard@akl-host6:~$ uname -a
Linux akl-host6.backend.net.nz 5.10.0-really170-rh178-20230226002031.xenU.x86_64 #1 SMP Sun Feb 26 00:21:44 UTC 2023 x86_64 GNU/Linux
richard@akl-host6:~$ sudo cat /proc/self/cgroup
0::/user.slice/user-1001.slice/session-6.scope
richard@akl-host6:~$ cat /proc/1/mounts ## mounts for other containers redacted
/dev/root / ext4 rw,relatime 0 0
devtmpfs /dev devtmpfs rw,relatime,size=2186448k,nr_inodes=546612,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,size=875780k,nr_inodes=819200,mode=755 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/mapper/vgssd0-LD1rh–wptest2 /guestfs/rh-wptest2-srv xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
/dev/mapper/vgssd0-LRrh–wptest2 /var/lib/lxc/rh-wptest2/rootfs xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
tmpfs /run/user/1001 tmpfs rw,nosuid,nodev,relatime,size=437888k,nr_inodes=109472,mode=700,uid=1001,gid=1001 0 0
Note this is all running under Xen in a commercial VPS, which probably explains the lack of /proc information.
Container config file:
# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template: --release=bullseye
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
lxc.net.0.type = veth
lxc.net.0.hwaddr = 00:16:3e:8a:2f:c0
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.apparmor.profile = generated
# lxc.apparmor.profile = unconfined
lxc.apparmor.allow_nesting = 1
lxc.rootfs.path = dir:/var/lib/lxc/rh-wptest2/rootfs
lxc.mount.entry = /guestfs/rh-wptest2-srv /var/lib/lxc/rh-wptest2/rootfs/srv/ none bind 0 0
# lxc.cgroup.devices.allow = c 10:299 rw
# lxc.cgroup2.devices.allow = c 10:299 rw
# lxc.cgroup.devices.allow = a
# lxc.cgroup2.devices.allow = a
# lxc.hook.autodev = sh -c "mknod ${LXC_ROOTFS_MOUNT}/dev/fuse c 10 299"
lxc.hook.start-host = /usr/local/bin/fuse.hook
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
# Container specific configuration
lxc.tty.max = 4
lxc.uts.name = rh-wptest2
lxc.arch = amd64
lxc.pty.max = 1024
lxc.start.auto = 1
I guess another useful question: Is this situation ok to continue with? Are there underlying problems I’m not seeing, or will it just keep working despite looking ugly?
Cheers,
Richard
Extract from your logs:
lxc-start rh-wptest2 20230322105600.980 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_list_add_device:503 - Reusing existing rule of bpf device program: type c, major 10, minor 229, access rwm, allow 1, global_rule -1
lxc-start rh-wptest2 20230322105600.980 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_finalize:313 - Implementing allowlist bpf device cgroup program
lxc-start rh-wptest2 20230322105600.980 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:351 - Loaded bpf program: (null)
lxc-start rh-wptest2 20230322105600.980 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_cgroup_attach:407 - Loaded and attached bpf program to cgroup /sys/fs/cgroup//lxc.payload.rh-wptest2
lxc-start rh-wptest2 20230322105600.980 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_cgroup_detach:442 - Detached bpf program from cgroup /sys/fs/cgroup//lxc.payload.rh-wptest2
lxc-start rh-wptest2 20230322105600.980 TRACE commands - commands.c:lxc_cmd_fd_cleanup:1655 - Closing client fd 13 for command "add_bpf_device_cgroup"
lxc-start rh-wptest2 20230322105600.984 TRACE commands - commands.c:lxc_cmd_accept:1743 - Accepted new client as fd 7 on command server fd 6
It means that /dev/fuse was already allowed at the time when you’ve called lxc-device -n ... add /dev/fuse.
As you can see on the early stage of container boot:
lxc-start rh-wptest2 20230322105547.913 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 10:229 rwm"`
/dev/fuse is in the whitelist already
Interesting, thanks. Do you know if something has changed (in Debian bullseye) recently, to add this? I’m sure it didn’t work before, but now I can just run mknod in the container, and the device works. So I can get rid of the hook, and just create a systemd unit or something to create the device.
Cheers,
Richard
So ‘just create a systemd unit’ for a device seems to be harder (to learn about) than I thought. Is that the right approach here? Or is there some other way to create the device node?
I got a reply via the systemd-devel mailing list: Don’t use a systemd device unit; use an entry in tmpfiles.d
I created /etc/tmpfiles.d/fuse.conf with the following contents:
#Type Path Mode User Group Age Argument
c! /dev/fuse 0666 root root - 10:229
… and it all seems to work fine 