Linux 6.1 (LTS) confirmed, but no fix for mknod yet

stgraber · February 17, 2023, 3:13pm

It’s a kernel bug, so not suitable for the LXD bug tracker.

stgraber · February 17, 2023, 3:14pm

@amikhalitsyn is that something you could take a quick look at?

amikhalitsyn · February 17, 2023, 3:15pm

Sure!

tomp · February 17, 2023, 5:04pm

Ah ok, I interpreted @brauner post here Security.syscalls.intercept.mknod still doesn't work - #3 by brauner that it was likely an emulation bug in the intercept code.

amikhalitsyn · February 23, 2023, 3:24pm

@k4my4b couldn’t you show us the following:

cat /proc/1/mountinfo (from inside the container)
lxc config show CTNAME -e
from the container where this problem is reproducible?

I’ve tried to reproduce the issue on my local environment with the Linux kernel 5.19, unprivileged container, idmapped bindmount (from the host) and then overlayfs inside the container with layers on the idmapped mount. And I’ve reproduced another issue. Overlayfs is always R/O because it fails to properly initialize workdir.

Before going into deep details and debugging I just want to understand your case better.

k4my4b · February 23, 2023, 5:03pm

HI @amikhalitsyn I’ve span up a VM to produce this data because I can’t do it on my production machine as it would break everything, I hope it’s still useful.

Expand

328 58 8:2 /var/lib/lxd/storage-pools/default/containers/arch-ct/rootfs / rw,relatime,idmapped shared:247 master:1 - ext4 /dev/sda2 rw
329 328 0:52 / /dev rw,relatime shared:248 - tmpfs none rw,size=492k,mode=755,uid=1000000,gid=1000000,inode64
330 328 0:55 / /proc rw,nosuid,nodev,noexec,relatime shared:266 - proc proc rw
331 328 0:56 / /sys rw,relatime shared:276 - sysfs sysfs rw
332 329 0:5 /fuse /dev/fuse rw,nosuid,relatime shared:249 master:2 - devtmpfs dev rw,size=8169404k,nr_inodes=2042351,mode=755,inode64
333 329 0:5 /net/tun /dev/net/tun rw,nosuid,relatime shared:250 master:2 - devtmpfs dev rw,size=8169404k,nr_inodes=2042351,mode=755,inode64
337 330 0:57 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:267 master:225 - binfmt_misc binfmt_misc rw
338 331 0:22 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:277 master:7 - efivarfs efivarfs rw
339 331 0:33 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:278 master:20 - fusectl fusectl rw
340 331 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:279 master:10 - pstore pstore rw
341 331 0:35 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:280 master:22 - configfs configfs rw
342 331 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:281 master:16 - debugfs debugfs rw
343 331 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:282 master:8 - securityfs securityfs rw
344 331 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:283 master:17 - tracefs tracefs rw
345 329 0:18 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:251 master:15 - mqueue mqueue rw
346 329 0:58 / /dev/.lxc/proc rw,relatime shared:252 - proc proc rw
347 329 0:56 / /dev/.lxc/sys rw,relatime shared:253 - sysfs sys rw
348 329 0:54 / /dev/lxd rw,relatime shared:254 master:296 - tmpfs tmpfs rw,size=100k,mode=755,inode64
349 329 0:53 /arch-ct /dev/.lxd-mounts rw,relatime master:260 - tmpfs tmpfs rw,size=100k,mode=711,inode64
350 331 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:284 - cgroup2 none rw
351 330 0:49 /proc/cpuinfo /proc/cpuinfo rw,nosuid,nodev,relatime shared:268 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
352 330 0:49 /proc/diskstats /proc/diskstats rw,nosuid,nodev,relatime shared:269 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
353 330 0:49 /proc/loadavg /proc/loadavg rw,nosuid,nodev,relatime shared:270 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
354 330 0:49 /proc/meminfo /proc/meminfo rw,nosuid,nodev,relatime shared:271 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
355 330 0:49 /proc/slabinfo /proc/slabinfo rw,nosuid,nodev,relatime shared:272 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
356 330 0:49 /proc/stat /proc/stat rw,nosuid,nodev,relatime shared:273 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
357 330 0:49 /proc/swaps /proc/swaps rw,nosuid,nodev,relatime shared:274 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
358 330 0:49 /proc/uptime /proc/uptime rw,nosuid,nodev,relatime shared:275 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
359 331 0:49 /sys/devices/system/cpu /sys/devices/system/cpu rw,nosuid,nodev,relatime shared:285 master:171 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
360 329 0:5 /full /dev/full rw,nosuid,relatime shared:256 master:2 - devtmpfs dev rw,size=8169404k,nr_inodes=2042351,mode=755,inode64
361 329 0:5 /null /dev/null rw,nosuid,relatime shared:257 master:2 - devtmpfs dev rw,size=8169404k,nr_inodes=2042351,mode=755,inode64
362 329 0:5 /random /dev/random rw,nosuid,relatime shared:258 master:2 - devtmpfs dev rw,size=8169404k,nr_inodes=2042351,mode=755,inode64
363 329 0:5 /tty /dev/tty rw,nosuid,relatime shared:259 master:2 - devtmpfs dev rw,size=8169404k,nr_inodes=2042351,mode=755,inode64
364 329 0:5 /urandom /dev/urandom rw,nosuid,relatime shared:261 master:2 - devtmpfs dev rw,size=8169404k,nr_inodes=2042351,mode=755,inode64
365 329 0:5 /zero /dev/zero rw,nosuid,relatime shared:262 master:2 - devtmpfs dev rw,size=8169404k,nr_inodes=2042351,mode=755,inode64
366 329 0:59 / /dev/pts rw,nosuid,noexec,relatime shared:263 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
367 329 0:59 /ptmx /dev/ptmx rw,nosuid,noexec,relatime shared:264 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
368 329 0:59 /0 /dev/console rw,nosuid,noexec,relatime shared:265 - devpts devpts rw,gid=1000005,mode=620,ptmxmode=666,max=1024
335 330 0:52 /.lxc-boot-id /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec,relatime shared:248 - tmpfs none rw,size=492k,mode=755,uid=1000000,gid=1000000,inode64
59 329 0:60 / /dev/shm rw,nosuid,nodev shared:255 - tmpfs tmpfs rw,uid=1000000,gid=1000000,inode64
60 328 0:61 / /run rw,nosuid,nodev shared:286 - tmpfs tmpfs rw,size=3271120k,nr_inodes=819200,mode=755,uid=1000000,gid=1000000,inode64
62 328 0:62 / /tmp rw,nosuid,nodev shared:287 - tmpfs tmpfs rw,nr_inodes=1048576,uid=1000000,gid=1000000,inode64
388 60 0:63 / /run/credentials/systemd-sysctl.service ro,nosuid,nodev,noexec,relatime shared:288 - ramfs ramfs rw,mode=700
390 60 0:64 / /run/credentials/systemd-sysusers.service ro,nosuid,nodev,noexec,relatime shared:289 - ramfs ramfs rw,mode=700
392 60 0:65 / /run/credentials/systemd-tmpfiles-setup-dev.service ro,nosuid,nodev,noexec,relatime shared:297 - ramfs ramfs rw,mode=700
404 60 0:68 / /run/credentials/systemd-tmpfiles-setup.service ro,nosuid,nodev,noexec,relatime shared:317 - ramfs ramfs rw,mode=700
506 60 0:70 / /run/credentials/systemd-resolved.service ro,nosuid,nodev,noexec,relatime shared:415 - ramfs ramfs rw,mode=700
384 328 0:75 / /var/lib/docker/overlay2/5cf5493fbf44b0fc8787279ce2df57b2382dc7e4b55023a96107174c46515222/merged rw,relatime shared:318 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/LFFDA37UGCHQZV25L4SXNFP7SM:/var/lib/docker/overlay2/l/5YMEJWYW444DAL3IC56TLBJ3TW:/var/lib/docker/overlay2/l/6TM2AI5ZSEEW32LX4GKSZG7YYY:/var/lib/docker/overlay2/l/Y65OXT37KGY4NY5VZO26FT3YBC:/var/lib/docker/overlay2/l/CVED6FNYA7CA3CYJQX4RBZHSNI:/var/lib/docker/overlay2/l/366EVQ3ZPVXAEP2W2FVE5E6U4W:/var/lib/docker/overlay2/l/U7MC7KG52JQBALO7VIL7JCQOYM:/var/lib/docker/overlay2/l/J7GTQWBHM5P5PIMPCNGCCMKKR6:/var/lib/docker/overlay2/l/B77DVCRHTZ5NWD677RRMLRC47X:/var/lib/docker/overlay2/l/ECKKLJ36HSTICS6PJYS6IX235O:/var/lib/docker/overlay2/l/E2TY4KFSKRWHZQGUMQDGUZ2ROF:/var/lib/docker/overlay2/l/NF5VB4V4OHMDKISV6OK7SWTZ4U:/var/lib/docker/overlay2/l/XQPUBGAP7RFMJPDR6SABRBXJ33:/var/lib/docker/overlay2/l/BVIEVOGX5PC362KRLH3UF2MAQN:/var/lib/docker/overlay2/l/IUHI5IXXBMTQMQGPOHFMUXQBB2:/var/lib/docker/overlay2/l/RRTRT576M2GEIUSB35G5AYHRVQ:/var/lib/docker/overlay2/l/OE66L3HOTZGJUDV7EWMKX2BQZE:/var/lib/docker/overlay2/l/EQUYVKX2CE65ZDXNQJ7MV3RWLL:/var/lib/docker/overlay2/l/QJQETBEFD3S5IKYB37AWFNDA6J:/var/lib/docker/overlay2/l/RKQS5PUIRF2B4YE5MVZJGPDRM3:/var/lib/docker/overlay2/l/YCPGIY73SXGMUPZOQ4BRZJIZUE:/var/lib/docker/overlay2/l/JTITBUS2NU3WHBQ46SFUD3V6IP:/var/lib/docker/overlay2/l/7L3DR5O566UTXD52UBVKPA5RTL:/var/lib/docker/overlay2/l/N3CRE6W7PGVDA27ALPG4RFAN7G:/var/lib/docker/overlay2/l/NNELH7Y6SR2U56HDQB3OOFOCSI:/var/lib/docker/overlay2/l/VJFWJUPUDWM7ECCVV4WSBNC5KK:/var/lib/docker/overlay2/l/FDPNJC5VHJI6ASZDINJW26FQ3C:/var/lib/docker/overlay2/l/2NV3H73ABERVZBUNL4NVAKCJR2:/var/lib/docker/overlay2/l/JCRUFPIJFFJ4FB2RYQZO57KXZA:/var/lib/docker/overlay2/l/3WHDDFKLVT5A73F5JPUDBTBHEY:/var/lib/docker/overlay2/l/W6BCTP2MO37T72HGBTLSD2RLBY:/var/lib/docker/overlay2/l/JJ7XWE54BV5SPPQOYSIYBNL247:/var/lib/docker/overlay2/l/53TQU3S47PCJAOZS44J52EGFOW:/var/lib/docker/overlay2/l/TZLDROS4ZUI357A5VJZC2PAMKR:/var/lib/docker/overlay2/l/DJUDUZRIWCUFXOFIESGMHYWEZ5:/var/lib/docker/overlay2/l/TDQ25NTKM4M3LLXBWC2O326U44,upperdir=/var/lib/docker/overlay2/5cf5493fbf44b0fc8787279ce2df57b2382dc7e4b55023a96107174c46515222/diff,workdir=/var/lib/docker/overlay2/5cf5493fbf44b0fc8787279ce2df57b2382dc7e4b55023a96107174c46515222/work,index=off,metacopy=off,userxattr
677 60 0:4 net:[4026532809] /run/docker/netns/d6536d562b23 rw shared:516 - nsfs nsfs rw
622 328 0:86 / /var/lib/docker/overlay2/155f149fef176cea7c8137165311e2bde80f11b731020dd5c551dbc79b389187/merged rw,relatime shared:530 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/LAU5RYWK43PIXBTQ6NTTMLFSS2:/var/lib/docker/overlay2/l/AZG7NILVEH6I4FYAWWC2ZYPOIB:/var/lib/docker/overlay2/l/KULDSPIM346LTC6OPJSDNXYBNJ:/var/lib/docker/overlay2/l/33DCBOALPQSVMH6KLA5FDM3SJM:/var/lib/docker/overlay2/l/SWGR6HSPSMCBSY7MKY5GOX5O42:/var/lib/docker/overlay2/l/XYSCJGP2EFFP5MGXNHA2V2LS56:/var/lib/docker/overlay2/l/SLTQPHKLRY67BUOXPKRRWV4JPE:/var/lib/docker/overlay2/l/YOZBXPIN3I2N5JCWCOQTV7UCSO,upperdir=/var/lib/docker/overlay2/155f149fef176cea7c8137165311e2bde80f11b731020dd5c551dbc79b389187/diff,workdir=/var/lib/docker/overlay2/155f149fef176cea7c8137165311e2bde80f11b731020dd5c551dbc79b389187/work,index=off,metacopy=off,userxattr
737 60 0:4 net:[4026532876] /run/docker/netns/2f8e07387fd2 rw shared:536 - nsfs nsfs rw
688 328 0:97 / /var/lib/docker/overlay2/c40d412533624ac6f08050e746bb0c25e5fe1d667137cf80a3152cf0214eaa86/merged rw,relatime shared:542 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/J4ENUEYPJARGYIVCL3FPXHFDPW:/var/lib/docker/overlay2/l/ZZ4B5MWP5X55XXYPFDAFATDY4O:/var/lib/docker/overlay2/l/K3GU3XYHCHEWLTRDNLVVW3YDCB:/var/lib/docker/overlay2/l/UQIWGMUCRAM2OJQLULJ7Q4HDTA:/var/lib/docker/overlay2/l/SIQ4ROQGOTGE5D47SZKILJOCS7:/var/lib/docker/overlay2/l/OC6BENJXRY3JKYP2BT46VT3RHE:/var/lib/docker/overlay2/l/VXE2QDWKE3V4UL4JZX2MDG662D:/var/lib/docker/overlay2/l/S3TKNH7P63CLSOJCDOFSBSEVBJ:/var/lib/docker/overlay2/l/2GF7P7MTFTG5V5VWBU7QPHOH7T:/var/lib/docker/overlay2/l/P7JRAXUU3JFVFOBMYR3OD5YVK7:/var/lib/docker/overlay2/l/DEUQPSC4DRXHEVB4765BJM34WE:/var/lib/docker/overlay2/l/MEQKQZ233AMA5O47TTL55IEM6N:/var/lib/docker/overlay2/l/6AV5OPOT36FPNNKK3MI4WQTGBK,upperdir=/var/lib/docker/overlay2/c40d412533624ac6f08050e746bb0c25e5fe1d667137cf80a3152cf0214eaa86/diff,workdir=/var/lib/docker/overlay2/c40d412533624ac6f08050e746bb0c25e5fe1d667137cf80a3152cf0214eaa86/work,index=off,metacopy=off,userxattr
796 60 0:4 net:[4026532941] /run/docker/netns/c7d8bb322a30 rw shared:548 - nsfs nsfs rw
756 328 0:108 / /var/lib/docker/overlay2/6b67fda60a346c5cd9d19c4848b8d600e47cd5b6ed91f428834ffea7bed7e740/merged rw,relatime shared:554 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/PSGY5IQTPBJARRK3UJXYYUZN2S:/var/lib/docker/overlay2/l/NTS526WBA6SGSO7C23XVMYKB44:/var/lib/docker/overlay2/l/4H7Y6ED5OHQQVUFIJOV6WAMDY2:/var/lib/docker/overlay2/l/4CBYIOAIB7NAGJ337E4Y4NXYZC:/var/lib/docker/overlay2/l/SIKVBTSL26OAYOSWF6IKEYD5BS:/var/lib/docker/overlay2/l/SBSI6OFPI3V3Y2EHSV4UHP4GE2:/var/lib/docker/overlay2/l/3RSREE7O5CYBDMTB65MAMGBCAP:/var/lib/docker/overlay2/l/AREOOEBSQK4VT3SKO3RJWHVR66:/var/lib/docker/overlay2/l/FRQRTGVPOK2DV7XUF4RPA5PVBQ:/var/lib/docker/overlay2/l/SGZN4QBSRGIU56XLDBCOCYQHLV:/var/lib/docker/overlay2/l/3FO6RBDAVIBU7NFZ7YRLFXXYCY:/var/lib/docker/overlay2/l/NKBP3RP7EKUWRJ2UHFTF5YRYND:/var/lib/docker/overlay2/l/YFZ4XNQTRCZ4FR4FX77ZUE2SUT:/var/lib/docker/overlay2/l/LWQ6J4NO5FVOJRU5QNXDDDGO2P:/var/lib/docker/overlay2/l/SIWMWBG7DDRZGWZMNJTJHBFUHT:/var/lib/docker/overlay2/l/NFXYQ756WEGTDKEAGPDSTIFEKU:/var/lib/docker/overlay2/l/SYLW2GM2DG3WJBLWZD5M6AC3L4:/var/lib/docker/overlay2/l/TDQ25NTKM4M3LLXBWC2O326U44,upperdir=/var/lib/docker/overlay2/6b67fda60a346c5cd9d19c4848b8d600e47cd5b6ed91f428834ffea7bed7e740/diff,workdir=/var/lib/docker/overlay2/6b67fda60a346c5cd9d19c4848b8d600e47cd5b6ed91f428834ffea7bed7e740/work,index=off,metacopy=off,userxattr
862 60 0:4 net:[4026533031] /run/docker/netns/106fb982a8d5 rw shared:560 - nsfs nsfs rw
811 328 0:119 / /var/lib/docker/overlay2/ed25e609c3a5cc69516b309e01cf7db4284984c94121d223301f8c92ea6d14d2/merged rw,relatime shared:566 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/I45KDIN72WYBCDCR5XZZTCFR7P:/var/lib/docker/overlay2/l/DHITS3HAQDK72MKH5IRA5C22UO:/var/lib/docker/overlay2/l/IJ4MO4CX7NRM35WY432KH27LKA:/var/lib/docker/overlay2/l/HKXUTE4VKFQRXTVXQGRLXIAKLT:/var/lib/docker/overlay2/l/TYWM7KFP5TE53NWBNAT33HBZUX:/var/lib/docker/overlay2/l/6NWQBHUX5LZMCHMHHIKHAMSHRC:/var/lib/docker/overlay2/l/KSTXJVGIWRFAGYWWXJHT4Y44VH:/var/lib/docker/overlay2/l/6PUXUCMO3HBRU7Z3NF2NAHDD7P:/var/lib/docker/overlay2/l/5SJYSNQN7IEOFRLETMCZFP6VG6:/var/lib/docker/overlay2/l/WN7TIJGHWLJI2CJKSYLR3TZVWN:/var/lib/docker/overlay2/l/TDQ25NTKM4M3LLXBWC2O326U44,upperdir=/var/lib/docker/overlay2/ed25e609c3a5cc69516b309e01cf7db4284984c94121d223301f8c92ea6d14d2/diff,workdir=/var/lib/docker/overlay2/ed25e609c3a5cc69516b309e01cf7db4284984c94121d223301f8c92ea6d14d2/work,index=off,metacopy=off,userxattr
920 60 0:4 net:[4026533101] /run/docker/netns/f78f340b5762 rw shared:572 - nsfs nsfs rw
841 328 0:130 / /var/lib/docker/overlay2/e63b72b394ab78b338971c726ed30a3652aeec2e8ef38c1ea9214e3f5dcd9cfa/merged rw,relatime shared:578 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/CML67P3ICLEWT7QSELNMJLRXFP:/var/lib/docker/overlay2/l/3527EGT7SG3EQL2SEHZUNJMOWK:/var/lib/docker/overlay2/l/M4IPZSLAJ2M5BOREK4TCY5Y5SP:/var/lib/docker/overlay2/l/SSNL4MKYWVGNINOUIEOON722N4:/var/lib/docker/overlay2/l/VQSL56BV4VHXSB37O4LKWXS6AS:/var/lib/docker/overlay2/l/ZXUNUPPERCJ42FYL5AY2GGCYNN:/var/lib/docker/overlay2/l/3EQNL6RF3O2ZIQPUEQ3KZLPTF5:/var/lib/docker/overlay2/l/SOM4PPWEVWAZP7CNEMAZPRNKTK:/var/lib/docker/overlay2/l/3I3VAVRQIJ2DYEOLC2LTTTUIDC:/var/lib/docker/overlay2/l/TDQ25NTKM4M3LLXBWC2O326U44,upperdir=/var/lib/docker/overlay2/e63b72b394ab78b338971c726ed30a3652aeec2e8ef38c1ea9214e3f5dcd9cfa/diff,workdir=/var/lib/docker/overlay2/e63b72b394ab78b338971c726ed30a3652aeec2e8ef38c1ea9214e3f5dcd9cfa/work,index=off,metacopy=off,userxattr
977 60 0:4 net:[4026533171] /run/docker/netns/7f68f42b9edf rw shared:584 - nsfs nsfs rw
921 328 0:141 / /var/lib/docker/overlay2/23cc9d1954338cc2e00f6bccfe2c50a45a0fd77199fef2b4bdb16110e392b3b7/merged rw,relatime shared:590 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/J5RGP7LLBZRACBUXUWQKRRKB74:/var/lib/docker/overlay2/l/AHXD2LGSW3BHQEAZ5N5T6AR72G:/var/lib/docker/overlay2/l/F7EDGK42VND3ISCMPPX2KYT3YJ:/var/lib/docker/overlay2/l/LL6FZJYOJYFANHV5LV7X6DLQI6:/var/lib/docker/overlay2/l/A2FM2FQ2JF5MW2722KKP5JDQB3:/var/lib/docker/overlay2/l/TOKSEXHEGF36PHGQTOY6M7KO27:/var/lib/docker/overlay2/l/ZIHXL5CMXBECDWYURRUOPB2IFB:/var/lib/docker/overlay2/l/G5EAK76UD2YDKQ6E7F3W3ABMTP:/var/lib/docker/overlay2/l/I5YWUFGOZJOM744FP5ZUSFNS4O:/var/lib/docker/overlay2/l/MLCILROJMMR4RHSBGNRWLEOI53:/var/lib/docker/overlay2/l/4NTJZ4VQV33PO3NRYEKHKSGKAS:/var/lib/docker/overlay2/l/Z7TVH57NWDVI6P5NC3JCG75XVB:/var/lib/docker/overlay2/l/JABUAV7BQ7BABK77XSF5R4ODAM:/var/lib/docker/overlay2/l/UTPNIXWNKIOEBMNSSHHGW2BOJP,upperdir=/var/lib/docker/overlay2/23cc9d1954338cc2e00f6bccfe2c50a45a0fd77199fef2b4bdb16110e392b3b7/diff,workdir=/var/lib/docker/overlay2/23cc9d1954338cc2e00f6bccfe2c50a45a0fd77199fef2b4bdb16110e392b3b7/work,index=off,metacopy=off,userxattr
1038 60 0:4 net:[4026533241] /run/docker/netns/a506718070ca rw shared:596 - nsfs nsfs rw
953 328 0:152 / /var/lib/docker/overlay2/ebd939222f63e68d983b8a51089682d4cb94de57459ea246b9e7d392cc52eb97/merged rw,relatime shared:602 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/YAWDU4ZXUW6VUFI32UQYORNYA7:/var/lib/docker/overlay2/l/6ZLE3JZXQTU5Z5YGAG5K2LE7V4:/var/lib/docker/overlay2/l/EME42YFDL4QKGXNWYYYNTKOMG2:/var/lib/docker/overlay2/l/W4QW37IMBM3KZKAUYK3URYSDAC:/var/lib/docker/overlay2/l/NLDA45X7KFLO4T2OB4UQEWES6Q:/var/lib/docker/overlay2/l/ZOSE7DSOHVRSFIDSVMM4RLYYX3:/var/lib/docker/overlay2/l/XYGTSFP7C3DCIFLAH26TPXQF7L,upperdir=/var/lib/docker/overlay2/ebd939222f63e68d983b8a51089682d4cb94de57459ea246b9e7d392cc52eb97/diff,workdir=/var/lib/docker/overlay2/ebd939222f63e68d983b8a51089682d4cb94de57459ea246b9e7d392cc52eb97/work,index=off,metacopy=off,userxattr
1091 60 0:4 net:[4026533311] /run/docker/netns/a0585fe70222 rw shared:608 - nsfs nsfs rw
1055 328 0:163 / /var/lib/docker/overlay2/384df17978059eb97524f7875388cbd317028485653fd5c3b832af4c9dfa4f31/merged rw,relatime shared:614 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/OACPINYGJSGAAM4OFTHC5OZZ7V:/var/lib/docker/overlay2/l/MUVHZF2QVIL6OE436VATAE5DJT:/var/lib/docker/overlay2/l/OWM65T6KS5GPBBBE33LYAGPEMZ:/var/lib/docker/overlay2/l/OFGFG7XHZ4W3LMK5MRSZAPZPEL:/var/lib/docker/overlay2/l/T5AZ7C6V2TOJRO5NYZGGINXJNR:/var/lib/docker/overlay2/l/JYDMKQAO5HD7NGFAQKG6OH3X3Y:/var/lib/docker/overlay2/l/PX2JY3O25PR7KFWZAINCFYMH2X:/var/lib/docker/overlay2/l/LYACELJMJI5H7B6ZK5HPVVC4AN:/var/lib/docker/overlay2/l/ZJFZOXZTTUIRGP35ETQQK63SX4:/var/lib/docker/overlay2/l/WBZILQGSOPXWNYANWJ4FSFVNGM:/var/lib/docker/overlay2/l/CRG3UMIXTEJJY626TPHH3HTMUO:/var/lib/docker/overlay2/l/57YR3WRPYZUPR4L2RBW66PFBLS:/var/lib/docker/overlay2/l/CJVRNO65RS6K5VPJUUOYRIBGOP:/var/lib/docker/overlay2/l/5P45OLDSJ672PN57Z4GO7PXJTJ:/var/lib/docker/overlay2/l/L4YDLWTQVLNIDLUU5ZIJGVLKTX:/var/lib/docker/overlay2/l/A77BHY4MHQVMPXBDTSIKCDQYT7:/var/lib/docker/overlay2/l/ZYR44Q4QEJB5RKNOW44V3FGZVD:/var/lib/docker/overlay2/l/TNRBUVYGGAOPJ5MXPZQQRMQLYQ:/var/lib/docker/overlay2/l/IPUVTVZIRVREQNJ4D7RQNNQZMR:/var/lib/docker/overlay2/l/QCBXP4MRBVRVQUFNMTCGLVILPD:/var/lib/docker/overlay2/l/ZRJB3KP7RUWDTTPSQ5A6O5IDK2:/var/lib/docker/overlay2/l/IAMLMEMH4LEI3M3W44PNEQ5CWA:/var/lib/docker/overlay2/l/FZHSEZCX45ZXNAJUOWBDQ2UPJB:/var/lib/docker/overlay2/l/XQJAAQZNXRGNUGYSL5WSPSAF7P:/var/lib/docker/overlay2/l/KQMHWR5XYA2IERTCY3ZCVB5Q7V:/var/lib/docker/overlay2/l/YSL55WAAJIF6KDGW7Q4Z27DONN:/var/lib/docker/overlay2/l/5JKHSMAR2DHWR2RL3XH5LCAAUX:/var/lib/docker/overlay2/l/IGRO5FNT3R4RKSTR2SX2UVCS5C:/var/lib/docker/overlay2/l/IZP5TGMQRMMCNCG66XK5PZ7LYJ:/var/lib/docker/overlay2/l/SOLRZUZ3YLUUQDHZHOVMHEFDSS:/var/lib/docker/overlay2/l/23XEU624GS4OCBSLSMHKPMB2ED:/var/lib/docker/overlay2/l/VZK2E77HQFMQMI7A33GJRMPIHQ,upperdir=/var/lib/docker/overlay2/384df17978059eb97524f7875388cbd317028485653fd5c3b832af4c9dfa4f31/diff,workdir=/var/lib/docker/overlay2/384df17978059eb97524f7875388cbd317028485653fd5c3b832af4c9dfa4f31/work,index=off,metacopy=off,userxattr
1171 60 0:4 net:[4026533381] /run/docker/netns/c27501586818 rw shared:620 - nsfs nsfs rw
1121 328 0:174 / /var/lib/docker/overlay2/2d550c436babf94769390d1e8fd48c82dc138e12cbc30891ec95cdd8887d8d4a/merged rw,relatime shared:626 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/ZO3HW3C3ZLYBBPW2KS3HX5VTTK:/var/lib/docker/overlay2/l/FUAJPNURF3DBAIWIGHZIH2AW4N:/var/lib/docker/overlay2/l/IJ2RY2GWGH6QUFPYCHG3OZETP4:/var/lib/docker/overlay2/l/3VGN653HV53BMSFA5OOK5RHICC:/var/lib/docker/overlay2/l/IPASCETATXB2XDUCIOUAMU43HQ:/var/lib/docker/overlay2/l/TS5FGPF6L4YM2UFHD4FFLEBFUB:/var/lib/docker/overlay2/l/QZCRBUPL3THRJ2S44NA6YMJEYD:/var/lib/docker/overlay2/l/2NGJM3XOIEWMEXUY5VQHNL4UV5:/var/lib/docker/overlay2/l/HMYD3AJEVEACZKUYFJ6FYLYJOK:/var/lib/docker/overlay2/l/3GWHWLECVJZJAO4KTFFOFEFOPH:/var/lib/docker/overlay2/l/WGR27OLL4LLLUCB2JVUHHLM5AR:/var/lib/docker/overlay2/l/THSKWCK74TIVCK23JN2HDQX2EJ:/var/lib/docker/overlay2/l/GI3E6UUVYFWNGM74YVPL56BCGH:/var/lib/docker/overlay2/l/Z3C7TNNCSPNIQTI43RSJPW2L4K:/var/lib/docker/overlay2/l/22WYMJGPJGB75VRDSF4KR4C4PL:/var/lib/docker/overlay2/l/4SDO32SPY7CCWUQK3HQ7B5IQ4S:/var/lib/docker/overlay2/l/4QSILMYNPL6J2CG5FBVS6D5U7G:/var/lib/docker/overlay2/l/L4RCDE5GBEH6ZEJ2PYHP5UEAIZ:/var/lib/docker/overlay2/l/LHQOCPTOC4I2PJLVSZJXJBZFFV:/var/lib/docker/overlay2/l/6PL4J2EGATZHXCZDQDSTVNEGTL:/var/lib/docker/overlay2/l/GTUKANBIDCO654XOBPQX3LGKJ2:/var/lib/docker/overlay2/l/TDQ25NTKM4M3LLXBWC2O326U44,upperdir=/var/lib/docker/overlay2/2d550c436babf94769390d1e8fd48c82dc138e12cbc30891ec95cdd8887d8d4a/diff,workdir=/var/lib/docker/overlay2/2d550c436babf94769390d1e8fd48c82dc138e12cbc30891ec95cdd8887d8d4a/work,index=off,metacopy=off,userxattr
1241 60 0:4 net:[4026533451] /run/docker/netns/32372a5fabc3 rw shared:632 - nsfs nsfs rw

Expand

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Archlinux current amd64 (20230223_04:18)
  image.os: Archlinux
  image.release: current
  image.requirements.secureboot: "false"
  image.serial: "20230223_04:18"
  image.type: squashfs
  image.variant: default
  security.nesting: "true"
  security.syscalls.intercept.mknod: "true"
  volatile.base_image: 01df01c32589cc437677247bc68d3dbf89be79b1ac471bad16d7f4e346709d5d
  volatile.cloud-init.instance-id: e7a4026d-1737-4a4d-80af-6ddc889833b1
  volatile.eth0.host_name: veth1f305122
  volatile.eth0.hwaddr: 00:16:3e:6e:07:4b
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: eb7297ea-8782-4c56-974f-0173f28211a2
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

amikhalitsyn · February 23, 2023, 8:17pm

thanks!

it is confirmed that you have an idmapped root in the container with overlayfs on top of the idmapped root. The question is why overlayfs get mounted as R/W for you without any problem. I need to take a look on this closely.

amikhalitsyn · March 2, 2023, 5:41pm

It looks like starting from 5.19 overlayfs on top of idmapped mounts is broken on Ubuntu kernels.
https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.19/+bug/2009065

k4my4b · March 3, 2023, 10:24am

@amikhalitsyn the bug report is regarding rootfs mounting as read-only but in my case, as you said, rootfs is mounted read/write but mknod emulation is still broken.

amikhalitsyn · March 3, 2023, 10:33am

that’s different problems.

Which Ubuntu version you have on your test environment? Which kernel version?
I think we need to try to construct a minimal reproducer without Docker and other stuff.

Couldn’t you try this:

cat /etc/os-release
uname -a
lxc exec your-test-container bash
mount | grep idmap
mkdir {work,upper,lower,ovl}
mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work ovl
mknod mknod /root/ovl/null c 1 3
stat /root/ovl/null
mount | grep null

I’ve managed to reproduce the problem with mknod on recent mainline kernel 6.2 and Ubuntu 22.04.
mknod on overlayfs works, but it works through fallback mechanism (bindmount). So, it’s a problem, but I can’t be fully sure that you have had the same problem.

k4my4b · March 3, 2023, 10:55am

@amikhalitsyn This is all taken from the same old VM I used last time, haven’t touched it since,
it’s an Arch Linux VM though I’ve tried it with fedora and opensuse before and mknod emulation was broken there too.

Expand

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"

Expand

Linux archlinux 6.1.13-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 22 Feb 2023 12:31:32 +0000 x86_64 GNU/Linux

Expand

/dev/sda2 on / type ext4 (rw,relatime,idmapped)

Expand

  File: /root/ovl/null
  Size: 0               Blocks: 0          IO Block: 4096   character special file
Device: 8,2     Inode: 528144      Links: 0     Device type: 1,3
Access: (0666/crw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-03 10:46:25.430722874 +0000
Modify: 2023-03-03 10:46:25.430722874 +0000
Change: 2023-03-03 10:46:25.444057339 +0000
 Birth: 2023-03-03 10:46:25.430722874 +0000

Expand

dev on /dev/null type devtmpfs (rw,nosuid,relatime,size=8169404k,nr_inodes=2042351,mode=755,inode64)
/dev/sda2 on /root/ovl/null type ext4 (rw,relatime)

amikhalitsyn · March 3, 2023, 10:58am

great. The same behavior for me. mknod interception works but through fallback mechanism.

amikhalitsyn · March 3, 2023, 5:56pm

@k4my4b couldn’t you perform the same test as you did before, but on the system where mknod interception works good for you? I mean 5.15 kernel (or 5.17?). As far as I can see this problem with interception is present on 5.15 kernel too.

k4my4b · March 4, 2023, 12:29pm

@amikhalitsyn mount -t overlay fails this time.

Expand

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

Expand

Linux arch-ct 5.15.94-1-lts #1 SMP Wed, 15 Feb 2023 07:09:02 +0000 x86_64 GNU/Linux

Expand

/dev/sda2 on / type ext4 (rw,relatime,idmapped)

Expand

mount: /root/ovl: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

Expand

  File: /root/ovl/null
  Size: 0         Blocks: 0          IO Block: 4096   character special file
Device: 8,2Inode: 1291311     Links: 1     Device type: 1,3
Access: (0644/crw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-04 12:26:07.374439142 +0000
Modify: 2023-03-04 12:26:07.374439142 +0000
Change: 2023-03-04 12:26:07.374439142 +0000
 Birth: 2023-03-04 12:26:07.374439142 +0000

Expand

dev on /dev/null type devtmpfs (rw,nosuid,relatime,size=8170840k,nr_inodes=2042710,mode=755,inode64)

amikhalitsyn · March 6, 2023, 4:03pm

you can check dmesg for errors after mount failed.

k4my4b · March 6, 2023, 4:52pm

@amikhalitsyn

[  164.809290] overlayfs: idmapped layers are currently not supported

Linux archlinux 5.15.94-1-lts #1 SMP Wed, 15 Feb 2023 07:09:02 +0000 x86_64 GNU/Linux

amikhalitsyn · March 6, 2023, 4:59pm

yep, that’s correct behavior for old kernel versions. Probably on the production environment with an old kernel versions you are not using idmapped mounts. But the questions is, how all of this works for you now? You’ve started a topic from the question that something get broken on a newer kernel versions, but AFAIU you already have a working setup on older versions. And what I whan is to understand your setup on this “old” versions, your idmappings setup, how mknod interception works for you and so on. Couldn’t you describe all of this in details? If I have a minimal possible reproducer which works on your old kernel and doesn’t work on a new versions then I’ll be able to fix it, otherwise it may took too much time and forces…

k4my4b · March 6, 2023, 5:34pm

@amikhalitsyn
I’m running the same exact kernel, 5.15.94-1-lts, on my production machine. The only main difference is that on my production machine I’m using BTRFS as opposed to EXT4.
Here’s everything from my production machine:

uname -a

Expand

Linux lxd 5.15.94-1-lts #1 SMP Wed, 15 Feb 2023 07:09:02 +0000 x86_64 GNU/Linux

cat /etc/os-release

Expand

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

cat /proc/cmdline

Expand

 lsm=landlock,lockdown,yama,integrity,apparmor,bpf  root=PARTUUID=cb40f3e4-6d53-4804-af33-ce12c85517a4 rootflags=subvol=@ rootfstype=btrfs rw  ipv6.disable_ipv6=1  intel_pstate=no_hwp  intel_iommu=on iommu=pt  loglevel=3 rd.systemd.show_status=auto rd.udev.log_level=3

cat /etc/sub{uid,gid}

Expand

root:1000000:1000000000
root:1000000:1000000000

cat /etc/sysctl.d/*

Expand

fs.aio-max-nr = 524288
fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576
kernel.dmesg_restrict = 1
kernel.keys.maxbytes = 2000000
kernel.keys.maxkeys = 2000
net.core.netdev_max_backlog = 182757
net.ipv4.neigh.default.gc_thresh3 = 8192
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.neigh.default.gc_thresh3 = 8192
kernel.unprivileged_userns_clone=1
vm.max_map_count = 262144

cat /etc/security/limits.conf

Expand

# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
#Also note that configuration files in /etc/security/limits.d directory,
#which are read in alphabetical order, override the settings in this
#file in case the domain is the same or more specific.
#That means, for example, that setting a limit for wildcard domain here
#can be overridden with a wildcard setting in a config file in the
#subdirectory, but a user specific setting here can be overridden only
#with a user specific setting in the subdirectory.
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# LXD recommendation
# Maximum number of open files and Maximum locked-in-memory address space (KB)
*       soft    nofile  1048576
*       hard    nofile  1048576
root    soft    nofile  1048576
root    hard    nofile  1048576
*       soft    memlock unlimited
*       hard    memlock unlimited
root    soft    memlock unlimited
root    hard    memlock unlimited

# Arch wiki recommendation
# You should disallow everyone except for root from having processes of 
# minimal niceness (-20), so that riit cab fix an unresponsive system.
*       hard    nice    -19
root    hard    nice    -20

# End of file

lxc config show -e arch-ct

Expand

architecture: x86_64
config:
  boot.autostart: "true"
  image.architecture: amd64
  image.description: Archlinux current amd64 (20230304_04:18)
  image.os: Archlinux
  image.release: current
  image.requirements.secureboot: "false"
  image.serial: "20230304_04:18"
  image.type: squashfs
  image.variant: default
  security.idmap.isolated: "true"
  security.idmap.size: "2000000"
  security.nesting: "true"
  security.privileged: "false"
  security.secureboot: "false"
  security.syscalls.intercept.mknod: "true"
  volatile.base_image: f41991a6c61c46505053fe0adc8948ca6fe3a2a3b9414178905c1ef0a58b630c
  volatile.cloud-init.instance-id: b0d5d126-876a-487e-b897-5dd717747587
  volatile.eth0.host_name: veth7185bb12
  volatile.eth0.hwaddr: 00:16:3e:84:34:07
  volatile.idmap.base: "29065536"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":29065536,"Nsid":0,"Maprange":2000000},{"Isuid":false,"Isgid":true,"Hostid":29065536,"Nsid":0,"Maprange":2000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":29065536,"Nsid":0,"Maprange":2000000},{"Isuid":false,"Isgid":true,"Hostid":29065536,"Nsid":0,"Maprange":2000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: db47167b-bf39-44fd-a807-ed609dd1d612
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

mount | grep idmap

Expand

/dev/sda2 on / type btrfs (rw,relatime,idmapped,ssd,space_cache=v2,user_subvol_rm_allowed,subvolid=41488,subvol=/@/var/lib/lxd/storage-pools/default/containers/arch-ct)

mkdir {work,upper,lower,ovl}
mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work ovl

Expand

mount: /root/ovl: wrong fs type, bad option, bad superblock on overlay, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

[1443732.798184] overlayfs: idmapped layers are currently not supported

amikhalitsyn · March 6, 2023, 5:54pm

as we can see overlayfs is not getting mounted on top of idmapped btrfs. Which is also correct (for old kernel versions).

Okay, then how your setup work at all? (-: I’ve read your old reports about collabora-online. Are you using docker to deploy it? Could you check which docker storage driver are you using? I can assume that your Docker uses btrfs storage driver instead of overlayfs. This may explain how docker with idmapped mounts works for you at all on such an old kernel version.

k4my4b · March 6, 2023, 7:22pm

@amikhalitsyn I am using docker indeed

docker info
...
 Storage Driver: btrfs
...