RedHat 8.7 kernel problems with starting LXC containers

LXC
1

With the new kernel in RH 8.7 we cannot start our LXC containers anymore (on RH 6.x,7.x and 8.6 everything worked ok)

When I start a test container with : lxc-start -n lxcgcc -o debug -l debug
I can see in the debug that it goes wrong after trying to mount the rootfs

lxc-start lxcgcc 20230317154527.486 DEBUG conf - …/src/lxc/conf.c:lxc_mount_rootfs:1439 - Mounted rootfs “/usr/test/lxc/containers/lxcgcc/rootfs” onto “/usr/test/lxc/lib64/lxc/rootfs” with options “(null)”
lxc-start lxcgcc 20230317154527.486 ERROR conf - …/src/lxc/conf.c:lxc_setup_rootfs_prepare_root:3993 - Failed to setup rootfs for
lxc-start lxcgcc 20230317154527.486 ERROR conf - …/src/lxc/conf.c:lxc_setup:4368 - Failed to setup rootfs
lxc-start lxcgcc 20230317154527.486 ERROR start - …/src/lxc/start.c:do_start:1272 - Failed to setup container “lxcgcc”

/usr/test is a separated filesystem:
/dev/mapper/vgusrtest-lvusrtest on /usr/test type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=4096,noquota)

RH8.7: Linux nlsaptstrh87 4.18.0-425.3.1.el8.x86_64 #1 SMP Fri Sep 30 11:45:06 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

The rootfs exists and also the lib64/lxc/rootfs,
if we change the kernel back to RH8.6, it starts the container correctly…

Same results with new stable builds of 4.012 and 5.02

Anybody any ideas ?

2

Ok, I looked at the conf.c at line 3993 and I added some addditional debug output

        DEBUG("Mounted rootfs \"%s\" onto \"%s\" with options \"%s\"",
              rootfs->path, rootfs->mount,
              rootfs->mnt_opts.raw_options ? rootfs->mnt_opts.raw_options : "(null)");

        rootfs->dfd_mnt = open_at(-EBADF, rootfs->mount, PROTECT_OPATH_DIRECTORY, PROTECT_LOOKUP_ABSOLUTE_XDEV, 0);
        DEBUG("open_at result %d\n",rootfs->dfd_mnt);
        if (rootfs->dfd_mnt < 0) {
                DEBUG("error at open_at %d\n",errno);
                return -errno;
        }

It goes wrong with the new kernel at the open_at syscall
It return errno 40 ELOOP Too many symbolic links encountered

So I started removing every softlink I had in the path to the rootfs of the container
I had 2 softlinks in the path:

  1. /usr/test/lxc pointed to different lxc versions I had in the same directory
  2. in the config I used a path to the rootfs which was a softlink to the rootfs directory

I found out that the first one was causing the problem… as soon as I made the /usr/test/lxc5rh the /usr/test/lxc directory it started to work
Seems this RH8.7 kernel does not like any softlinks in the lxc path

Anyone knows why there are no softlinks allowed in the new kernel ?