Container not start when mounting file into if it presents inside container - Too many levels of symbolic links

Hi,
I want to inject a custom resolv.conf file into the container:

$ lxc profile show rnic2
config: {}
devices:
  eth0:
    nictype: routed
    type: nic
  resolvconf:
    path: /etc/resolv.conf
    readonly: "true"
    source: /etc/resolv.conf
    type: disk
  root:
    path: /
    pool: default
    size: 512GiB
    type: disk
  shared:
    path: /shared
    readonly: "true"
    source: /shared
    type: disk
name: rnic2
used_by:
- /1.0/instances/c100

And container with this doesn’t start.
In case of removing resolvconf section all work as expected.

$ lxc info --show-log c100
Name: c100
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2022/09/07 14:28 EEST
Last Used: 2022/09/07 14:28 EEST

Log:

lxc c100 20220907112823.456 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:fchowmodat:1252 - No such file or directory - Failed to fchownat(42, memory.oom.group, 65536, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc c100 20220907112823.561 ERROR    utils - ../src/lxc/utils.c:open_without_symlink:1085 - Too many levels of symbolic links - resolv.conf in /usr/lib/lxc/rootfs/etc/resolv.conf was a symbolic link!
lxc c100 20220907112823.561 ERROR    conf - ../src/lxc/conf.c:mount_entry:2410 - Too many levels of symbolic links - Failed to mount "/var/lib/lxd/devices/c100/disk.resolvconf.etc-resolv.conf" on "/usr/lib/lxc/rootfs/etc/resolv.conf"
lxc c100 20220907112823.561 ERROR    conf - ../src/lxc/conf.c:lxc_setup:4375 - Failed to setup mount entries
lxc c100 20220907112823.561 ERROR    start - ../src/lxc/start.c:do_start:1275 - Failed to setup container "c100"
lxc c100 20220907112823.561 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc c100 20220907112823.569 WARN     network - ../src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from "eth0" to its initial name "veth71596fcd"
lxc c100 20220907112823.569 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING"
lxc c100 20220907112823.569 ERROR    start - ../src/lxc/start.c:__lxc_start:2074 - Failed to spawn container "c100"
lxc c100 20220907112823.570 WARN     start - ../src/lxc/start.c:lxc_abort:1039 - No such process - Failed to send SIGKILL via pidfd 43 for process 27689
lxc 20220907112828.773 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220907112828.773 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"
lxc 20220907112828.773 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220907112828.773 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"

P.S. I found if I delete resolv.conf file manually m /var/lib/lxd/containers/c100/rootfs/etc/resolv.conf container starts.
Can I overwrite an existing file or maybe there is more correct way to configure DNS in routed network without parent interface?

Where does /etc/resolv.conf point to on the host and in the container (if you remove the disk section you added).

This is a standard system file. On the host it lies on /etc and should mapped to /etc folder.
Looks like it cannot overwrite it.

Btw, Is there is more correct way to configure DNS in routed container? :wink:

I meant where does /etc/resolv.conf resolve to?

The normal way to do is is to use cloud-init, see How to get LXD containers get IP from the LAN with routed network

This way the container’s config isn’t tightly coupled to the host’s.

It simply contains default DNS servers:

search example.com
nameserver 8.8.8.8
nameserver 1.1.1.1

Thanks for link, but it doesn’t work for me, maybe because I don’t create a separate profile for every container. This config does’n do any good:

# lxc profile show rnic2
config:
  user.network-config: |
    version: 2
    ethernets:
      eth0:
        nameservers:
          addresses:
          - 8.8.8.8
          - 1.1.1.1
          search: []
description: Routed LXD profile

In case I try to execute on boot commands it doesn’t work too:

config:
  user.user-data: |
    bootcmd:
      - [sh, -c, "truncate -s 0 /etc/resolv.conf"]
      - [sh, -c, "echo 'nameserver 8.8.8.8' >> /etc/resolv.conf"]
      - [sh, -c, "echo 'nameserver 1.1.1.1' >> /etc/resolv.conf"]
      - [sh, -c, "echo 'nameserver 1.1.1.1' >> /etc/resolv2.conf"]
description: Routed LXD profile

Even /etc/resolv2.conf doesn’t created :smiling_face_with_tear:

I meant what does ls -la /etc/resolv.conf show?

Are you using an image that includes cloud init, such as images:ubuntu/22.04/cloud or ubuntu:22.04?

Default outputs are.
Host system:

ls -la /etc/resolv.conf 
-rw-rw-r-- 1 root root 203 Feb  5  2021 /etc/resolv.conf

Container is a link :thinking:

# ls -la /etc/resolv.conf 
lrwxrwxrwx 1 root root 39 Aug 24 21:39 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

Do I need to overwrite ./run/systemd/resolve/stub-resolv.conf ?

I use 18.04 or 20.04 with no luck.

lxc init ubuntu:18.04 c100 -p rnic2
lxc init ubuntu:20.04 c101 -p rnic2
config:
  user.user-data: |
    runcmd:
      - truncate -s 0 /etc/resolv.conf
      - echo 'nameserver 8.8.8.8' >> /etc/resolv.conf
      - echo 'nameserver 1.1.1.1' >> /etc/resolv.conf
      - echo 'nameserver 1.1.1.1' >> /etc/resolv2.conf

Looks like runcmd or bootcmd are ignored.

Ignoring DNS for the moment.

How are you currently configuring the IPs on eth0 inside the container?

Ok, after creating container I simply execute lxc config device override c100 eth0 ipv4.address=X.X.X.126 and this creates container section:

devices:
  eth0:
    ipv4.address: X.X.X.126
    nictype: routed
    type: nic

And does that work? Normally the default DHCP client inside the container will wipe the IP config from eth0 when the container starts? Hence why that article suggests using cloud-init for IP config as well.

What do you you see for ip a and ip r inside the container?

Yes with your help, you forget :wink:
Removing default netplan config inside container do the work.

Right, I didn’t realise those posts were associated to this one.

Because systemd-resolved uses a symlink at /etc/resolv.conf to its only dynamically managed file at /run/systemd/resolve/stub-resolv.conf I suggest that as you’re already manually modifiyng the container (by removing the default netplan config) that you also remove /etc/resolv.conf from your container using:

rm /etc/resolv.conf 

This should then allow you to add the resolve disk you were originally trying.
If that doesn’t work then you could also try specifying the disk’s source as where the /etc/resolv.conf symlink resolves to on our LXD host.

Understand, thanks for your time :grinning:

1 Like