Unable to pull image using docker inside LXC Container

LXD
1

When I run docker pull nginx I get the following error

root@still-katydid:~# docker pull nginx
Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: TLS handshake timeout

Here is my configuration

I have launched a Ubuntu 20.04 LXC container on RedHat 8.6 using the following command

lxc launch ubuntu:20.04 --storage default -c security.privileged=true -c boot.autostart=true --network lxdbr0

I installed docker in the container

docker version
Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:02:21 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 18:00:04 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.12
  GitCommit:        a05d175400b1145e5e6a735a6710579d181e7fb0
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I configured my network proxy settings in /etc/systemd/system/docker.service.d/http-proxy.conf and then restarted the docker service.


[Service]
Environment="HTTP_PROXY=<proxy address redacted>"
Environment="HTTPS_PROXY=<proxy address redacted>"

Looking at /var/log/syslog I can see some errors

Dec  8 11:57:36 still-katydid dockerd[9606]: time="2022-12-08T11:57:36.486272382Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Dec  8 11:57:36 still-katydid dockerd[9606]: time="2022-12-08T11:57:36.486290117Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Dec  8 11:57:36 still-katydid dockerd[9606]: time="2022-12-08T11:57:36.487001498Z" level=info msg="Loading containers: start."
Dec  8 11:57:36 still-katydid dockerd[9606]: time="2022-12-08T11:57:36.489111401Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: WARNING: Module bridge not found in directory /lib/modules/4.18.0-372.9.1.el8.x86_64\nmodprobe: WARNING: Module br_netfilter not found in directory /lib/modules/4.18.0-372.9.1.el8.x86_64\n, error: exit status 1"
Dec  8 11:57:36 still-katydid dockerd[9606]: time="2022-12-08T11:57:36.591405424Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"

I know the proxy address settings are correct, as I also have them configured in my .bashrc file and I can curl addresses such https://www.bbc.co.uk successfully

Also curling https://registry-1.docker.io/v2/ gives me

root@still-katydid:~# curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

I have rebooted the container and the host several times.

The firewall on the host and in the container are disabled.

2

TLS handshake timeout makes me think of a potential MTU problem.

If your host uses an abnormal MTU (not 1500), you may want to replicate that into your bridge.mtu config on the LXD bridge.

3

Agreed, my first thought too, compare ip l on the host for MTUs less than 1500 on external interfaces.

4

In addition, it’s useful to trigger the load br_netfilter module from the host by modprobe br_netfilter. Otherwise, you can step into problems with forwarding traffic between the bridge of your docker containers and your LXC container veth device (which connects the LXC container with the host). I’m not sure if docker pull is using a separate net namespace or not, so, this may help.

Also, check the output of systemctl show --property=Environment docker can you see your env variables?

5

I would actually caution against that, at least initially, as if there are existing firewall rules on the system, loading that module will result in intra-bridge traffic having those rules applied to them and this can cause unexpected connectivity loss.

This is the reason LXD does not enable that module by default, even in situations where it needs it for a particular feature to work properly.

1 Like