Docker on Debian and Ubuntu

Hello. I am running Docker on both Ubuntu and Debian LXD containers. There seems to be no issues with the Docker on Ubuntu, but with Docker on Debian, I am getting network issues within the docker container.

Just now I found that from docker info that Docker on Debian doesn’t support the cgroup set, and the cgroup driver is set to ‘systemd’ whereas cgroup driver on Ubuntu is set to “cgroupfs”.

Is it because there’s no systemd in docker I am not having internet access inside containers on Debian instance? I only have network access (within the docker container) only when the Docker container is privileged.

Sorry, my post may be little confusing…thank you in advance for taking your time reading it.

So…yeah…here is the docker info from both LXD Ubuntu and Debian.

Docker Info from Debian 11

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 1
 Server Version: 20.10.14
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.8.0-55-generic
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 5
 Total Memory: 2.328GiB
 Name: debiandocker
 ID: UZJC:7VTX:HV77:XHTC:ARBQ:ZWYZ:46LL:XAPS:FU4H:3NZO:L3EN:WKMX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support

Docker info from Ubuntu Focal

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.1-docker)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 20.10.14
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.8.0-48-generic
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 9.313GiB
 Name: avatar
 ID: GQQM:OUL4:4B6C:DPZX:NUKL:TTEH:7RA6:LX5N:GJ7C:J5PF:MIW3:KJKQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I really appreciate your help. Please let me know if you need any other information that you may find useful to troubleshoot this issue. I’ll be happy to provide. Thanks.!

EDIT: Just so you know, IP Forwarding is enabled on both hosts.

I wouldn’t expect the cgroup driver to have any impact on this.
I guess the first thing to check would be whether the container got an IP address at all, then if it did, use tcpdump against the Docker bridge to see if the traffic is making it that far. Try to ping the docker bridge address from within the container. If all that works fine, then start running tcpdump on eth0 in the LXD container to see if the traffic is making it out.

Hello @stgraber

Yes, the containers are getting an IP address from the docker bridge (both privileged and unprivileged). From an unprivileged container I can’t ping the docker bridge, and ifconfig shows no interfaces (but how can docker inspect show that an IP is assigned). However, with a privileged container, ping works and ifconfig shows the eth0 and loopback interface.

Pinging the docker bridge IP from the inside of an unprivileged container gives this error:

ping: socket: Permission denied

Ah, so that’s a capability problem, maybe not a network problem.

Instead of using ping, can you maybe use something like netcat, curl or wget to check actual connectivity?

Ping is a bit special because it requires additional privileges for a normal user to run it and it looks like something in your Docker layers may be interfering with it.

@stgraber I tried “apt update” inside the container. On a privileged container, I had no issues, the packages were updated. But on the unprivileged, it didn’t work.

I checked the nameservers in resolv.conf file in both privileged and unprivileged containers. They look perfect. The contents of the file are below.

nameserver 8.8.8.8
nameserver 1.1.1.1

Is there something else I need to test?

@stgraber And, ask you asked, here is output from wget.

In unprivileged container

root@a0bc80dc5904:~# wget https://google.com
--2022-04-05 08:42:50--  https://google.com/
Resolving google.com (google.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘google.com’

In privileged container:

root@e77205cd5fa0:~# wget https://google.com
--2022-04-05 09:11:46--  https://google.com/
Resolving google.com (google.com)... 142.250.74.78, 2a00:1450:400f:802::200e
Connecting to google.com (google.com)|142.250.74.78|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.google.com/ [following]
--2022-04-05 09:11:47--  https://www.google.com/
Resolving www.google.com (www.google.com)... 216.58.207.196, 2a00:1450:400f:80b::2004
Connecting to www.google.com (www.google.com)|216.58.207.196|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                      [ <=>                                                                                     ]  16.40K  --.-KB/s    in 0.03s   

2022-04-05 09:11:47 (625 KB/s) - ‘index.html’ saved [16789]

Ok, what does ip -4 a and ip -4 r look like in the container?
If that looks correct, then I’d recommend running tcpdump -ni docker0 or whatever the bridge is called to see what kind of traffic makes it from the container.

@stgraber First of all, thank you very much for your help. You have been following up on this issue. Once again, thanks a lot!

Okay. Looks like it’s not correct.

On unprivileged container:

root@a061d76b020f:~# ip -4 a
Cannot open netlink socket: Permission denied
root@a061d76b020f:~# ip -4 r
Cannot open netlink socket: Permission denied

On privileged container, as usual, there are no issues:

root@c9c50efcd04a:~# ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
19: eth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default  link-netnsid 0
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
root@c9c50efcd04a:~# ip -4 r
default via 172.17.0.1 dev eth0 
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.3

Regarding tcpdump, I see no output but I am unsure if I have to run inside the LXD container or on the host level as root.

Hi @stgraber

As posted in this GH issue, I removed AppArmor and an unprivileged Docker container got internet access. But people (in that issue) say it’s not a good solution. What do I do?

Are you seeing similar apparmor errors in dmesg?

@stgraber I’m not sure. I’m running Docker inside an unprivileged LXD Debian container, so I have no access to dmesg.

You’d need to look at dmesg from the host.

The host itself is an unprivileged Debian container. It’s an LXD VPS (instead of KVM, or something else).

Ah, okay, so yeah, that limits your options for debugging :slight_smile:

@stgraber I’ve another server (also an LXD VPS). I run Docker inside Ubuntu LXD container. On that server too apparmor is installed, but I haven’t seen any issues. It’s on Debian I’m having these issues.

Well. Would there be any problem if I just uninstall AppArmor and use Docker - as I get internet access in unprivileged Docker containers (while this was not the case before)? Any security concerns without AppArmor?

Given the “host” is itself an unprivileged container, I wouldn’t be too concerned by it, no.

@stgraber

It’s been too long…

I see errors in the dmesg output. I am try to spin up an Nginx docker container inside an LXD Debain Bullseye container.

[ 1334.886165] audit: type=1400 audit(1695623693.319:93): apparmor="DENIED" operation="create" namespace="root//lxd-bullseye_<var-snap-lxd-common-lxd>" profile="docker-default" pid=5079 comm="nginx" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create" addr=none
[ 1334.886170] audit: type=1400 audit(1695623693.319:94): apparmor="DENIED" operation="create" namespace="root//lxd-bullseye_<var-snap-lxd-common-lxd>" profile="docker-default" pid=5079 comm="nginx" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create" addr=none
[ 1334.886193] audit: type=1400 audit(1695623693.319:95): apparmor="DENIED" operation="create" namespace="root//lxd-bullseye_<var-snap-lxd-common-lxd>" profile="docker-default" pid=5079 comm="nginx" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create" addr=none
[ 1334.886196] audit: type=1400 audit(1695623693.319:96): apparmor="DENIED" operation="create" namespace="root//lxd-bullseye_<var-snap-lxd-common-lxd>" profile="docker-default" pid=5079 comm="nginx" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create" addr=none
[ 1334.887506] audit: type=1400 audit(1695623693.319:97): apparmor="DENIED" operation="create" namespace="root//lxd-bullseye_<var-snap-lxd-common-lxd>" profile="docker-default" pid=5079 comm="nginx" family="inet" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create"

Let me know if you need more information.