No LXD containers can access DNS in k8s cluster (but LXD host can)

cbc02009 · September 26, 2022, 2:27pm

I have a DNS resolver in my k8s cluster that is accessible outside the cluster at 10.0.1.4:53, but for some reason none of my LXD containers can access it, even though all hosts and external PCs can. on the hosts I can use it to resolve hosts:

❯ dig postgres-rw.databases.svc.cluster.local @10.0.1.4

; <<>> DiG 9.18.7 <<>> postgres-rw.databases.svc.cluster.local @10.0.1.4
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4421
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: de427f2b147379be (echoed)
;; QUESTION SECTION:
;postgres-rw.databases.svc.cluster.local. IN A

;; ANSWER SECTION:
postgres-rw.databases.svc.cluster.local. 30 IN A 10.10.8.145

;; Query time: 0 msec
;; SERVER: 10.0.1.4#53(10.0.1.4) (UDP)
;; WHEN: Mon Sep 26 10:21:25 EDT 2022
;; MSG SIZE  rcvd: 135

but from inside an lxd container it times out:

❯ dig postgres-rw.databases.svc.cluster.local @10.0.1.4

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> postgres-rw.databases.svc.cluster.local @10.0.1.4
;; global options: +cmd
;; connection timed out; no servers could be reached

The LXD host is running version 5.6 on archlinux (kernel 5.19), and the network adapter for LXD is a single-device bridge configured via systemd-networkd. The only thing I can think of is that maybe LXD has some sort of firewall set up?

I’ve tried with multiple different LXD container images, and even spun up a fresh LXD instance on a raspberry pi, and all have produced the same results.

I’ve also tried with the k8s cluster’s fqdn (postgres.{external domain}) instead of the cluster.local addresses with the same result.

Any help on this would be greatly appreciated, I’m not really an expert on DNS.

tomp · September 27, 2022, 9:33am

Please show ip a and ip r inside the container, as well as on the LXD host.

Please also show lxc config show <instance> --expanded

cbc02009 · September 27, 2022, 10:30am

Sure thing!

Container config:

❯ lxc config show Pihole --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 22.04 LTS amd64 (release) (20220923)
  image.label: release
  image.os: ubuntu
  image.release: jammy
  image.serial: "20220923"
  image.type: squashfs
  image.version: "22.04"
  volatile.base_image: 4979c8f15a0003b2b72cac34e9676b4107ae330874935284d0ba5d54199c7744
  volatile.cloud-init.instance-id: 92e57abd-12d0-4d38-9232-1c53a69f1194
  volatile.eth0.host_name: vetha11bd094
  volatile.eth0.hwaddr: 00:16:3e:51:55:24
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: b84cab5e-bcab-4225-9f8f-24741e8cf3d6
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

ip commands from inside container:

❯ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:51:55:24 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.90/23 brd 10.0.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe51:5524/64 scope link
       valid_lft forever preferred_lft forever
❯ ip r
default via 10.0.0.1 dev eth0 proto static
10.0.0.0/23 dev eth0 proto kernel scope link src 10.0.0.90

ip commands from host:

❯ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether e4:1d:2d:30:91:b1 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.10/23 brd 10.0.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::e61d:2dff:fe30:91b1/64 scope link
       valid_lft forever preferred_lft forever
3: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether e4:1d:2d:30:91:b2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e61d:2dff:fe30:91b2/64 scope link
       valid_lft forever preferred_lft forever
4: enp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 34:97:f6:31:80:43 brd ff:ff:ff:ff:ff:ff
5: vlan-k8s@br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:1d:2d:30:91:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.10/24 brd 10.0.3.255 scope global vlan-k8s
       valid_lft forever preferred_lft forever
    inet6 fe80::e61d:2dff:fe30:91b2/64 scope link
       valid_lft forever preferred_lft forever
6: vlan-san@br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether e4:1d:2d:30:91:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.10/24 brd 10.0.4.255 scope global vlan-san
       valid_lft forever preferred_lft forever
    inet6 fe80::e61d:2dff:fe30:91b2/64 scope link
       valid_lft forever preferred_lft forever
7: enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master br0 state UP group default qlen 1000
    link/ether e4:1d:2d:30:91:b1 brd ff:ff:ff:ff:ff:ff
8: enp10s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master br1 state UP group default qlen 1000
    link/ether e4:1d:2d:30:91:b2 brd ff:ff:ff:ff:ff:ff
10: vethc412cd50@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether b2:f1:25:92:95:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
12: veth5698c457@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether ee:59:b2:97:39:bc brd ff:ff:ff:ff:ff:ff link-netnsid 1
14: vetha60457b8@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 3e:e1:f8:e2:35:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 2
16: vetha11bd094@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether a6:4a:c8:c7:25:4f brd ff:ff:ff:ff:ff:ff link-netnsid 3
18: veth0535c0af@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 1a:f7:af:0e:fa:40 brd ff:ff:ff:ff:ff:ff link-netnsid 4
19: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 06:2c:e5:d3:2d:7e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::42c:e5ff:fed3:2d7e/64 scope link
       valid_lft forever preferred_lft forever
20: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether ae:a3:f6:07:d2:70 brd ff:ff:ff:ff:ff:ff
    inet 10.11.2.158/32 scope link cilium_host
       valid_lft forever preferred_lft forever
    inet6 fe80::aca3:f6ff:fe07:d270/64 scope link
       valid_lft forever preferred_lft forever
21: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 02:5d:d7:e6:35:ac brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5d:d7ff:fee6:35ac/64 scope link
       valid_lft forever preferred_lft forever
23: lxc_health@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether a2:e8:c6:f3:e7:27 brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::a0e8:c6ff:fef3:e727/64 scope link
       valid_lft forever preferred_lft forever
25: lxcffe6a56636bf@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 2e:da:76:62:86:cf brd ff:ff:ff:ff:ff:ff link-netns 964733a0-6a19-4545-bbfa-8bdd190e3ce4
    inet6 fe80::2cda:76ff:fe62:86cf/64 scope link
       valid_lft forever preferred_lft forever
27: lxc10a950ba207a@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether a2:2c:d2:9c:fb:20 brd ff:ff:ff:ff:ff:ff link-netns 8b508cce-dec2-4c17-8f58-8dfb45bef1fe
    inet6 fe80::a02c:d2ff:fe9c:fb20/64 scope link
       valid_lft forever preferred_lft forever
29: lxc46bb7c038203@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:bf:b9:45:1e:c4 brd ff:ff:ff:ff:ff:ff link-netns 61ce169b-5750-4d24-95fe-c6100c9e7029
    inet6 fe80::78bf:b9ff:fe45:1ec4/64 scope link
       valid_lft forever preferred_lft forever
33: lxc28df28e17d3c@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether b6:37:e1:91:15:cd brd ff:ff:ff:ff:ff:ff link-netns 63d996c5-ede2-4b9b-996b-740196cc5956
    inet6 fe80::b437:e1ff:fe91:15cd/64 scope link
       valid_lft forever preferred_lft forever
35: lxca1239a8bed5c@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 02:7a:4f:40:5b:9f brd ff:ff:ff:ff:ff:ff link-netns edd58594-707d-4d01-9989-489df265bf1e
    inet6 fe80::7a:4fff:fe40:5b9f/64 scope link
       valid_lft forever preferred_lft forever
69: lxc5c2aa4c00e4f@if68: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether e2:c9:c7:92:22:e5 brd ff:ff:ff:ff:ff:ff link-netns d0b03a2d-faa3-4380-9e5b-b65a54f1dfab
    inet6 fe80::e0c9:c7ff:fe92:22e5/64 scope link
       valid_lft forever preferred_lft forever
71: lxc6bef8bccd841@if70: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether 9e:49:b9:ea:1c:ff brd ff:ff:ff:ff:ff:ff link-netns b7c48c14-3ad1-4eef-a607-44b3ac74a84e
    inet6 fe80::9c49:b9ff:feea:1cff/64 scope link
       valid_lft forever preferred_lft forever
❯ ip r
default via 10.0.0.1 dev br0 proto static
default via 10.0.3.1 dev vlan-k8s proto static
10.0.0.0/23 dev br0 proto kernel scope link src 10.0.0.10
10.0.3.0/24 dev vlan-k8s proto kernel scope link src 10.0.3.10
10.0.4.0/24 dev vlan-san proto kernel scope link src 10.0.4.10
10.11.0.0/24 via 10.11.2.158 dev cilium_host src 10.11.2.158 mtu 8950
10.11.1.0/24 via 10.11.2.158 dev cilium_host src 10.11.2.158 mtu 8950
10.11.2.0/24 via 10.11.2.158 dev cilium_host src 10.11.2.158
10.11.2.158 dev cilium_host scope link

And just for completeness here are the results from the Pi I also used to test (just to see if it was because k8s and lxd were running on the same host):

lxd config:

❯ lxc config show Pihole2 --expanded
architecture: aarch64
config:
  image.architecture: arm64
  image.description: Ubuntu jammy arm64 (20220925_07:42)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20220925_07:42"
  image.type: squashfs
  image.variant: cloud
  security.nesting: "true"
  user.vendor-data: |
    #cloud-config
    users:
      - name: ...
        ssh_authorized_keys:
          - ...
        plain_text_passwd: '...'
        lock_passwd: false
        sudo: ALL=(ALL) ALL
        shell: /bin/bash
    package_update: "true"
    package_upgrade: "true"
    packages:
      - git
      - zsh
      - micro
    timezone: America/New_York
    locale: en_US.UTF-8
    runcmd:
      - <clone zsh config and chsh to zsh>
  volatile.base_image: 576d80fd17e1c7b1890584c496da91a106270fd5ce0d826f796f46404eed79b8
  volatile.cloud-init.instance-id: b2763d51-4ac6-41d7-aae5-2050576056d4
  volatile.eth0.host_name: veth332aab5c
  volatile.eth0.hwaddr: 00:16:3e:97:12:aa
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 8b409179-d2fb-4808-b71a-4106d79e8565
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- cloud-default
stateful: false
description: ""

ip commands from container:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:97:12:aa brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.1.141/23 metric 100 brd 10.0.1.255 scope global dynamic eth0
       valid_lft 480041sec preferred_lft 480041sec
    inet6 fe80::216:3eff:fe97:12aa/64 scope link
       valid_lft forever preferred_lft forever
Pihole2# ip r
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.1.141 metric 100
10.0.0.0/23 dev eth0 proto kernel scope link src 10.0.1.141 metric 100
10.0.0.1 dev eth0 proto dhcp scope link src 10.0.1.141 metric 100
10.0.0.90 dev eth0 proto dhcp scope link src 10.0.1.141 metric 100

ip commands from host:

❯ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether dc:a6:32:7c:e6:e5 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether dc:a6:32:7c:e6:e5 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.40/23 brd 10.0.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::dea6:32ff:fe7c:e6e5/64 scope link
       valid_lft forever preferred_lft forever
4: veth2bd59447@veth2b741687: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:16:3e:38:d4:82 brd ff:ff:ff:ff:ff:ff
5: veth2b741687@veth2bd59447: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master br0 state LOWERLAYERDOWN group default qlen 1000
    link/ether 42:87:62:c0:f4:42 brd ff:ff:ff:ff:ff:ff
13: veth332aab5c@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether f6:00:4e:9f:05:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
❯ ip r
default via 10.0.0.1 dev br0 proto static
10.0.0.0/23 dev br0 proto kernel scope link src 10.0.0.40

Also just to be clear, the container on the Pi is not running any software (like Pihole) other than what’s on the base image plus what’s defined in the cloud-config. This was to test if it was something about pihole that was causing the issue.

tomp · September 28, 2022, 10:23am

I think running sudo tcpdump -i br0 host 10.0.1.4 port 53 and then running a dig command from inside the container would help to confirm that the packet is making it onto the bridge.

Then using a similar command on the k8s host to check if the request is making it in.

And then observe if the return packets are exiting k8s host and arriving at the br0 interface.