How to get IP from LAN DHCP server?

mirodin · July 17, 2020, 10:18am

Hey guys,

I dug through a lot of posts and forums today trying to figure out how to setup my containers so they are reachable from the outside like a physical machine would. I read Simos’ great writeups and tried to “translate” his LXD setup but failed unfortunately. Over on reddit I got told that I might have better luck here…

My desktop runs VoidLinux x86_64, this is what I did after reading dozens of posts about bridge networking:

First I created the network bridge:

> ip link add name br0 type bridge
> ip link set dev br0 up
> ip link set dev enp2s0 master br0

When I ran ip a I could see that my DHCP server had given the IP address which was usually given to enp2so to br0.

> ip a

enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
link/ether 50:e5:49:5a:bb:74 brd ff:ff:ff:ff:ff:ff
inet6 fe80::52e5:49ff:fe5a:bb74/64 scope link 
   valid_lft forever preferred_lft forever

br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 50:e5:49:5a:bb:74 brd ff:ff:ff:ff:ff:ff
inet 10.1.0.2/16 brd 10.1.255.255 scope global dynamic noprefixroute br0
   valid_lft 41997sec preferred_lft 34843sec
inet6 fd07:ca45:2468::851/128 scope global noprefixroute 
   valid_lft forever preferred_lft forever
inet6 fd07:ca45:2468:0:d4c7:eea1:eff5:b69a/64 scope global mngtmpaddr noprefixroute 
   valid_lft forever preferred_lft forever
inet6 fe80::3043:61ff:fe8a:170/64 scope link 
   valid_lft forever preferred_lft forever

After this I created the container:

> lxc-create -t download -n test

I selected a VoidLinux container so there was no “switching” between tools for me.

> cat /var/lib/lxc/test/config

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template:
# Template script checksum (SHA-1): 9893b2e0dba7be0d74cf38537bebe0af939c269c
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)


# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = linux64

# Container specific configuration
lxc.rootfs.path = dir:/var/lib/lxc/test/rootfs
lxc.uts.name = test

# Network configuration
lxc.net.0.type = empty

Which I changed to this:

> cat /var/lib/lxc/test/config

...

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.veth.pair = test

When I start this container I do not get a IPv4 from my DHCP server, but I can ping its IPv6 address from my router:

> ip a # on host machine

test@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
link/ether fe:cb:01:72:6d:5c brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::fccb:1ff:fe72:6d5c/64 scope link 
   valid_lft forever preferred_lft forever

> lxc-ls -f
NAME STATE   AUTOSTART GROUPS IPV4 IPV6                                 UNPRIVILEGED 
test RUNNING 0         -      -    fd07:ca45:2468:0:1c5b:e5ff:fe48:ed04 false

I think I missed something here but since I just started with LXC I can not figure out where I messed up.

Any help would be very much appreciated, thanks in advance.

tomp · July 20, 2020, 8:17am

What is the output of ip a inside your container?

mirodin · July 23, 2020, 12:44pm

Sorry for my late reply, I must have missed the notification. After fiddeling around with this problem and not getting to any result I completly scrubbed this setup and redid my server. Now everything seems to work just fine, every container gets its public IP as I planned.

Thank you for answering anyways

mirodin · July 23, 2020, 5:02pm

@tomp Well, seems like I jumped to soon. Now the same problem turned up again. I did nothing but restart my machine once today and now it does not work anymore. This is my output when I run dhcpcd inside the container:

> dhcpcd -B eth0
dhcpcd-9.1.4 starting
DUID 00:01:00:01:26:ac:7e:72:aa:00:00:00:00:12
eth0: IAID 00:00:00:12
eth0: soliciting an IPv6 router
eth0: Router Advertisement from fe80::220c:c8ff:fe46:55c0
eth0: adding address fd07:ca45:2468:0:df66:131e:6757:5a47/64
eth0: adding route to fd07:ca45:2468::/64
eth0: soliciting a DHCPv6 lease
eth0: soliciting a DHCP lease
eth0: ADV fd07:ca45:2468::347/128 from fe80::220c:c8ff:fe46:55c0
eth0: REPLY6 received from fe80::220c:c8ff:fe46:55c0
eth0: adding address fd07:ca45:2468::347/128
eth0: renew in 21600, rebind in 34560, expire in 4294967295 seconds
eth0: probing for an IPv4LL address
eth0: using IPv4LL address 169.254.234.216
eth0: adding route to 169.254.0.0/16
eth0: adding default route

Ouptut from ip a:

> ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:00:00:00:00:12 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.234.216/16 brd 169.254.255.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fd07:ca45:2468::347/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fd07:ca45:2468:0:df66:131e:6757:5a47/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fd07:ca45:2468:0:a800:ff:fe00:12/64 scope global dynamic mngtmpaddr 
       valid_lft forever preferred_lft forever
    inet6 fe80::a800:ff:fe00:12/64 scope link 
       valid_lft forever preferred_lft forever

tomp · July 24, 2020, 8:27am

So we can see you’re getting an IPv6 router advertisement from the external network, and there is bi-directional communication occurring in the form of a DHCPv6 request/response.

So that shows your bridge is working.

As for IPv4, the next step is to use something like tcpdump on various interfaces (e.g. start at the bridge then follow it to the external interface and potentially beyond) to track the DHCPv4 request/response and see where it is getting lost. It may be a firewall somewhere.

mirodin · July 24, 2020, 12:57pm

I never did any package sniffing so I might have done something wrong. But when I ran

tcpdump -i br0 -vvv 'port 67 or port 68'

and started the container I did not get any output. So from my noobish view no packages reach the bridge interface in the first place?

tomp · July 24, 2020, 1:15pm

Try running it on the veth-pair interface that connects to the bridge. ip l will show you the interfaces on your LXD host, if you change your container’s config to add the lxc.net.[i].veth.pair setting to specify a name of the host-side interface, it will make it easier to identify which interface is for your container.

mirodin · July 24, 2020, 8:30pm

Thank you @tomp for your help, I very much appreciate that!

I ran tcpdump on the bridge again and then on the interface created by the container just to verify. I opened the resulting dump in Wireshark and filtered for UDP ports 67 and 68. This is my result:

No.	Time	Source	Destination	Protocol	Length	Info
272	15.040393	0.0.0.0	255.255.255.255	DHCP	342	DHCP Discover - Transaction ID 0x9dcbb5da
316	18.351811	0.0.0.0	255.255.255.255	DHCP	342	DHCP Discover - Transaction ID 0x9dcbb5da
377	26.331501	0.0.0.0	255.255.255.255	DHCP	342	DHCP Discover - Transaction ID 0x9dcbb5da

So it seems there is some DHCP request getting outside. Fiddeling around I was wondering if a docker install on the same machine might have messed with something since I installed docker after having setup br0. Is there any known issue?

mirodin · July 26, 2020, 7:42am

I did some more testing via tcpdump and I get the same output for the bridge and the interface of the container.

Inside the container:

> dhcpcd -B eth0
dhcpcd-9.1.4 starting
DUID 00:01:00:01:26:ac:7e:72:aa:00:00:00:00:12
eth0: IAID 00:00:00:12
eth0: soliciting an IPv6 router
eth0: Router Advertisement from fe80::220c:c8ff:fe46:55c0
eth0: adding address fd07:ca45:2468:0:df66:131e:6757:5a47/64
eth0: adding route to fd07:ca45:2468::/64
eth0: confirming prior DHCPv6 lease
eth0: soliciting a DHCP lease
eth0: DHCPv6 REPLY: Not On Link
eth0: ADV fd07:ca45:2468::610/128 from fe80::220c:c8ff:fe46:55c0
eth0: REPLY6 received from fe80::220c:c8ff:fe46:55c0
eth0: adding address fd07:ca45:2468::610/128
eth0: renew in 21600, rebind in 34560, expire in 4294967295 seconds

On the host machine:

> sudo tcpdump -i br0 -v -n 'port 67 or port 68 or port 69'
tcpdump: listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:36:06.079445 IP (tos 0x0, ttl 64, id 4786, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:00:00:00:00:12, length 300, xid 0x37ef3620, Flags [none]
	  Client-Ethernet-Address aa:00:00:00:00:12
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Parameter-Request Option 55, length 14: 
	      Subnet-Mask, Classless-Static-Route, Default-Gateway, Domain-Name-Server
	      Hostname, Domain-Name, MTU, BR
	      Static-Route, Lease-Time, Server-ID, RN
	      RB, Option 119
	    MSZ Option 57, length 2: 1472
	    Client-ID Option 61, length 19: hardware-type 255, 00:00:00:12:00:01:00:01:26:ac:7e:72:aa:00:00:00:00:12
	    SLP-NA Option 80, length 0""
	    NOAUTO Option 116, length 1: Y
	    T145 Option 145, length 1: 1
09:36:09.810574 IP (tos 0x0, ttl 64, id 30369, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:00:00:00:00:12, length 300, xid 0x37ef3620, secs 3, Flags [none]
	  Client-Ethernet-Address aa:00:00:00:00:12
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Parameter-Request Option 55, length 14: 
	      Subnet-Mask, Classless-Static-Route, Default-Gateway, Domain-Name-Server
	      Hostname, Domain-Name, MTU, BR
	      Static-Route, Lease-Time, Server-ID, RN
	      RB, Option 119
	    MSZ Option 57, length 2: 1472
	    Client-ID Option 61, length 19: hardware-type 255, 00:00:00:12:00:01:00:01:26:ac:7e:72:aa:00:00:00:00:12
	    SLP-NA Option 80, length 0""
	    NOAUTO Option 116, length 1: Y
	    T145 Option 145, length 1: 1
09:36:16.866833 IP (tos 0x0, ttl 64, id 10764, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:00:00:00:00:12, length 300, xid 0x37ef3620, secs 10, Flags [none]
	  Client-Ethernet-Address aa:00:00:00:00:12
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Parameter-Request Option 55, length 14: 
	      Subnet-Mask, Classless-Static-Route, Default-Gateway, Domain-Name-Server
	      Hostname, Domain-Name, MTU, BR
	      Static-Route, Lease-Time, Server-ID, RN
	      RB, Option 119
	    MSZ Option 57, length 2: 1472
	    Client-ID Option 61, length 19: hardware-type 255, 00:00:00:12:00:01:00:01:26:ac:7e:72:aa:00:00:00:00:12
	    SLP-NA Option 80, length 0""
	    NOAUTO Option 116, length 1: Y
	    T145 Option 145, length 1: 1

So I think there is no problem with my host, is there? What would be the next step in debugging this?

tomp · July 27, 2020, 8:03am

Yes docker is well known for blocking LXD traffic, specifically DHCP. Have a search in these forums for docker. Its default firewall blocks other DHCP requests.

simos · July 27, 2020, 8:48am

If you really really need to run Docker on the same host along with LXD, I suggest to check whether the Docker application can communicate with the Internet over a Unix socket. For example, Discourse does. Then, you can setup a container with a reverse proxy (HAProxy and nginx are OK), and pass that Unix socket to the reverse proxy. By doing so, Docker will not mess up the host with firewall rules.

mirodin · July 27, 2020, 8:54pm

Yes docker is well known for blocking LXD traffic

Well that is a bummer.

If you really really need to run Docker on the same host along with LXD

I do not need Docker to run on the same host. I would rather have docker run inside an unprivileged LXC container. But I could not manage to get Docker start inside a container so I did the next best thing. All instructions I could find about Docker inside LXC are about LXD which I could not reproduce - mostliy because I do not know enough about LXC.

mirodin · August 1, 2020, 5:11am

Is there some place, I can find instructions on how to run Docker inside of a LXC container? Almost all I found was about LXD setups which I am not experienced enough to translate to LXC unfortunately.

simos · August 1, 2020, 6:01pm

I am not experienced in LXC to help you here. I can only offer some overview help.

LXC, LXD and Docker use features from the Linux kernel to run containers.
If you want to nest one inside another, then the outer needs to be able to support nesting.
LXD does support nesting (to work as the outer container) when you create a container with the security.nesting=true flag.
You need to figure out how to enable nesting in LXC, if such a thing is possible.

Then, Docker has specific needs for its filesystem. It would rather have direct access to a device or something similar. But with nesting, that is not the case. For the combination LXD (outer) with Docker (inner), it works better if the LXD storage pool is on btrfs rather than zfs. You will need to find some filesystem that will make Docker happy.

I hope this helps you.