Is it possible to use the builtin LXC DHCP to give out valid local addresses for your LAN? Or, is it required to use an external DHCP server if you want those valid IPs?
It would need to be the only DHCP server on the bridged network.
In this case it would be. There would be no other DHCP servers on that subnet. How would I configure the ranges for the bridge and the DHCP scopes it gives out?
Is this for LXC or LXD btw?
LXD. It’s on Rocky Linux. I used this tutorial to get it going Creating a full LXD Server - Rocky Linux Documentation
I’ve changed this post’s category from LXC to LXD to make it clear it covers LXD.
Can you send me the output of lxc config show <instance> --expanded and the output of lxc network show <network>, as well as ip a and ip r from the LXD host and the instance.
Thanks
Sure.
architecture: x86_64
config:
image.architecture: amd64
image.description: Centos 8 amd64 (20210813_07:08)
image.os: Centos
image.release: "8"
image.serial: "20210813_07:08"
image.type: squashfs
image.variant: default
volatile.base_image: 8b170418fab377f120645b134f8ed1c53cf0e680fef17ca3168bffe88ab19634
volatile.eth0.host_name: mac26fec4b7
volatile.eth0.hwaddr: 00:16:3e:c6:c7:ea
volatile.eth0.last_state.created: "false"
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.power: RUNNING
volatile.uuid: 5531f6ce-0142-4a28-a46a-b4d2fdb8ec78
devices:
eth0:
name: eth0
nictype: macvlan
parent: ens192
type: nic
root:
path: /
pool: storage
type: disk
ephemeral: false
profiles:
- default
- macvlan
stateful: false
description: ""
config:
ipv4.address: 10.209.198.1/24
ipv4.nat: "true"
ipv6.address: none
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/ubuntu-test
- /1.0/profiles/default
managed: true
status: Created
locations:
- none
config: {}
description: ""
name: ens192
type: physical
used_by:
- /1.0/instances/centos-test
- /1.0/profiles/macvlan
managed: false
status: ""
locations: []
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:bf:ac:b1 brd ff:ff:ff:ff:ff:ff
inet 10.1.22.73/24 brd 10.1.22.255 scope global noprefixroute ens192
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:febf:adb0/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:61:43:b4 brd ff:ff:ff:ff:ff:ff
inet 10.209.198.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe61:43b4/64 scope link
valid_lft forever preferred_lft forever
7: veth30c0de5e@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 46:c5:8c:0f:1e:7e brd ff:ff:ff:ff:ff:ff link-netnsid 1
default via 10.1.22.1 dev ens192 proto static metric 100
10.209.198.0/24 dev lxdbr0 proto kernel scope link src 10.209.198.1
10.1.22.0/24 dev ens192 proto kernel scope link src 10.1.22.73 metric 100Your instance isn’t connected to the managed network lxdbr0 that would provide DHCP and DNS services. Do you want your instance connected to the lxdbr0 network, and in turn the lxdbr0 bridge be connected to the wider external network?
That is correct. I was messing around with macvlan with that instance, but I would like it to use the lxdbr0 bridge and connect that bridge to the larger network and use a DHCP range I define.
OK so this may be a little tricky.
When you want to connect an external interface to a Linux bridge (such as lxdbr0), you need to remove the IP addresses from that external interface (as they won’t work), and instead the bridge interface on your LXD host effectively becomes that external interface for the host (so lxdbr0 in this case).
This means you would need to ensure that the LXD lxdbr0 network is configured with the same address settings as ens192 has and then use the bridge network’s bridge.external_interfaces setting to specify that ens192 should be connected to lxdbr0.
This will then expose the lxdbr0 DHCP server to the external network.
The problem is that in order for your LXD host to have external internet access you need to keep that existing default route to 10.1.22.1, which if you move the addressing from ens192 to lxdbr0, won’t happen, as LXD doesn’t support setting up default gateway routes.
Out of interest, why do you need LXD’s DHCP server on the external network, rather than, say, running your own copy of dnsmasq listening on ens192?
Gotcha. There isn’t a requirement to use LXD’s DHCP server. I could just set it up on that host listening on that ens192, but I wasn’t sure if that would conflict with LXD since I know it runs its own internal instance of dnsmasq.
Do you want the LXD instances to use it too?
Yes
OK so I suggest setting up a new independent bridge, such as br0, connected to ens192, with the networking config from ens192 moved to br0. You can do this using netplan: Netplan | Backend-agnostic network configuration in YAML
Then setup dnsmasq DHCP to listen only on the br0 interface. This will then provide DHCP to your external network on ens192.
Finally, update your instances to connect to br0 rather than lxdbr0, such as:
lxc config device add <instance> eth0 nic nictype=bridged parent=br0
Do I need to keep the lxdbr0 device around? I tried to start dnsmasq and it errored because it looks like lxd is already listening on its bridge address on port 53
You don’t need to keep it around, but lxdbr0’s dnsmasq only listens on lxdbr0 interface, so if that is preventing you starting another dnsmasq, it means you new dnsmasq is misconfigured to try and listen on all interfaces.
Try setting interface=ens192 on the new dnsmasq config.
I was successfully able to get the DHCP working and see it assigned to the container. But, I can’t access that IP from outside the host (such as SSH into it). I disabled iptables on the host just to eliminate that. The host is a VM on VMware if that helps.
I also noticed the containers themselves cannot get out to the internet. Should their default gateway be the IP of the host machine or the actual default gateway for that subnet?
Now I’m really confused 
Are you trying to use dnsmasq running inside a VMware VM to provide DHCP to both the external network outside of the VM and the LXD instances running inside of it?
Does your network outside of the VM have a DHCP server? If so why not just use that?
Please can you provide a full statement of what you’re trying to achieve and your current network setup, as otherwise its hard to see the full picture and the advice might not be relevant.
Gotcha. I can clarify. I’m using a VMware VM to provide DHCP to ONLY the LXD instances. I don’t need it to provide DHCP outside of the VM. But the IPs it gives out must be valid on the local network. Currently there is no DHCP on that subnet outside of that VM. The end goal is that developers will SSH to these instances from their workstations to utilize them.