LXD: Network problems with bridged network

Dear community

I’m pretty new to the LXD community and try to setup an LXD environment in a VM on my NAS but I encounter some strange network behaviours.

My LXD installation now runs in a Centos 7 VM and it’s installed through snap.

My idea ist to assign two ip addresses for some containers (e.g. HAProxy).
So I have two network interfaces for my container one from the private LXD subnet (lxdbr0) and one which I created with a bridge on my host machine (br0). I created the br0 interface using this tutorial and created a second lxd profile to assign this interface to eth1 of the containers.

[admin@VSRV04 ~]$ brctl show
bridge name bridge id STP enabled interfaces
br0 8000.0289a1532050 yes eth0
veth6732e271
vethe09cd369
vethe32bdcbb
lxdbr0 8000.0ae1324968e2 no veth2da124f4
veth37ca0142
veth9afba33d

[admin@VSRV04 ~]$ lxc profile show bridge
config: {}
description: Bridged networking LXD profile
devices:
eth1:
name: eth1
nictype: bridged
parent: br0
type: nic
name: bridge
used_by:

  • /1.0/instances/centos8
  • /1.0/instances/centos7
  • /1.0/instances/ubuntu

Now I encountered two issues.

  1. If i run lxc list to check my containers only one container has an assigned IPv4 address from the local network all other containers only has an IPv6 address on the eth1 interface. I found out that this has something to do with the installed distro. A Centos 8 container gets a correct IPv4 address from the local network and is accessible through this IP address. All other containers doesn’t get an IPv4 address from the local network (10.10.111.x). And thus are not able to be reached within the local network. The Centos 8 is the only container which has a local IP.

[admin@VSRV04 ~]$ lxc list
±--------±--------±---------------------±----------------------------------------------±----------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±--------±--------±---------------------±----------------------------------------------±----------±----------+
| centos7 | RUNNING | 10.10.122.97 (eth0) | fdde:9f9f:c813:0:216:3eff:feeb:86dd (eth1) | CONTAINER | 0 |
| | | | fd42:96b7:bbc8:8207:216:3eff:fe1b:fdb9 (eth0) | | |
| | | | 2a02:169:35f4:0:216:3eff:feeb:86dd (eth1) | | |
±--------±--------±---------------------±----------------------------------------------±----------±----------+
| centos8 | RUNNING | 10.10.122.198 (eth0) | fdde:9f9f:c813::9f7 (eth1) | CONTAINER | 0 |
| | | 10.10.111.119 (eth1) | fdde:9f9f:c813:0:78a9:c81c:84a0:c7cb (eth1) | | |
| | | | fd42:96b7:bbc8:8207:216:3eff:fe2e:3d89 (eth0) | | |
| | | | 2a02:169:35f4::9f7 (eth1) | | |
| | | | 2a02:169:35f4:0:36c9:cc6a:24fe:7d24 (eth1) | | |
±--------±--------±---------------------±----------------------------------------------±----------±----------+
| ubuntu | RUNNING | 10.10.122.115 (eth0) | fdde:9f9f:c813:0:216:3eff:fe10:f6c3 (eth1) | CONTAINER | 0 |
| | | | fd42:96b7:bbc8:8207:216:3eff:fe05:3d2e (eth0) | | |
| | | | 2a02:169:35f4:0:216:3eff:fe10:f6c3 (eth1) | | |
±--------±--------±---------------------±----------------------------------------------±----------±----------+

All of the containers have the same profiles assigned:

Centos 8 container:

[admin@VSRV04 ~]$ lxc info centos8
Name: centos8
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/05/02 12:19 UTC
Status: Running
Type: container
Profiles: default, bridge
Pid: 24491
Ips:
eth0: inet 10.10.122.198 veth2da124f4
eth0: inet6 fd42:96b7:bbc8:8207:216:3eff:fe2e:3d89 veth2da124f4
eth0: inet6 fe80::216:3eff:fe2e:3d89 veth2da124f4
eth1: inet 10.10.111.119 vethe09cd369
eth1: inet6 2a02:169:35f4::9f7 vethe09cd369
eth1: inet6 fdde:9f9f:c813::9f7 vethe09cd369
eth1: inet6 fdde:9f9f:c813:0:78a9:c81c:84a0:c7cb vethe09cd369
eth1: inet6 2a02:169:35f4:0:36c9:cc6a:24fe:7d24 vethe09cd369
eth1: inet6 fe80::a56f:fc0e:ec1a:6887 vethe09cd369
lo: inet 127.0.0.1
lo: inet6 ::1
Resources:
Processes: 226
CPU usage:
CPU usage (in seconds): 7
Memory usage:
Memory (current): 62.52MB
Memory (peak): 97.02MB
Swap (current): 4.10kB
Swap (peak): 4.10kB
Network usage:
eth0:
Bytes received: 4.13kB
Bytes sent: 2.16kB
Packets received: 41
Packets sent: 24
eth1:
Bytes received: 560.35kB
Bytes sent: 10.73kB
Packets received: 5052
Packets sent: 93
lo:
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0

Centos 7 container:

[admin@VSRV04 ~]$ lxc info centos7
Name: centos7
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/05/02 12:22 UTC
Status: Running
Type: container
Profiles: default, bridge
Pid: 26748
Ips:
eth0: inet 10.10.122.97 veth9afba33d
eth0: inet6 fd42:96b7:bbc8:8207:216:3eff:fe1b:fdb9 veth9afba33d
eth0: inet6 fe80::216:3eff:fe1b:fdb9 veth9afba33d
eth1: inet6 fdde:9f9f:c813:0:216:3eff:feeb:86dd veth6732e271
eth1: inet6 2a02:169:35f4:0:216:3eff:feeb:86dd veth6732e271
eth1: inet6 fe80::216:3eff:feeb:86dd veth6732e271
lo: inet 127.0.0.1
lo: inet6 ::1
Resources:
Processes: 11
CPU usage:
CPU usage (in seconds): 2
Memory usage:
Memory (current): 19.52MB
Memory (peak): 34.40MB
Swap (current): 32.77kB
Swap (peak): 32.77kB
Network usage:
eth0:
Bytes received: 3.86kB
Bytes sent: 1.65kB
Packets received: 37
Packets sent: 19
eth1:
Bytes received: 670.91kB
Bytes sent: 1.69kB
Packets received: 6013
Packets sent: 20
lo:
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0

My second problem is that this container which has a local ip address is not accessible via WAN (Internet).
If I point my no-ip dns to my public IP address and forward port 80 to the container I am able to enter the web address in the browser. If I am in the local network I’m able to reach the apache web server which is installed in the container without any issues. But if I try to reach the container from WAN (mobile phone, 4G hotspot or through VPN) I’m not able to reach the apache test server. I guess this has something to do with the iptables on the host machine since when I install a web server in docker on my NAS and forward port 80 to it’s IP it just works like it should so I guess it has something to do with the firewall on the host (I guess iptables).

[admin@VSRV04 ~]$ sudo iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp – lxdbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53 /* generated for LXD network lxdbr0 /
426 27960 ACCEPT udp – lxdbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 /
generated for LXD network lxdbr0 /
754 239K ACCEPT udp – lxdbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67 /
generated for LXD network lxdbr0 /
79600 256M ACCEPT all – * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
112 6904 ACCEPT all – lo * 0.0.0.0/0 0.0.0.0/0
1126K 113M INPUT_direct all – * * 0.0.0.0/0 0.0.0.0/0
1126K 113M INPUT_ZONES_SOURCE all – * * 0.0.0.0/0 0.0.0.0/0
1126K 113M INPUT_ZONES all – * * 0.0.0.0/0 0.0.0.0/0
9 1428 DROP all – * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
1126K 113M REJECT all – * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all – * lxdbr0 0.0.0.0/0 0.0.0.0/0 /
generated for LXD network lxdbr0 /
0 0 ACCEPT all – lxdbr0 * 0.0.0.0/0 0.0.0.0/0 /
generated for LXD network lxdbr0 /
0 0 ACCEPT all – * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all – lo * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_direct all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_IN_ZONES_SOURCE all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_IN_ZONES all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_OUT_ZONES_SOURCE all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FORWARD_OUT_ZONES all – * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP all – * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
0 0 REJECT all – * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 2412 packets, 174K bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp – * lxdbr0 0.0.0.0/0 0.0.0.0/0 tcp spt:53 /
generated for LXD network lxdbr0 /
419 67323 ACCEPT udp – * lxdbr0 0.0.0.0/0 0.0.0.0/0 udp spt:53 /
generated for LXD network lxdbr0 /
754 251K ACCEPT udp – * lxdbr0 0.0.0.0/0 0.0.0.0/0 udp spt:67 /
generated for LXD network lxdbr0 */
1616 2944K ACCEPT all – * lo 0.0.0.0/0 0.0.0.0/0
41675 3806K OUTPUT_direct all – * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD_IN_ZONES (1 references)
pkts bytes target prot opt in out source destination
0 0 FWDI_public all – vethb8e44c1a * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – veth4067c516 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – vethf645ba96 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – veth454ab4e5 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – veth88860e51 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – vethe3be3187 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – vethaac0ed08 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – vethb20daa51 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – eth0 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – br0 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_trusted all – lxdbr0 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDI_public all – + * 0.0.0.0/0 0.0.0.0/0 [goto]
Chain FORWARD_IN_ZONES_SOURCE (1 references)
pkts bytes target prot opt in out source destination
Chain FORWARD_OUT_ZONES (1 references)
pkts bytes target prot opt in out source destination
0 0 FWDO_public all – * vethb8e44c1a 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * veth4067c516 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * vethf645ba96 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * veth454ab4e5 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * veth88860e51 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * vethe3be3187 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * vethaac0ed08 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * vethb20daa51 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * eth0 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * br0 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_trusted all – * lxdbr0 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 FWDO_public all – * + 0.0.0.0/0 0.0.0.0/0 [goto]
Chain FORWARD_OUT_ZONES_SOURCE (1 references)
pkts bytes target prot opt in out source destination
Chain FORWARD_direct (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_public (11 references)
pkts bytes target prot opt in out source destination
0 0 FWDI_public_log all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDI_public_deny all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDI_public_allow all – * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT icmp – * * 0.0.0.0/0 0.0.0.0/0
Chain FWDI_public_allow (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_public_deny (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_public_log (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_trusted (1 references)
pkts bytes target prot opt in out source destination
0 0 FWDI_trusted_log all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDI_trusted_deny all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDI_trusted_allow all – * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all – * * 0.0.0.0/0 0.0.0.0/0
Chain FWDI_trusted_allow (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_trusted_deny (1 references)
pkts bytes target prot opt in out source destination
Chain FWDI_trusted_log (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_public (11 references)
pkts bytes target prot opt in out source destination
0 0 FWDO_public_log all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDO_public_deny all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDO_public_allow all – * * 0.0.0.0/0 0.0.0.0/0
Chain FWDO_public_allow (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_public_deny (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_public_log (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_trusted (1 references)
pkts bytes target prot opt in out source destination
0 0 FWDO_trusted_log all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDO_trusted_deny all – * * 0.0.0.0/0 0.0.0.0/0
0 0 FWDO_trusted_allow all – * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all – * * 0.0.0.0/0 0.0.0.0/0
Chain FWDO_trusted_allow (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_trusted_deny (1 references)
pkts bytes target prot opt in out source destination
Chain FWDO_trusted_log (1 references)
pkts bytes target prot opt in out source destination
Chain INPUT_ZONES (1 references)
pkts bytes target prot opt in out source destination
0 0 IN_public all – vethb8e44c1a * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – veth4067c516 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – vethf645ba96 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – veth454ab4e5 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – veth88860e51 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – vethe3be3187 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – vethaac0ed08 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – vethb20daa51 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – eth0 * 0.0.0.0/0 0.0.0.0/0 [goto]
1124K 113M IN_public all – br0 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_trusted all – lxdbr0 * 0.0.0.0/0 0.0.0.0/0 [goto]
0 0 IN_public all – + * 0.0.0.0/0 0.0.0.0/0 [goto]
Chain INPUT_ZONES_SOURCE (1 references)
pkts bytes target prot opt in out source destination
Chain INPUT_direct (1 references)
pkts bytes target prot opt in out source destination
Chain IN_public (11 references)
pkts bytes target prot opt in out source destination
1126K 113M IN_public_log all – * * 0.0.0.0/0 0.0.0.0/0
1126K 113M IN_public_deny all – * * 0.0.0.0/0 0.0.0.0/0
1126K 113M IN_public_allow all – * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT icmp – * * 0.0.0.0/0 0.0.0.0/0
Chain IN_public_allow (1 references)
pkts bytes target prot opt in out source destination
12 948 ACCEPT tcp – * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate NEW,UNTRACKED
1 64 ACCEPT tcp – * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:9090 ctstate NEW,UNTRACKED
Chain IN_public_deny (1 references)
pkts bytes target prot opt in out source destination
Chain IN_public_log (1 references)
pkts bytes target prot opt in out source destination
Chain IN_trusted (1 references)
pkts bytes target prot opt in out source destination
0 0 IN_trusted_log all – * * 0.0.0.0/0 0.0.0.0/0
0 0 IN_trusted_deny all – * * 0.0.0.0/0 0.0.0.0/0
0 0 IN_trusted_allow all – * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all – * * 0.0.0.0/0 0.0.0.0/0
Chain IN_trusted_allow (1 references)
pkts bytes target prot opt in out source destination
Chain IN_trusted_deny (1 references)
pkts bytes target prot opt in out source destination
Chain IN_trusted_log (1 references)
pkts bytes target prot opt in out source destination
Chain OUTPUT_direct (1 references)
pkts bytes target prot opt in out source destination

I would be very glad if somebody can point me in the right direction in order to get my LXD installation up and running. :slight_smile:

Thank you in advance and have a nice weekend!
Cheers,
Patrick

Have you considered using the proxy container device to forward a port on your host to the container?

https://linuxcontainers.org/lxd/docs/master/instances#type-proxy

Hi Thomas

Thank you for your reply. Yes I thought about the built-in proxy but I guess it would be a cleaner install with two network cards. But I will try out and let you know.

Thanks a lot for your input! :slight_smile:

Nice weekend to everybody! Suggestions are still very welcome! :nerd_face: