Issue with adding Bridge on top of Bond in Centos7.5

Priit3344 · September 6, 2018, 12:17pm

Hi guys ! I’ve been reading this forum for a while and getting good ideas and tips- - thanks for that and respect of doing it.

I’ve been trying to solve my case for a while now and thought I might need discussion or help from here. Maybe someone else has been struggling with the similar issue.

Background story and spec.

Dell FX 610 server.
Operating system - Centos 7.5
Main interface - em1 and em2 which is bonded together to Bond0 interface. * miimon=100 and balance-abl.
em3 and em4 I’ve left for FCoE connection.

So everything is fine until I’m starting create and mange LXC containers. I believe its due to fact that I’m not able to configure Bridge connection properly or I don’t understand it much as needed yet. ( Im able to assigne IPs manually inside the containers and they all can talk to each other but as soon as I try to go outside the containers - ping bridge IP or Bond0 IP I get unreachable answer.) ( Outside the containers in host machine I have working networking and Im able to ping/ talk also with the created Bridge interface )

I’ve tried to set it up by manually and link it against to Bond0 or setting it after I do LXD init and I assigne correct IP for it with proper IPv4 details.

My end goal is to have running Centos 7.5 with working LXD/LXC and I can create containers by assigning static IPs for them manually.

I know there is not a lot detailed information here but maybe someone have had similar issue or knows what to seek or to keep an eye while I’m configuring bridge and containers.

Happy to answer questions.

simos · September 6, 2018, 9:24pm

When you perform the default configuration of LXD, does lxd init complete successfully? If not, what errors?

When you create a container, does it get an IPv4 address? IPv6?

Priit3344 · September 7, 2018, 5:22am

Hi !

Default LXD init works every time like a charm. Ive tried to create bridge while init offers it and configured bridge myself and later assigned it while doing initial lxd init.

Everything seems to work perfectly till I’m trying to establish connection from my container to outside world. Im not able to ping my bridge interface, host bond or anything from outside world ( yum update and so on ). The only thing that seems to work is ping or ssh to another LXC container running on the same host machine.

About the container IP. Well Im not aiming to get it from DHCP. I have reserved static ip for my new containers and listed it into my local NS. After doing init to my desired container I change eth0 configration manually and add static IP into it.

simos · September 7, 2018, 8:04am

Let’s look next into the iptables chains that LXD has enabled for you.
Here is how you get them:

iptables -L
iptables -L -t nat

The LXD chains will have a comment that refers to lxdbr0.

Try to figure out whether there are additional chains that interfere with the LXD chains.
Try to figure out whether the LXD chains are not what they should be.

Priit3344 · September 9, 2018, 2:48pm

Hi !

It seems I figured out my problem. 2 FCoE interfaces also recieved ipv4 address and NM somehow reprioritized traffic over my em1, em2 what was bonded together to bond0. After I excluded em3, em4 from NM and reinstalled centos I managed to get into state where I can ping my containers and I can ping my bridge and other interfaces from created container. Now I’m facying an issue that im not able to get my container to yum/ speak out of the host machine. Nameserver is not answering or any other host in this network.

Im trying to catch where it gets blocked with tcpdump and hopefully resolve this issue.

iptables -L -t nat shows . - X.72.8.0/22 is my assigned database network.

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
PREROUTING_direct  all  --  anywhere             anywhere
PREROUTING_ZONES_SOURCE  all  --  anywhere             anywhere
PREROUTING_ZONES  all  --  anywhere             anywhere

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
OUTPUT_direct  all  --  anywhere             anywhere

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  X.72.8.0/22       !X.72.8.0/22        /* generated for LXD network br0 */
POSTROUTING_direct  all  --  anywhere             anywhere
POSTROUTING_ZONES_SOURCE  all  --  anywhere             anywhere
POSTROUTING_ZONES  all  --  anywhere             anywhere

Chain OUTPUT_direct (1 references)
target     prot opt source               destination

Chain POSTROUTING_ZONES (1 references)
target     prot opt source               destination
POST_public  all  --  anywhere             anywhere            [goto]
POST_public  all  --  anywhere             anywhere            [goto]
POST_public  all  --  anywhere             anywhere            [goto]
POST_public  all  --  anywhere             anywhere            [goto]

Chain POSTROUTING_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain POSTROUTING_direct (1 references)
target     prot opt source               destination

Chain POST_public (4 references)
target     prot opt source               destination
POST_public_log  all  --  anywhere             anywhere
POST_public_deny  all  --  anywhere             anywhere
POST_public_allow  all  --  anywhere             anywhere

Chain POST_public_allow (1 references)
target     prot opt source               destination

Chain POST_public_deny (1 references)
target     prot opt source               destination

Chain POST_public_log (1 references)
target     prot opt source               destination

Chain PREROUTING_ZONES (1 references)
target     prot opt source               destination
PRE_public  all  --  anywhere             anywhere            [goto]
PRE_public  all  --  anywhere             anywhere            [goto]
PRE_public  all  --  anywhere             anywhere            [goto]
PRE_public  all  --  anywhere             anywhere            [goto]

Chain PREROUTING_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain PREROUTING_direct (1 references)
target     prot opt source               destination

Chain PRE_public (4 references)
target     prot opt source               destination
PRE_public_log  all  --  anywhere             anywhere
PRE_public_deny  all  --  anywhere             anywhere
PRE_public_allow  all  --  anywhere             anywhere

Chain PRE_public_allow (1 references)
target     prot opt source               destination

Chain PRE_public_deny (1 references)
target     prot opt source               destination

Chain PRE_public_log (1 references)
target     prot opt source               destination

Priit3344 · September 11, 2018, 6:34am

And back in state where my containers are not pinging. I just cant figure out whats the block at the moment.

I configured my br0 manually before init lxd. Assigned that br0 in init process.

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 X.72.11.254 0.0.0.0 UG 0 0 0 bond0
X.72.8.0 0.0.0.0 255.255.252.0 U 0 0 0 bond0
X.72.8.0 0.0.0.0 255.255.252.0 U 0 0 0 br0
F.254.0.0 0.0.0.0 255.255.0.0 U 1004 0 0 em3
F.254.0.0 0.0.0.0 255.255.0.0 U 1005 0 0 em4
F.254.0.0 0.0.0.0 255.255.0.0 U 1008 0 0 bond0
F.254.0.0 0.0.0.0 255.255.0.0 U 1009 0 0 br0

IPtables,

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
PREROUTING_direct all – anywhere anywhere
PREROUTING_ZONES_SOURCE all – anywhere anywhere
PREROUTING_ZONES all – anywhere anywhere

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
OUTPUT_direct all – anywhere anywhere

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
POSTROUTING_direct all – anywhere anywhere
POSTROUTING_ZONES_SOURCE all – anywhere anywhere
POSTROUTING_ZONES all – anywhere anywhere

Chain OUTPUT_direct (1 references)
target prot opt source destination

Chain POSTROUTING_ZONES (1 references)
target prot opt source destination
POST_public all – anywhere anywhere [goto]

Chain POSTROUTING_ZONES_SOURCE (1 references)
target prot opt source destination

Chain POSTROUTING_direct (1 references)
target prot opt source destination

Chain POST_public (1 references)
target prot opt source destination
POST_public_log all – anywhere anywhere
POST_public_deny all – anywhere anywhere
POST_public_allow all – anywhere anywhere

Chain POST_public_allow (1 references)
target prot opt source destination

Chain POST_public_deny (1 references)
target prot opt source destination

Chain POST_public_log (1 references)
target prot opt source destination

Chain PREROUTING_ZONES (1 references)
target prot opt source destination
PRE_public all – anywhere anywhere [goto]

Chain PREROUTING_ZONES_SOURCE (1 references)
target prot opt source destination

Chain PREROUTING_direct (1 references)
target prot opt source destination

Chain PRE_public (1 references)
target prot opt source destination
PRE_public_log all – anywhere anywhere
PRE_public_deny all – anywhere anywhere
PRE_public_allow all – anywhere anywhere

Chain PRE_public_allow (1 references)
target prot opt source destination

Chain PRE_public_deny (1 references)
target prot opt source destination

Chain PRE_public_log (1 references)
target prot opt source destination

I also disabled network manager, tho that might block the traffic but no.

bodleytunes · September 11, 2018, 11:58am

Hiya

Looks from your routing table you are trying to bridge physical interface bond0 and virtual br0 together so your containers appear on the physical LAN?

Are you able to ping the upstream gateway x.72.11.254 from the server (sourcing from bond0)?

If so then it could be that the bridging part isn’t working.

I only ever use ubuntu / debian so not well up on CentOS and redhat but i would normally edit /etc/network/interfaces and using openvswitch would create a bridge (br0) this would then have ports of bond0 which means it should bridge L2 from your physical bond, into the machines virtual bridge, so broadcast traffic / arp etc for that vlan will be switched/bridged inside and the containers will receive it as if they were plugged into the local LAN segment.

The other option is to route your containers or NAT on egress of bond0 but you would use a different subnet for that, if routed then upstream devices need to have routes to get back, if you don’t want the hassle of adding routes on your upstream switches/routers then you just nat masquerade the new subnet on egress of bond0.

I personally prefer routing over bridging always when possible. But its drilled into me being a Network engineer.

Also check the output of iptables -L -vt nat and iptables -L -vt just in case something going on there.

bodleytunes · September 11, 2018, 12:14pm

To bridge L2, with openvswitch would be something along the lines of this.
You would want to plumb your containers into br0 editing the config files making sure the parent bridge is br0.
I’m sure it can be translated to centos style config.

allow-ovs br0
iface br0 inet manual
  ovs_type OVSBridge
  ovs_ports vlanif_xyz bond0 

allow-br0 vlanif_xyz
iface vlanif_xyz inet static
  ovs_type OVSIntPort
  ovs_bridge br0
  address x.72.8.10
  netmask 255.255.255.0
  gateway x.72.11.254
  dns-nameservers  8.8.8.8 9.9.9.9

Priit3344 · September 19, 2018, 8:31am

Thx for your help and suggestions guys !

I managed to fix the issue.

Firstly I provisioned my host with bond0 interface into other subnet. Created sub-bond with correct database vlan tag. Also disabled NM for em3 and em4 which is used for FCoE.

vi /etc/sysconfig/network-scripts/ifcfg-bond0.202

DEVICE=bond0.202
VLAN=yes
BOOTPROTO=none
NM_CONTROLLED=no
BRIDGE=br0
MTU=1500

And after that created bridge

vi /etc/sysconfig/network-scripts/ifcfg-br0

[root@fx15-1-db priit.jurgenson]# cat /etc/sysconfig/network-scripts/ifcfg-br0
TYPE=“Bridge”
PROXY_METHOD=“none”
BROWSER_ONLY=“no”
BOOTPROTO=“none”
IPV4_FAILURE_FATAL=“no”
NAME=“br0”
DEVICE=“br0”
ONBOOT=“yes”
IPADDR=“X.72.8.170”
PREFIX=“Y”
GATEWAY=“X.72.11.254”
DEFROUTE=no
DELAY=0

source info and idea came from here - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-networkscripts-interfaces_network-bridge