Incus OVN IPv6 route announcements to upstream not working

Incus
, ,
1

Hello,

I’ve setup a Incus cluster with 4 members, OVN configured and working and a FRR software router pair as redirection gateway to my other classical network infrastructure. My network is mainly dual-stack. IPv4 route announcements work normally, but no IPv6 announcements reach my router. I even tried using a IPv6 BGP Connection for v6 Announcements, but this connection also only transmits IPv4 routing information.

Attached see my configuration.

Thank you!

vms.txt (1.6 KB)

v201.txt (395 Bytes)

netconf.txt (901 Bytes)

net.txt (3.8 KB)

incconf.txt (277 Bytes)

2

It seems that the integrated BGP Server doesn’t support sending unicast IPv6 route announcements.

FRR gives me following information:

[...]
BGP version 4, remote router ID 10.0.100.171, local router ID 10.0.100.175
  BGP state = Established, up for 1d01h52m
  Last read 00:00:23, Last write 00:00:23
  Hold time is 90 seconds, keepalive interval is 30 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 1448
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised
    AddPath:
      IPv4 Unicast: RX advertised
    Paths-Limit:
      IPv4 Unicast: advertised (0)
    Extended nexthop: received
      Address families by peer:
                   IPv4 Unicast
    Long-lived Graceful Restart: advertised
    Route refresh: advertised and received
    Enhanced Route Refresh: advertised
    Address Family IPv4 Unicast: advertised and received
    Address Family IPv6 Unicast: received
    Hostname Capability: advertised (name: xyzabbgp01,domain name: n/a) received (name: xyzabinc01,domain name: n/a)
    Version Capability: advertised software version (FRRouting/10.3) received software version (GoBGP/3.37.0)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 3600 seconds
      Address families by peer:
        IPv4 Unicast(not preserved)
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received: IPv4 Unicast
[...]
3

We sure do IPv6 :slight_smile:

fw-wan01# show bgp summary 

IPv4 Unicast Summary:
BGP router identifier 45.45.148.249, local AS number 399760 VRF default vrf-id 0
BGP table version 298
RIB entries 51, using 6528 bytes of memory
Peers 8, using 188 KiB of memory
Peer groups 3, using 192 bytes of memory

Neighbor                            V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
2602:fc62:b:100::100                4     399760     56548     56293      298    0    0 00:03:21            8        0 GoBGP/3.37.0
2602:fc62:b:100::101                4     399760     56495     56237      298    0    0 00:03:22            8        0 GoBGP/3.37.0
2602:fc62:b:100::102                4     399760     56402     56281      298    0    0 00:03:18            5        0 GoBGP/3.37.0
45.45.148.251                       4     399760   1687833   1687943      298    0    0 02w5d12h            1        8 FRRouting/10.5.0
45.45.148.252                       4     399760   1687827   1687943      298    0    0 02w5d12h            1        8 FRRouting/10.5.0
45.45.148.253                       4     399760   1688641   1687940      298    0    0 02w5d12h            1        8 FRRouting/10.5.0
45.45.148.254                       4     399760   1688571   1687943      298    0    0 02w5d12h            1        8 FRRouting/10.5.0
2602:fc62:c:1000:fa5:c146:6375:ef4d 4     399760     56285     56278      298    0    0 01w5d01h            0        0 GoBGP/3.37.0

Total number of neighbors 8

IPv6 Unicast Summary:
BGP router identifier 45.45.148.249, local AS number 399760 VRF default vrf-id 0
BGP table version 577
RIB entries 84, using 10752 bytes of memory
Peers 8, using 188 KiB of memory
Peer groups 3, using 192 bytes of memory

Neighbor                            V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
2602:fc62:b:100::100                4     399760     56548     56293      577    0    0 00:03:21           15        0 GoBGP/3.37.0
2602:fc62:b:100::101                4     399760     56495     56237      577    0    0 00:03:22           21        0 GoBGP/3.37.0
2602:fc62:b:100::102                4     399760     56402     56281      577    0    0 00:03:18           13        0 GoBGP/3.37.0
2602:fc62:b:101::251                4     399760   1687833   1688045      577    0    0 02w5d12h            1       41 FRRouting/10.5.0
2602:fc62:b:101::252                4     399760   1687827   1688045      577    0    0 02w5d12h            1       41 FRRouting/10.5.0
2602:fc62:b:101::253                4     399760   1688642   1688040      577    0    0 02w5d12h            1       41 FRRouting/10.5.0
2602:fc62:b:101::254                4     399760   1688571   1688045      577    0    0 02w5d12h            1       41 FRRouting/10.5.0
2602:fc62:c:1000:fa5:c146:6375:ef4d 4     399760     56285     56278      577    0    0 01w5d01h            1        0 GoBGP/3.37.0

Total number of neighbors 8
fw-wan01# show bgp neighbors 2602:fc62:b:100::100
BGP neighbor is 2602:fc62:b:100::100, remote AS 399760, local AS 399760, internal link
  Local Role: undefined
  Remote Role: undefined
Hostname: chulak
 Member of peer-group INCUS for session parameters
  BGP version 4, remote router ID 10.0.0.100, local router ID 45.45.148.249
  BGP state = Established, up for 00:03:34
  Last read 00:00:04, Last write 00:00:04
  Hold time is 90 seconds, keepalive interval is 30 seconds
  Configured hold time is 180 seconds, keepalive interval is 60 seconds
  Configured tcp-mss is 0, synced tcp-mss is 8928
  Configured conditional advertisements interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    Extended Message: advertised
    AddPath:
      IPv4 Unicast: RX advertised
      IPv6 Unicast: RX advertised
    Paths-Limit:
      IPv4 Unicast: advertised (0)
      IPv6 Unicast: advertised (0)
    Extended nexthop: received
      Address families by peer:
                   IPv4 Unicast
    Long-lived Graceful Restart: advertised
    Route refresh: advertised and received
    Enhanced Route Refresh: advertised
    Address Family IPv4 Unicast: advertised and received
    Address Family IPv6 Unicast: advertised and received
    Hostname Capability: advertised (name: fw-wan01,domain name: n/a) received (name: chulak,domain name: n/a)
    Version Capability: advertised software version (FRRouting/10.5.0) received software version (GoBGP/3.37.0)
    Link-Local Next Hop Capability: not advertised not received
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 3600 seconds
      Address families by peer:
            Graceful Restart Capability: advertised and received
      Remote Restart timer is 3600 seconds
      Address families by peer:
        IPv4 Unicast(not preserved), IPv6 Unicast(not preserved)
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast, IPv6 Unicast
    End-of-RIB received: IPv4 Unicast, IPv6 Unicast
    Local GR Mode: Helper*
    Remote GR Mode: Restart

    R bit: False
    N bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 3600
      Configured LLGR Stale Path Time(sec): 0
    IPv4 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
        LLGR Stale Path Time(sec): 0
    IPv6 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
        LLGR Stale Path Time(sec): 0
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  9          9
    Notifications:          0          0
    Updates:               18        273
    Keepalives:         56267      56267
    Route Refresh:          0          0
    Capability:             0          0
    Total:              56294      56549

  Prefix statistics:
    Inbound filtered: 0
    AS-PATH loop: 0
    Originator loop: 0
    Cluster loop: 0
    Invalid next-hop: 0
    Withdrawn: 0
    Attributes discarded: 0

  Minimum time between advertisement runs is 0 seconds
  Update delay timer is 0 seconds (remaining: 0)

 For address family: IPv4 Unicast
  INCUS peer-group member
  Update group 15, subgroup 51
  Packet Queue length 0
  Route-Reflector Client
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  Outbound path policy configured
  Outgoing update prefix filter list is *bgp-nothing
  8 accepted, 0 sent prefixes

 For address family: IPv6 Unicast
  INCUS peer-group member
  Update group 16, subgroup 52
  Packet Queue length 0
  Route-Reflector Client
  Inbound soft reconfiguration allowed
  Community attribute sent to this neighbor(all)
  Outbound path policy configured
  Outgoing update prefix filter list is *bgp-nothing
  15 accepted, 0 sent prefixes

  Connections established 9; dropped 8
  Last reset 00:03:39,  NSF peer closed the session (GoBGP/3.37.0)
  Internal BGP neighbor may be up to 255 hops away.
Local host: 2602:fc62:b:100::2, Local port: 179
Foreign host: 2602:fc62:b:100::100, Foreign port: 32959
Nexthop: 45.45.148.249
Nexthop global: 2602:fc62:b:100::2
Nexthop local: fe80::216:3eff:fe91:87bc
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 30
Estimated round trip time: 2 ms
Read thread: on  Write thread: on  FD used: 27


fw-wan01# show bgp neighbor 2602:fc62:b:100::100 received-routes 
BGP table version is 577, local router ID is 45.45.148.249, vrf id 0
Default local pref 100, local AS 399760
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 2602:fc62:b:1::1/128
                    2602:fc62:b:200::101
                                                    100      0 i
 *> 2602:fc62:b:1::3/128
                    2602:fc62:b:200::105
                                                    100      0 i
 *> 2602:fc62:b:10::1/128
                    2602:fc62:b:200::101
                                                    100      0 i
 *> 2602:fc62:b:10::7/128
                    2602:fc62:b:200::105
                                                    100      0 i
 *> 2602:fc62:b:8000::/64
                    2602:fc62:b:200::100
                                                    100      0 i
 *> 2602:fc62:b:8001::/64
                    2602:fc62:b:200::101
                                                    100      0 i
 *> 2602:fc62:b:8002::/64
                    2602:fc62:b:200::102
                                                    100      0 i
 *> 2602:fc62:b:8003::/64
                    2602:fc62:b:200::103
                                                    100      0 i
 *> 2602:fc62:b:8004::/64
                    2602:fc62:b:200::104
                                                    100      0 i
 *> 2602:fc62:b:8005::/64
                    2602:fc62:b:200::105
                                                    100      0 i
 *> 2602:fc62:b:8006::/64
                    2602:fc62:b:200::106
                                                    100      0 i
 *> 2602:fc62:b:8007::/64
                    2602:fc62:b:200::10e
                                                    100      0 i
 *> 2602:fc62:b:8008::/64
                    2602:fc62:b:200::108
                                                    100      0 i
 *> 2602:fc62:b:8009::/64
                    2602:fc62:b:200::10f
                                                    100      0 i
 *> 2602:fc62:fe:1000::/64
                    2602:fc62:b:200::10c
                                                    100      0 i

Total number of prefixes 15
fw-wan01# show ip bgp neighbor 2602:fc62:b:100::100 received-routes 
BGP table version is 298, local router ID is 45.45.148.249, vrf id 0
Default local pref 100, local AS 399760
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 45.45.148.226/32 172.17.200.111                100      0 i
 *> 45.45.148.227/32 172.17.200.111                100      0 i
 *> 45.45.148.228/32 172.17.200.118                100      0 i
 *> 45.45.148.240/32 172.17.200.101                100      0 i
 *> 45.45.148.242/32 172.17.200.105                100      0 i
 *> 172.17.20.10/32  172.17.200.101                100      0 i
 *> 172.17.20.12/32  172.17.200.105                100      0 i
 *> 172.17.20.13/32  172.17.200.105                100      0 i

Total number of prefixes 8
fw-wan01#
4

Hello Stéphane,

thank you for your reply. Do you have any recommendations on what to check to get the routing up and running?

My setup is running each node on a single NIC and using a unmanaged bridged network (br0). Unfortunately documentation for creating such configuration isn’t really available. So it’s mainly trial and error. But at the moment I‘m pretty much stuck.

If you need additional information just let me know.

Mario

5

You can run incus query /internal/debug/bgp to see what’s currently being advertised by Incus.

6

I’m getting this output with one network created:

# incus query /internal/debug/bgp
{
	"peers": [
		{
			"address": "2001:xxxx:yyyy:100::1750",
			"asn": 65001,
			"count": 1,
			"holdtime": 0,
			"password": ""
		},
		{
			"address": "2001:xxxx:yyyy:100::1760",
			"asn": 65001,
			"count": 1,
			"holdtime": 0,
			"password": ""
		}
	],
	"prefixes": [
		{
			"nexthop": "10.0.100.4",
			"owner": "network_6",
			"prefix": "10.10.201.0/24"
		},
		{
			"nexthop": "2001:xxxx:yyyy:100::4",
			"owner": "network_6",
			"prefix": "2001:xxxx:yyyy:a201::/64"
		}
	],
	"server": {
		"address": "2001:xxxx:yyyy:100::1710",
		"asn": 65001,
		"router_id": "10.0.100.171",
		"running": true
	}
}

My FRR config:

# sho run
Building configuration...

Current configuration:
!
frr version 10.3
frr defaults traditional
hostname svl0tbgp01
log syslog informational
service integrated-vtysh-config
!
ipv6 prefix-list OVN_SUBNETS seq 5 permit 2001:xxxx:yyyy:a000::/52
!
router bgp 65001
 bgp router-id 10.0.100.175
 neighbor 2001:xxxx:yyyy:100::1710 remote-as 65001
 neighbor 2001:xxxx:yyyy:100::1720 remote-as 65001
 neighbor 2001:xxxx:yyyy:100::1730 remote-as 65001
 neighbor 2001:xxxx:yyyy:100::1740 remote-as 65001
 !
 address-family ipv4 unicast
  maximum-paths 4
 exit-address-family
 !
 address-family ipv6 unicast
  maximum-paths 4
 exit-address-family
exit
!
end
7

What do you get under:

  • show ip bgp neighbor XXXX received-routes wide (ipv4)
  • show bgp neighbor XXXX received-routes wide (ipv6)

For each of your 4 peers.

8

I’m getting following:

bgp01# sho ip bgp neighbors 2001:xxxx:yyyy:100::1710 received-routes wide 
BGP table version is 6, local router ID is 10.0.100.175, vrf id 0
Default local pref 100, local AS 65001
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network                                      Next Hop                                  Metric LocPrf Weight Path
 *> 10.10.201.0/24                               10.0.100.4                                          100      0 i
 *> 10.10.202.0/24                               10.0.100.5                                          100      0 i

Total number of prefixes 2
bgp01# sho ip bgp neighbors 2001:xxxx:yyyy:100::1720 received-routes wide 
BGP table version is 6, local router ID is 10.0.100.175, vrf id 0
Default local pref 100, local AS 65001
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network                                      Next Hop                                  Metric LocPrf Weight Path
 *> 10.10.201.0/24                               10.0.100.4                                          100      0 i
 *> 10.10.202.0/24                               10.0.100.5                                          100      0 i

Total number of prefixes 2
bgp01# sho ip bgp neighbors 2001:xxxx:yyyy:100::1730 received-routes wide 
BGP table version is 6, local router ID is 10.0.100.175, vrf id 0
Default local pref 100, local AS 65001
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network                                      Next Hop                                  Metric LocPrf Weight Path
 *> 10.10.201.0/24                               10.0.100.4                                          100      0 i
 *> 10.10.202.0/24                               10.0.100.5                                          100      0 i

Total number of prefixes 2
svl0tbgp01# sho ip bgp neighbors 2001:xxxx:yyyy:100::1740 received-routes wide 
BGP table version is 6, local router ID is 10.0.100.175, vrf id 0
Default local pref 100, local AS 65001
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network                                      Next Hop                                  Metric LocPrf Weight Path
 *> 10.10.201.0/24                               10.0.100.4                                          100      0 i
 *> 10.10.202.0/24                               10.0.100.5                                          100      0 i

Total number of prefixes 2
svl0tbgp01# sho bgp neighbors 2001:xxxx:yyyy:100::1710 received-routes wide 
% No such neighbor or address family
svl0tbgp01# sho bgp neighbors 2001:xxxx:yyyy:100::1720 received-routes wide 
% No such neighbor or address family
svl0tbgp01# sho bgp neighbors 2001:xxxx:yyyy:100::1730 received-routes wide 
% No such neighbor or address family
svl0tbgp01# sho bgp neighbors 2001:xxxx:yyyy:100::1740 received-routes wide 
% No such neighbor or address family

What caught my attention was that FRR shows some errors in the logs but I can’t really figure out why they occur and setting the log verbosity higher didn’t give me more information:

Dec 12 19:38:02 svl0tbgp01 zebra[1531]: [V98V0-MTWPF] client 46 says hello and bids fair to announce only bgp routes vrf=0
Dec 12 19:38:02 svl0tbgp01 bgpd[1538]: [H4B4J-DCW2R][EC 33554455] 2001:xxxx:yyyy:100::1720 [Error] bgp_read_packet error: Connection reset by peer
Dec 12 19:38:02 svl0tbgp01 bgpd[1538]: [H4B4J-DCW2R][EC 33554455] 2001:xxxx:yyyy:100::1730 [Error] bgp_read_packet error: Connection reset by peer
Dec 12 19:38:02 svl0tbgp01 bgpd[1538]: [H4B4J-DCW2R][EC 33554455] 2001:xxxx:yyyy:100::1710 [Error] bgp_read_packet error: Connection reset by peer
Dec 12 19:38:02 svl0tbgp01 bgpd[1538]: [H4B4J-DCW2R][EC 33554455] 2001:xxxx:yyyy:100::1740 [Error] bgp_read_packet error: Connection reset by peer
Dec 12 19:38:07 svl0tbgp01 bgpd[1538]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 2001:xxxx:yyyy:100::1730 in vrf default
Dec 12 19:38:07 svl0tbgp01 bgpd[1538]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 2001:xxxx:yyyy:100::1720 in vrf default
Dec 12 19:38:07 svl0tbgp01 bgpd[1538]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 2001:xxxx:yyyy:100::1740 in vrf default
Dec 12 19:38:07 svl0tbgp01 bgpd[1538]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 2001:xxxx:yyyy:100::1710 in vrf default
Dec 12 22:33:44 svl0tbgp01 bgpd[1538]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 2001:xxxx:yyyy:100::1710 in vrf default
Dec 12 22:34:14 svl0tbgp01 bgpd[1538]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 2001:xxxx:yyyy:100::1720 in vrf default
Dec 12 22:34:51 svl0tbgp01 bgpd[1538]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 2001:xxxx:yyyy:100::1730 in vrf default
Dec 12 22:35:26 svl0tbgp01 bgpd[1538]: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from 2001:xxxx:yyyy:100::1740 in vrf default
9

Try putting neighbor XYZ activate in both the address-family sections.

10

Yes!!

That did the trick. I also added

neighbor 2001:xxxx:yyyy:100::<each IP of the Incus Servers> soft-reconfiguration inbound

to all of the address family ipv4 and ipv6 unicast sections.

Final config of FRR:

svl0tbgp01# sho run
Building configuration...

Current configuration:
!
frr version 10.3
frr defaults traditional
hostname svl0tbgp01
log syslog
service integrated-vtysh-config
!
ipv6 prefix-list OVN_SUBNETS seq 5 permit 2001:xxxx:yyyy:a000::/52
!
router bgp 65001
 bgp router-id 10.0.100.175
 neighbor 2001:xxxx:yyyy:100::1710 remote-as 65001
 neighbor 2001:xxxx:yyyy:100::1720 remote-as 65001
 neighbor 2001:xxxx:yyyy:100::1730 remote-as 65001
 neighbor 2001:xxxx:yyyy:100::1740 remote-as 65001
 !
 address-family ipv4 unicast
  neighbor 2001:xxxx:yyyy:100::1710 soft-reconfiguration inbound
  neighbor 2001:xxxx:yyyy:100::1720 soft-reconfiguration inbound
  neighbor 2001:xxxx:yyyy:100::1730 soft-reconfiguration inbound
  neighbor 2001:xxxx:yyyy:100::1740 soft-reconfiguration inbound
  maximum-paths 4
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor 2001:xxxx:yyyy:100::1710 activate
  neighbor 2001:xxxx:yyyy:100::1710 soft-reconfiguration inbound
  neighbor 2001:xxxx:yyyy:100::1720 activate
  neighbor 2001:xxxx:yyyy:100::1720 soft-reconfiguration inbound
  neighbor 2001:xxxx:yyyy:100::1730 activate
  neighbor 2001:xxxx:yyyy:100::1730 soft-reconfiguration inbound
  neighbor 2001:xxxx:yyyy:100::1740 activate
  neighbor 2001:xxxx:yyyy:100::1740 soft-reconfiguration inbound
  maximum-paths 4
 exit-address-family
exit
!
end

Thanks again!!!