NSX ALB (Avi) BGP scaling using ECMP and RHI

    NSX ALB (ex Avi Networks) solution supports different methods for scaling inside data plane, providing required resources in throughput, high availability and scalability vertically and horizontally. In typical scale-out mechanism for virtual service, which btw can be manual or automatic setup, service engines (SE) are added to group and existing VIP (Virtual Service IP) is maintained (scale-in process is of course opposite one). Basically different options are available for these scaling methods, known as:
  • L2 scale-out mode - VIP always mapped to "primary" SE, which implies that "organization" of traffic and sending to others SE's in group is also responsibility of primary SE --> heavier operations and dependency on this node are possible. Return traffic can go directly from the secondary SE's via the Direct Secondary Return mode or via the primary SE (Tunnel mode). Direct Secondary Return (DSR) mode gives option to return traffic to use VIP as the source IP and the secondary SE’s MAC as the source MAC. The well-known "ARP Inspection" must be disabled in the network, i.e. the network layer should not inspect/block/learn the MAC of the VIP from these packets (possible in some specific OpenStack, Cisco ACI setups etc.), and
  • L3 scale-out using BGP - usage of BGP specific features like Route Health Injection (RHI) and Equal Cost multi-path (ECMP) for providing dynamic VS load balancing and scaling. DSR is not possible here because of fact that ARP process is done for the next-hop, which is the upstream BGP router neighbour, which in turn does the ECMP to individual SEs. The return traffic uses respective SE’s MAC as source MAC and VIP as source IP.
    Because of fact that L2 scaling is pretty much easy to test and implement, I would like to show you BGP setup for this purpose, using RHI and ECMP behind the scene. This type of setup is currently supported in following ecosystems - VMware, Linux server (bare metal) and OpenShift/Kubernetes - where last cloud type is not available above Avi v18. Main setup for lab-ing this up can be seen on image below:

Picture 1. Demo setup inside lab environment

In summary, setup is pretty much very simple:
- two BGP routers for neighborship (RIGHT and LEFT),
- NSX ALB setup with controllers and service engines in data plane - vSphere cloud integration created for test,
- Data network - for SE-BGP neighbour establishment and rest of data traffic - 192.168.102.0/24,
- VIP network - virtual services L3 information - 192.168.103.0/24,
- backend web server inside NSX-T segment, on own L3 segment.
Basic configuration involves fact that BGP session is established between SE's in vSphere cloud, with VIP route exchange as /32 host route out (RHI part) of NSX ALB domain. ECMP is included inside BGP support, with different vendor specific configurations in terms of maximum paths.
BGP networking configuration can be done through GUI or CLI, and inside GUI it's available under Infrastructure -> Cloud resources -> Routing -> BGP peering, with per configured cloud option:

Picture 2. BGP peering overview per configured cloud (Avi is using Quagga based daemon for BGP configuration and setup)

Picture 3. Routing options information inside BGP profile

Picture 4. BGP peers configuration inside profile

    Interesting option, beside others which are pretty much very clear, is Label field - which is available under routing and BGP peer configuration. This config gives an option for per-neighbour route announcement inside configuration (routes with same label values per BGP peer are appropriately announced).
    Now, if other BGP side is checked, in this example routers RIGHT and LEFT based on Cisco IOS system, results are confirmed and routes are showing inside routing table as host routes:

Picture 5. NSX ALB service engines acting as BGP neighbours

Picture 6. Virtual service IP shown as host route inside routing table

    In summary, BGP setup is having couple of limitations and you should not expect full BGP configuration possibilities, but from perspective of reachability to adequate virtual services everything is configurable and possible. L3 scale out is fully supported and for large scale services could be planned as the best option inside Customer infrastructure.
    At the end, I would like to mention very useful links with deeper explanations on above subject directly from Avi Networks portal:
BGP support for virtual services
Virtual services scaling and explanation

Comments

Post a Comment

Popular posts from this blog

NSX ALB LetsEncrypt with DNS-01 challenge - BIND example

    In some of my previous posts, mentioned here LetsEncrypt script integration and Script parameter usage , I explained how useful it can be for NSX advanced load balancing solutions to utilise this kind of approach for free and automatic certificate manipulation, especially in environments with large number of web services inside. This approach utilises HTTP-01 based challenge with LetsEncrypt systems and L7 HTTP/S virtual services on NSX ALB side.     Now, from some time ago there is enhancement in this area, developed by official Avi Networks devops page, in terms of using DNS-01 challenge also. I tried on-prem option using Bind DNS as server which works very well.     Steps are pretty much similar to HTTP-01 option, which can be summarised as following:      - Create L7 virtual service with publicly available FQDN - certificates resolved can be in both options RSA or ECDSA, as configured during creation;        - Do...
Read more

VMware SD WAN - multiple locations - LAN IP address space overlapping with NAT

Image
     Different scenarios are possible in terms of routing, NAT-ing and IP overlapping setups using VMware Velocloud SD WAN technology in Customer environments.      Recently I had an PoC with my Customer for the on-prem option with this VMware solution, where different use cases were interesting to show and demonstrate - one of them is something I would like to share and it relates to possibility of LAN-side NAT on Edge (branch) locations, with purpose to have IP overlapped on these setups. Next picture is showing typical Hub&Spoke setup where it can be possible to make this type of configuration: Picture 1. VMware SD WAN lab on-prem environment     Basically thing which needs to be accomplished is appropriate NAT solution for LANs on every branch Edge which are and needs to be the same (192.168.1.0/24 in this example) - as it is shown on Picture 1.      Honestly speaking, NAT is not one of so powerful things insi...
Read more

NSX-T Layer 2 bridging - scenarios & use cases

Image
     Layer 2 bridging is very useful feature of NSX-T, which provides connection to a VLAN backed port group or a device, such as a gateway, that resides outside of NSX-T DC environment. Useful scenarios, among others, are: Workload migration from VLAN-backed to NSX overlay segment, NSX-V to NSX-T migration in Customer environments, Security features leverage using NSX-T Gateway firewall etc. L2 bridging feature requires usage of Edge clusters and Edge Bridge profiles.      Deployments should consider different options, with most important scenarios, for implementation below (this covers Edge VM deployment option as typical use case): Edge VM on VSS portgroup  --> promiscuous and forged transmit on portgroup REQUIRED / ESXi host (with Edge VM) command " esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1 " REQUIRED / Active and Standby Edge VMs should be on different hosts, Edge VM on VDS 6.6 (or later) portgroup  -->...
Read more