NSX VXLAN Control Plane Replication Modes Design Decision

So in a recent customer engagement I kind of struggled a bit to provide the customer with definite guidelines on the selection of the VXLAN replication modes, and as such I snooped around and along with the great help of Waleed Akl I thought I would just write down my thoughts on this and hopefully this blog post will be updated with fresh ideas/decisions (based on what I face in the field and based on your comments below).

Please do note that the here-under is more related to :

1. NSX-v and not NSX-T (as with NSX-T there  only Unicast and Hybrid replication modes available).
2. What to select rather than a deep dive into each mode as you will find a lot of content around this topic over the internet.

In brief, NSX supports multiple replication modes to handle BUM (Broadcast, Unknown Unicast and Multicast) where in essence is to handle L2 traffic.

A technical definition on the 3 replication modes would be:

1. Unicast:
   1. All replication occurs using unicast.
   2. Underlay agnostic – L2 or L3
   3. Comes with ARP suppression.
2. Multicast:
   1. Entire replication is off-loaded to physical network
   2. Requires IGMP/Querier & and multicast routing for L3(PIM)
3. Hybrid:
   1. Local replication (L2 domain) offloaded to physical network, while remote replication occurs via unicast
   2. Most practical without the complexity of multicast mode
   3. Only requires IGMP Snooping/Querier. Does not require L3 PIM (but it is still recommended in some cases where it gets leveraged to be a requirement, depending on the setup and the underlying hardware).

From an architectural perspective I would have them defined as below:

1. Unicast:
   1. Totally decoupling physical from virtual.
   2. Used with small to medium sized implementations.
   3. No hardware complex configurations involved.
   4. Requires an NSX controller cluster.
   5. Scales via L3 which also enhances performance as well (each rack/cluster has its own VTEP subnet).
   6. Having VXLAN offloading on the NIC aligns well here.
2. Multicast:
   1. No decoupling of physical from virtual.
   2. No scale set for this mode to be honest.
   3. Requires an extra complex configuration at the physical layer.
   4. Requires an extra configuration on NSX (IP Multicast Range).
3. Hybrid:
   1. Combining both virtual and physical at the same time.
   2. Used with large scale sized implementations.
   3. Requires an NSX controller cluster.
   4. Requires an extra configuration at the physical layer.
   5. Scales well in a spine/leaf topology.

What to choose?

To be honest, if you come to an environment where all the above options are available and could be to a certain extent viable. How to make a choice? Especially when we have the scales of small, medium and large? What defines such a scale? If you go through the NSX design guide there are different metrics to say that this is a small,medium or large deployment and this decision could be based on a number of factors including vCenter, hosts and the underlay networking.

Waleed pointed me out to a VMworld session done by Shahzad Ali titled VMware NSX in Small Data Centers for Small and Medium Businesses and within it Shahzad had pulled two slides around that scale, where in brief:

1. Large Data Centers
   1. More than a 100 hosts, multiple clusters/pods (Management, Edge and Compute), thousands of VMs and more than 10Gbps requirements.
2. Medium Data Centers
   1. 10 to 100 hosts, collapsed management and edge clusters and separate clusters for compute, and presumably 10Gbps are sufficient here.
3. Small Data Centers
   1. 3 to 10 hosts, everything collapsed and again presumably 10Gbps could be used but might not be a requirement to cater for the workloads.

So where are we at?

With all the above mentioned, it should be easier to determine which replication mode should be used in addition I do believe that the replication mode  could also be figured out in-context and is not necessarily determined only based on scale and standard predefined design decisions.

If you have anything to add/adjust on the above please do comment below and I will be more than glad to add your notes to this blog post.

(Abdullah)^2