Nesting vSphere vDS on NSX-T N-VDS

Since I moved everything to NSX-T at the physical layer, I am now in the process of building an NSX-v nested environment so that still cater to the different projects that have to do with NSX-v as a networking integration point.

This is how the VLANs/Segments are pieced together:

• A VLAN for ESXi management (used for both physical and nested hosts).
• A Segment for the nested VTEP/TEP.
• A Segment for the vCD stack (direct and nested where this will be mainly used for a single NSX-v Controller and ESG/s).

Bringing up the environment went smooth until I reached a point where I had to deploy the NSX-v Controller, this is when things went south and I could not attain connectivity.

Taking (from experience) how it has been with nested environments before NSX-T, where the vSphere vDS was the main port-group configuration, we used to reconfigure the security parameters to accept:

• Promiscuous mode.
• MAC address changes.
• Forged transmits.

When it comes to the N-VDS segment that is pushed to show in the vCenter Server Networking, there is no possibility of applying any of the vSphere vDS settings, so I started to look at other ways to see if these settings exist in the first place and then see how to change them.

• I attempted to play around in the segment profiles and no break-throughs there.
• I did a quick search on the topic “NSX-T Promiscuous mode” and most of what I found where related to how to nest an NSX-T deployment.
• Then I started looking at the same search string but added CLI to it, one of the interesting articles I came across was this “How-to: Configure VMware NSX-T for high performance networks on VMware NFV Platform with Mellanox supported adapters“, reading this article helped in getting to the next step of resolution where I got to meet my new friend “nsxdp-cli“.
• Again, back to the search engine, I did a quick search on nsxdp-cli documentation but I couldn’t find documentation similar to the nsxcli reference and I also came across this “vCloud NFV 3.0 Performance Tuning” where nsxdp-cli was referenced, so at this point in time it was back to doing things as normal geeks do! Trial and Error! RIGHT!

Usage: nsxdp-cli [OPTIONS] COMMAND [ARGS]...

  Command line interface to access settings on NSX datapath

Options:
  -h, --help  Show this message and exit.

Commands:
  bfd          Command line interface to access bfd module.
  discovery    Commands to access discovery protocol related...
  ens          Command line interface to access Enhanced...
  fc           Commands to access Flow Cache related...
  healthcheck  Commands to access health check related...
  ipfix        Command line interface to access ipfix...
  kcp          Command line interface to access kcp module.
  lacp         Commands to access LACP related features on a...
  netdvs       Commands to provide netdvs compatible output
  netsched     Commands to access net scheduler related...
  spf          Network service module CLI.
  swsec        Commands to access switch security related...
  vdl2         Command line interface to access vdl2 module.
  vswitch      Command line interface to access settings on...

With this new discovery, and like I always do when I am across new things, I stray from what I was supposed to do and took my time exploring the nsxdp-cli which I presume is short for “NSX DataPlane Command Line Interface“.

Long story short and conveniently (the last option in the list *face palm*) was the vswitch and basically this is what I needed to be looking at.

nsxdp-cli vswitch
Usage: nsxdp-cli vswitch [OPTIONS] COMMAND [ARGS]...

  Command line interface to access settings on a vswitch.

Options:
  -h, --help  Show this message and exit.

Commands:
  dvmirror      Commands to access DVMirror configuration on...
  dvport        Commands to access DVPort related features on...
  eth_filter    Commands to access extra ether filter related...
  instance      Commands to create, destroy and list vswitch...
  l2sec         Commands to access port L2 security related...
  mac-learning  Commands to access MAC Learning related...
  mac-table     Commands to access MAC Table associated with...
  mcast_filter  Commands to access multicast filter related...
  mtu           Commands to access MTU related features on a...
  promisc       Commands to access promiscuous sessions...
  rarp          Commands to access RARP related features on a...
  runtime       Commands to access configurable runtime...
  sink          Commands to access sink port related features...
  svcport       Commands to access service port related...
  teaming       Commands to access teaming related features...
  uplink        Commands to access uplink related features on...
  vlan          Commands to access VLAN related features on a...

Reading through the different sub-options, this caught my attention “l2sec -> Commands to access port L2 security related“, an AHA moment indeed %).

So, let’s dig deeper!

 nsxdp-cli vswitch l2sec --help
Usage: nsxdp-cli vswitch l2sec [OPTIONS] COMMAND [ARGS]...

  Commands to access port L2 security related features on a vswitch.

Options:
  -h, --help  Show this message and exit.

Commands:
  get  Commands to retrieve L2 security settings of...
  set  Commands to configure L2 security settings of...

Two options, GET and SET, so what is needed?

nsxdp-cli vswitch l2sec get --help
Usage: nsxdp-cli vswitch l2sec get [OPTIONS]

  Commands to retrieve L2 security settings of a port on vswitch

Options:
  -p, --dvport TEXT       DVPort ID  [required]
  -dvs, --dvs-alias TEXT  DVS alias name  [required]
  -j, --json-output       Output in JSON format.
  -h, --help              Show this message and exit.

Based on the above, we need the DVPort ID and DVS Name as mandatory requirements, I know that the ports can be retrieved via “nsxcli -> get ports | or nsxcli -c get ports” and this is issued on the ESXi host itself and I already know the name of the N-VDS.

The output from nsxcli -c get ports:

PortNum     Switch                Client                          DVSPort                         MAC                   Uplink
==================================================================================================================================
50331676    DvsPortset-0          VCF-ESXi01.FQDN.eth  45b10ecd-41a8-4fac-ab57-d703c4  00:50:56:ad:29:f3     vmnic6
                                  4                               e07d01
50331677    DvsPortset-0          VCF-ESXi01.FQDN.eth  8f8ec266-2340-4fb7-8efa-4717b6  00:50:56:ad:bf:04     vmnic6
                                  5                               42d356
50331690    DvsPortset-0          VCF-ESXi02.FQDN.eth  018133bc-abdb-4d53-ac45-0eb88f  00:50:56:ad:66:c3     vmnic7
                                  4                               0ba00b
50331691    DvsPortset-0          VCF-ESXi02.FQDN.eth  3aa627b2-a9b9-4ae7-a898-d08934  00:50:56:ad:db:66     vmnic7

Since the nested ESXi host has multiple virtual NICs attached to them, two ways to be able to filter the needed ones that are related to NSX-T’s N-VDS:

• Look at the MAC Addresses of the VMs and match them.
• Simply look at the uplink (in my case its vmnic6 and vmnic7) that are configured for the transport zone uplink profile.

Now let us issue the nsxdp-cli command(get):

nsxdp-cli vswitch l2sec get --dvport 45b10ecd-41a8-4fac-ab57-d703c4e07d01 -dvs Global-NVDS
deny promiscuous
deny mac change
deny forged frames

You’ve guessed it, another AHA! moment %), and next is obviously the set option.

nsxdp-cli vswitch l2sec set [OPTIONS]

  Commands to configure L2 security settings of a port on vswitch

Options:
  --promisc / --no-promisc        Allow promiscuous mode to be enabled on the
                                  DVPort
  --mac-change / --no-mac-change  Allow changing MAC address on the DVPort
  --forge-src / --no-forge-src    Allow forge TX frames from the DVPort
  -p, --dvport TEXT               DVPort ID  [required]
  -dvs, --dvs-alias TEXT          DVS alias name  [required]
  -h, --help                      Show this message and exit.

We also need here the DVPort ID and the DVS Name, in addition to specifying the needed options “promisc and mac-change and orge-src“.

nsxdp-cli vswitch l2sec set --dvport 3aa627b2-a9b9-4ae7-a898-d0893412323c -dvs Global-NVDS --mac-change --forge-src --promisc

Results (get):

nsxdp-cli vswitch l2sec get --dvport 3aa627b2-a9b9-4ae7-a898-d0893412323c -dvs Global-NVDS
allow promiscuous
allow mac change
allow forged frames

Moments of truth, ping started to reply and MISSION ACCOMPLISHED! On the other hand, it doesn’t end here as I wanted to test this configuration further.

I vMotioned the nested ESXi host to another physical host, and based on the results, the nsxdp-cli configuration doesn’t stick and it needs to be re-applied if the VM moves between hosts, although the port number is still the same.

I haven’t been able to find a way to make this configuration stick in, looked into the CLI of NSX Manager and I didn’t find much related to L2Security and I took a look at the “NSX-T 2.5 API Guide” and nothing there as well.

In conclusion, this achieves the intended results and I was able to continue with the nesting paradigm with all connectivity established.