VMware Cloud Director: Going Beyond 9 Edge Gateway Non-Distributed Routed Networks with NSX-T
For those who have worked with VMware Cloud Director and NSX-v, it is known that internal-routed interfaces are limited to “1” uplink and “9” downlinks. This is because the Edge Services Gateway is a virtual machine, and the number of virtual machine NICs is limited to 10 (vSphere constraint).
Now comparing this to NSX-T Data Center, the ESG functionality was split between Tier-0 and Tier-1 gateways, where the tenant Edge Gateway is now a Tier-1 gateway in the backend, and Tier-1 gateways run within the overlay. They’re distributed, so they do not have the same limitations compared to NSX-v since the dependency on the virtual machine aspect is no longer there, and by having a look at the configuration maximums (based on NSX 4.1.0) you can confirm that any Tier-1 gateway can have up to 1000 segments or service interfaces attached to it.
Based on the previous assumptions I went to the lab, I tested creating more than “9” non-distributed routed networks on a tenant Edge Gateway that is backed by a Tier-1 gateway (which creates a service interface on the Tier-1 gateway rather than a connected segment). I ended up with an error stating that more than 9 non-distributed networks is not allowed (similar to the NSX-v limitation).
This didn’t make much sense, and I asked internally about the reason for this limitation. After a few discussions, I learned about a configuration attribute that can be configured via the cell management tool to increase the number of non-distributed routed interfaces (thank you, Robert Meyer).
The configuration attribute is “networking.gatewayInterfaceCount“, and by listing the existing default value for it, it should return 10.
./cell-management-tool manage-config -n networking.gatewayInterfaceCount -l
I changed this value to “20” and then verified that the change was applied.
./cell-management-tool manage-config -n networking.gatewayInterfaceCount -v 20
To test this, I started creating networks until I reached the maximum count I set in the configuration.
It worked smoothly, and I realized the additional networks in VCD. Also, you can check the service interfaces count on the Tier-1 gateway.
I haven’t tested the effect of this on NSX-v (in the event there is still an on-going NSX-v to NSX-T migration), if I managed to do so, I’ll update the blog post accordingly, and I hope this will be helpful to someone :-).