vCenter Server Appliance: A Tale of Rejuvenation

For the past couple of years, I’ve been involved with a lot of Service Provider projects where the primary focus was/still on Public/Provider/Government cloud designs where vCloud Director (vCD) is the awesomeness that brings the entire stack together.

Almost every time when discussing the vCenter Server and its important role for both management and resource clusters the topic of vCenter Server recovery is brought forth and customers are interested in knowing about the process, the impact and what needs to be done.

In each vCloud Director environment you will mostly see:

• A vCenter Server managing the management cluster.
• A vCenter Server managing the resource (computer & edge) cluster/s.
• The vCenter Server applinace of the resource clusters/s is hosted on the management cluster.

The recovery options for both vCenter Server are:

• VM based backups.
• File-Level recovery backups via the out-of-the-box backup and recovery options.

Let’s consider the following:

• The management cluster has vDS or two configured with 2x10Gbps or 4x10Gbps uplinks.
• The resources vCenter Server is lost, and in this case, you should be able to restore the vCenter Server Appliance via one of the mentioned options above.
• The management vCenter Server is lost, and in this case, restoring the vCenter Server Appliance requires:
  • Disconnecting the management host you want to restore to from the vCenter Server.
  • Remove one of the vDS uplinks and configure a standard switch to be able to recover the virtual machine and the reason for this come because spinning a new port at the vDS level requires the vCenter Server.

Reality of things:

• I found that most of the customers are reluctant to touch on a management host in terms of vDS re-configuration for reasons related to SLA (availability might come into place here if the vDS has only two uplinks), performance, long change management processes.
• I also noticed that backup vendors in a lot of the cases they recommend a physical host that is configured with a standard switch to be available all the time for the purposes of management vCenter Server restoration.
• I found that customers are reluctant against getting a single host for the purpose of management vCenter Server restoration.

Options/Alternatives?

• ALTERNATIVE 1: Although it is something that is NOT supported (https://kb.vmware.com/s/article/2009916), an ideal design decision would be to have a nested ESXi host on the management cluster for the purpose of restoring the vCenter Server from backups:
  • You will need to set up a proxy appliance on the nested host in the event the backup solution required one.
  • You will need to provide the nested ESXi VM with enough memory and storage to accommodate backup proxy and the vCenter Server appliance restoration.
  • When the vCenter Server Appliance is restored you will obviously need to add the nested ESXi host to it and Storage+Compute vMotion the vCenter Server Appliance back to the management cluster.
  • Also, you can use the same nested ESXi host to do a file-level restore of the vCenter Sever in the event the VM backup fails for any reason.
• ALTERNATIVE 2:
  • Usually, it is not the case, but if you happen to have approval on restoring the management vCenter Server Appliance on the resources cluster and via the resources vCenter Server then you’ll be to do the restoration without the need for a nested ESXi host.
  • In this case, you will also need to allow the VM management VLAN to be tagged on the resources uplinks and you will need to allow vMotion between the management and resources hosts from a communication perspective so that to be able to migration the management vCenter Server to the management cluster.

The above alternative works and is reliable and can be configured part of day 0 activities and can be used as a means to test the vCenter Server appliance restore. Sadly, it is not supported, so don’t expect VMware’s GSS to help you in this process and I strongly urge staying away from putting the vCenter Server Appliance in production mode on that nested ESXi host beyond the restoration period.

So as soon as you have it restored and the services are up and running, the first thing you should do is migrate the appliance to the management production cluster.