VMware VSA Export VMs from broken appliances

So the title sounds disastrous and its a fact that I had recently faced the issue at one of our customers where one of the VSA ESXi nodes failed due to hardware issues and the second VSA node didn’t export the replica datastore and instead it went into maintenance mode:

Even when we got the faulty node up the status of the VSA appliances remained the same and both were in maintenance mode, whenever you try to exit them from maintenance mode (using both GUI and CLI) the VSA appliances will reboot around 5 times and will get themselves back into maintenance mode.

After calling VMware support, it seems that the database on both VSA appliances was corrupted so the logical question was to ask them if there is away to just export the NFS datastore so that we can retrieve the virtual machines, finally after jumping through a couple of engineers someone from the escalation team gave me what I want, and I thought I would share it here because it WILL help you a lot in case you got stuck in such a situation.

Lets do this:

1. First log-on to any of the ESXi hosts and power off one of the VSA appliances (only one appliance should be present).
2. Open the console and logon to that VSA appliance using svaadmin:svapass.
3. Change into super user: sudo su –
4. Type: vgchange -ay ( you should get something like [2 logical volume(s) in volume group “SVAVolGroup00” now active]).
5. Type: ls /dev/SVAVolGroup00/ (you should get something like [V0000000D0000001  V0000001D0000000] where these are the split datastore).
6. MDADM[1]: This is the Linux utility to manage the RAID array as such type (the result will include MD_LEVEL, MD_DEVICES, MD_NAME, MD_UUID, MD_UPDATE_TIME, MD_DEV_UUID, MD_EVENTS):
   1. mdadm – -examine – -export /dev/SVAVolGroup00/V0000000D0000001
   2. mdadm – -examine – -export /dev/SVAVolGroup00/V0000001D0000000
      
7. Now we need to stop the firewall:
   1. /etc/init.d/SuSEfirewall2_setup stop
   2. /etc/init.d/SuSEfirewall2_init stop
      
8. MDADM[2] (result will be like [mdadm: /dev/md0 has been started with 1/2 drive (out of 2)]:
   1. mdadm – -assemble – -run – -force /dev/md0 /dev/SVAVolGroup00/V0000000D0000001
   2. mdadm – -assemble – -run – -force /dev/md0 /dev/SVAVolGroup00/V0000001D0000000
      
9.  Start the NFS server service: service nfsserver start (output will be something like [Starting kernel based NFS server: idmapd mountd statd nfsd sm-notify done]).
10. Mounting the volumes:
    1. mkdir /exports/V0000000D0000000
    2. mkdir /exports/V0000001D0000001
    3. mount -rw -o barrier=0,data=ordered,nodelalloc,errors=panic /dev/md0 /exports/V0000000D0000000
    4. mount -rw -o barrier=0,data=ordered,nodelalloc,errors=panic /dev/md1 /exports/V0000001D0000001
    5. exportfs -o rw,no_root_squash,sync,no_all_squash,insecure :/exports/V0000000D0000000
    6. exportfs -o rw,no_root_squash,sync,no_all_squash,insecure :/exports/V0000001D0000001
    7. After using the exportfs command you should get something like [exportfs: No host name given with /exports/V0000000D0000000 …etc… to avoid warning] this means that the volumes were exported successfully.
11. Verify volumes before adding them to the ESXi hosts:
    1. Type: ls -als /exports/V0000000D0000000 and ls -als /exports/V0000000D0000000  and you should see the virtual machines residing in them.
12. Finally you can go to the host configuration -> storage -> add the new NFS datastores /exports/V0000000D0000000 and /exports/V0000000D0000000 using the IP address of the VSA appliance.
13. Make sure you move the virtual machines out of the NFS datasore and keep them safe because you will need to delete the VSA cluster and start a new one from scratch.

Hopefully this will send you back home early to sleep rather than staying up all night eating your fingernails and drinking caffeine ;-).

(Abdullah)^2