Upgrading from vSphere 6.0u2 to vSphere 6.5a

I finally had the time to upgrade my home lab last week and today I am able to write about it, as I have mentioned earlier in several posts and for those who know me I like to get my hands dirty before reading any manual (it is kind of the feeling you get when you do a VCAP, it you and the terminal..follow the white rabbit ;-)). It is going to be long post, nothing much interesting with the vCenter upgrades! Most of the fun was with my ESXi hosts %).

My lab currently is at 6.0u2 and I intend to upgrade it to 6.5a, I couldn’t get it to 6.5 earlier because NSX was still not yet available/supported and I was some tests on the NSX platform so it was a no no :/ at that time, but we’re good now as per the interoperability matrix here & here, also I have a Citrix installation in my lab and I double checked and found out that 6.5 is also supported here with some known issues.

vCenter Servers:

1. Management cluster vCenter is the Windows flavor of vCenter: (I have to keep in case I need to test something on that specific distribution). This is single server that hosts both PSC and vCenter services.
   1. Just a few clicks here and the upgrade went fine all the way without any single error.
      
      
2. Test cluster vCenter is the appliance favor, this a combination of two servers where I have 1 PSC and 1 VCSA:
   1. I mounted the bits and launched the installer, after filling the information and went through as I went through the validation window I got an error telling me that I have to do the PSC first [My thoughts: I would have preferred it to just go and contact the PSC and prepare it within the same single upgrade task].
      
   2. I re-launched the setup and targeted the PSC instead, and amazingly the upgrade was super smooth and straight forward and it didn’t even flinch!
      
   3. Now its the vCenter Server back on the menu, I kicked in the upgrade with my heart full of confidence that it is going to work BUT sadly it popped another error.
      
   4. After digging into it, I found out that I had issues with my NTP server (my Active Directory server) and the second management ESXi had its time totally messed up, I know from previous experiences that it is imperative for the PSC to have the same time synchronization as the vCenter even if the time is wrong but as long as they are both the same the installation or migration would work smoothly. So after I fixed my AD and got everything synchronizing the time properly I re-attempted the upgrade and it went successful without any further issues [My thoughts: I really like the fact that I did not have to re-run the setup again, I only had to access the newly deployed appliance and things went straight forward from there].
      

ESXi Hosts:

1. For the management cluster I have two HPE Elitebook workstations (8540w and 8560P):
   1. Normally with my Elitebook workstation since the early days of ESXi 5.x I had to create my own customized image of ESXi with the specific Intel VIB injected into the ISO (blog post here).
   2. I prepared the ISO, injected the VIB, placed it on a USB, booted from the USB, waited for the installation to kick-in AND I got a notification on the screen telling me that the setup could not find a valid NIC to support ESXi *sniff* here I was suspicious that I might have not replaced the VIB when the I was prompted to but after checking I actually did so there was nothing wrong with the ISO.
   3. Digging further into the topic and after a lot of reading I deduced that ESXi now should have the drivers that I want without even having to inject any VIB.
   4. So I rebooted the host, I already had the ESXi 6.5a offline bundle on one of my NFS datastores, connected to the host via ssh and kicked-in the updated via “esxcli software profile update -d /vmfs/volumes/br00mWalkerNFS/updates/ESXi650-201701001.zip -p ESXi-6.5.0-20170104001-standard“.
   5. The update went fine, upon completion it required a reboot.
   6. The host came up AND not network *double sniff*, here I was curious to see the modules that were loaded by ESXi. Again using esxcli but from the console this time I ran “esxcli system module list | grep 1000” to see if ESXi has actually loaded the driver that I want, and I found two drivers being loaded the first e1000e but it was disabled and then ne1000 and it was enabled.
      
   7. So I thought what if I disabled the ne1000 and enabled the e1000e to see if this would work or I would have to go back to 6.0u2 until I figure things out, again using the trusty esxcli I ran “esxcli system module set –enabled=false –module=ne1000” [it is a double dash] and rebooted the host.
   8. AND IT BLOODY WORKED! *started dancing in the house – while my wife looked at me as if I was a lunatic :-P which could be true you know*
      
   9. I upgraded my second workstation the same way and now my management cluster is at 6.5a successfully.
2. For the testing cluster I Have three HPE ML150 G9 hosts:
   1. These hosts were running NSX 6.2.4 and I had the installation cleaned up due to a lot of messy tests that I had it undergo.
   2. I kicked in the upgrade via esxcli on the first host and I got the following error:
      
   3. So this is perfectly normal, I did the NSX cleanup but I did not remove the VIBs from the ESXi hosts and since ESXi 6.5a requires NSX 6.3 I am presuming it is related to that specific reason, so after I removed the NSX vibs (esxcli software vib remove  -n esx-vxlan || esxcli software vib remove  -n esx-vdpi || esxcli software vib remove  -n char-hpcru) and the HP vib as well (the 6.02 installation was done via HP’s customized image and it was not released at that point).
   4. After removing the VIBs, I ran the update and it completed successfully without any issues.

Good luck with your upgrade and I hope this help,
(Abdullah)^2