Updating vSphere Customized Images: The VIB Space Conflict
My first blog after almost 3 months of 0 blogs (due to various reasons that I will share soon as it’s a lot to write about ^_^).
With the recent release of NSX-T 2.5, I spared some sleeping hours to perform the upgrade. NSX-T went fine with 0 issues, the vCenter Server from 6.7U2 to U3 went fine as well with no issues, with vSphere things started to go south because my lab knows that I need to sleep and it just enjoys depriving me from it :-P.
My hosts are HPE based (all Gen9), the management is not prepared for NSX-T and the resource are.
With that said, I got the HPE offline bundle (VMware-ESXi-6.7.0-Update3-14320388-HPE-Gen9plus-670.U3.10.4.5.19-Aug2019-depot.zip) and with all the self-trust I had, I issued the update command and waited for the process to be done so that I reboot the host and move on to the other two hosts, however, the results were not what I was expecting and I got this instead:
[InstallationError]
The pending transaction requires 243 MB free space, however the maximum supported size is 239 MB.
Please refer to the log file for more details.I did a quick research and found this KB article (https://kb.vmware.com/s/article/2144200) and it seems that I have to remove VIBs. This is the first time I have faced this issue and I have been upgrading my hosts normally using offline bundles since the very day I started managing their life cycle, and the only difference at this point in time is that this is the first time I do it while having the NSX-T VIBs installed (28 VIBs that gets pushed automatically upon transport node preparation to be exact).
The first thing I tried was to un-configure NSX-T on the one of the hosts and attempt the upgrade (requires a reboot for the VIB removal to be realized even if the VIB removal is done automatically), after the host came up, I was able to issue the upgrade command successfully, and now comes the fun part, I put the host again in the cluster and waited for it to be automatically prepared again, sadly upon installing the NSX-T VIBs the same installation error related to free space popped and it looked like removing VIBs was imminent.
For my next attempt, I imagined that I am in a production environment and I might have everything on an NVDS and un-configuring the host was a matter out of the question, so:
- I filtered out all the non-VMware VIBs
- Cross-checked the devices that are using drivers from that list.
- Was keen on staying away from removing any of the HPE VIBs.
- Also I had the Synology VAAI VIB installed, so I wanted to keep away from removing that as well.
Based on the different trial and error attempts, I was able to pull-off the upgrade by removing the following VIBs (this will clearly be different
esxcli softwarevib remove -n i40en -nigbn -nixgben esxcli softwarevib remove -n nmlx5-core -n nmlx5-rdma -nnmst esxcli softwarevib remove -nelx –esx –libelxima .so -nelx –esx –libelxima -8169922.soesxcli softwarevib remove -nbrcmfcoe esxcli softwarevib remove -nelxiscsi esxcli softwarevib remove -nelxnet esxcli softwarevib remove -nlpfc esxcli softwarevib remove -n net-bnx2 -n net-bnx2xesxcli softwarevib remove -nqlnativefc esxcli softwarevib remove -nscsi -bnx2fc -nscsi -bnx2i- esxcli software vib remove -n net-cnic -n misc-cnic-register
- esxcli software vib remove -n smartpqi
- esxcli software vib remove -n nhpsa
- esxcli software vib remove -n lsi-mr3
esxcli softwarevib remove -nbnxtnet -nbnxtroce
After the removal of each VIB (to see if this was enough or not, so you can imagine a lot of reboots were involved) I tried to apply the update but it kept failing until I reached the last one found in the list above, and that is where the command was issued successfully.
The curious thing is that after the update was applied, I went back and listed the VIBs that were installed and found that a lot of the VIBs that I had removed were part of the update itself and got re-installed.
Based on the KB mentioned in the beginning of this blog, there is no resolution for this and this would only be related to ESXi deployments with OEM customized images (which I suspect is widely used in production environments).
Once I have further updates on this, I will definitely add it to this blog post.
(Abdullah)^2
Same issue, but on production server with vSAN and GPU installed, I am not able to uninstall same drivers as you.. Opened Support Ticket on HPE. Hope they helps..
We are in a similar situation. I have been working with the NSX-T rapid deploy team trying to get it installed in our environment and ran into this problem. I ended up creating a custom ISO image with only the NIC drivers we needed and then rebuilt each of our production ESXi hosts with the new image. We had no problem installing NSX-T until we started on our VDI environment. We use nVidia GRID cards and of course with the VIBs installed for that we do not have enough space to install NSX-T.
VMware support has been less than helpful with this. They keep telling me to remove unnecessary VIBs, which I thought I accomplished with the custom image, however installing the HP Management extensions also installs many VIBs which may be unnecessary. VMware wants me to go to HPE support now.
One small ray of light is that the NSX-T guy said in vSphere 7 they will increase the size of the bootbank to hopefully eliminate this issue. But who knows what that’ll launch AND be stable enough to run?
Josef Zach, I’d like to hear if HPE was able to help you in your case?
Hello, nope HPE doesn’t recommend removing VIBs sadly. Similar to you, I do look for vSphere 7 where the issue will be finaly resolved (hopefully).
https://support.hpe.com/hpesc/public/docDisplay?docId=a00105350en_us&docLocale=en_US please refer HPE documentation about removal of VIBs.