HPE ESXi 6.7 U2 Customized Image and HPE Dynamic Smart Array B140i Controller Driver

I’ve been busy for the past two days upgrading my home lab to 6.7 U2 and one of my servers is a ProLiant ML150 Gen9 and its RAID controller is a B140i. This host was on 6.5u1 and based on the upgrade path in the interoperability matrix a single shot upgrade is supported.

In the lab usually I attempt different ways of upgrading the hosts, so that (1) to keep my know-how around the client fresh, (2) to make sure that things actually work as they’re expected as if I was going to do them for a customer.

For this specific server I used the offline bundle (HPE Gen9 Plus Custom Image for ESXi 6.7 Update 2 Offline Bundle). Since we were on 6.5 and not on the 6.7 GA I couldn’t use the “esxcli software vib update” command and I had to use the “esxcli software vib install” command instead because it complained about the esx-base image.

After the upgrade was done, the server booted but the system fans kept screaming. At this point in time and even before logging in to the system I knew something broke *YIKES* and at that point in time my wife had plans to go out with the kids and I had around 30 minutes before I start getting threats of unplugging the home-lab cabinet from power :-P.

I had the latest HPE SPP, so I applied it on the system but still, the ESXi couldn’t recognize the array controller. I did a quick search and found that there is a newer driver from HPE with the title [* RECOMMENDED * HPE Dynamic Smart Array B140i Controller Driver for VMware vSphere 6.7 (Driver Component).]

In that updated driver it clearly says “This issue is only encounter when upgrading to version 2018.09.31 (5.5.0.66-1). All other versions are not affected.“

To verify and validate that this was caused by the update and that the driver wasn’t actually part of my 6.5u1 installation I went to the esxupdate logs in /var/log/esxupdate.log

2019-06-07T13:11:41Z esxupdate: 15329961: BootBankInstaller.pyc: INFO: Copying /bootbank/intelcim.v00 to /tmp/stagebootbank/intelcim.v00
2019-06-07T13:11:41Z esxupdate: 15329961: HostImage: DEBUG:  --- Stage: BootBankInstaller adding [QLogic_bootbank_qlnativefc_3.1.16.0-1OEM.670.0.0.8169922, INT_bootbank_ixgben_1.7.10-1OEM.670.0.0.8169922, VMW_bootbank_misc-drivers_6.7.0-2.48.13006603, EMU_bootbank_brcmfcoe_12.0.1211.0-1OEM.670.0.0.8169922, QLC_bootbank_qedi_2.10.5.3-1OEM.670.0.0.8169922, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.670.0.0.8169922, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.670.0.0.8169922, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-devintf_39.1-5vmw.670.1.28.10302608, VMware_bootbank_rste_2.0.2.0088-7vmw.670.0.0.8169922, VMware_bootbank_cpu-microcode_6.7.0-1.28.10302608, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.670.0.0.8169922, VMW_bootbank_block-cciss_3.6.14-10vmw.670.0.0.8169922, VMW_boot
2019-06-07T13:11:41Z esxupdate: bank_scsi-libfc-92_1.0.40.9.3-5vmw.670.0.0.8169922, VMW_bootbank_lpnic_11.4.59.0-1vmw.670.0.0.8169922, VMware_bootbank_vsanhealth_6.7.0-2.48.12775454, VMW_bootbank_sata-sata-sil_2.3-4vmw.670.0.0.8169922, VMW_bootbank_ata-pata-via_0.3.3-2vmw.670.0.0.8169922, VMW_bootbank_vmkplexer-vmkplexer_6.7.0-0.0.8169922, VMW_bootbank_iavmd_1.2.0.1011-2vmw.670.0.0.8169922, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.670.0.0.8169922, VMW_bootbank_sata-sata-promise_2.12-3vmw.670.0.0.8169922, VMW_bootbank_sata-ahci_3.0-26vmw.670.0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_scsi-ips_7.12.05-4vmw.670.0.0.8169922, VMW_bootbank_nvmxnet3_2.0.0.29-1vmw.670.1.28.10302608, HPE_bootbank_oem-build_670.U2.10.4.1-7535516, QLC_bootbank_qedrntv_3.10.23.0-1OEM.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-si-drv_39.1-5vmw.670.1.28.10302608,
2019-06-07T13:11:41Z esxupdate: EMU_bootbank_lpfc_12.0.257.5-1OEM.670.0.0.8169922, VMW_bootbank_hid-hid_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ne1000_0.8.4-2vmw.670.2.48.13006603, QLC_bootbank_qfle3i_1.0.20.0-1OEM.670.0.0.8169922, VMware_bootbank_vsan_6.7.0-2.48.12775451, VMW_bootbank_xhci-xhci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.670.0.0.8169922, BCM_bootbank_bnxtnet_212.0.119.0-1OEM.670.0.0.8169922, VMW_bootbank_ntg3_4.1.3.2-1vmw.670.1.28.10302608, Avago_bootbank_lsi-mr3_7.706.08.00-1OEM.670.0.0.8169922, VMW_bootbank_lsi-msgpt2_20.00.05.00-1vmw.670.2.48.13006603, VMW_bootbank_net-e1000_8.0.3.1-5vmw.670.0.0.8169922, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.670.0.0.8169922, EMU_bootbank_elxnet_12.0.1115.0-1OEM.670.0.0.7535516, VMW_bootbank_sata-sata-sil24_1.1-1vmw.670.0.0.8169922, VMW_bootbank_net-usbnet_1.0-3vmw.670.0.0.8169922, HPE_bootbank_smx-provider_670.03.15.00.3-7535516, VMW_bootb
2019-06-07T13:11:41Z esxupdate: ank_ohci-usb-ohci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.670.0.0.8169922, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2.48.13006603, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.670.0.0.8169922, VMW_bootbank_vmkfcoe_1.0.0.1-1vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-7vmw.670.0.0.8169922, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.670.0.0.8169922, QLC_bootbank_qedf_1.3.22.0-1OEM.600.0.0.2768847, QLC_bootbank_qedentv_3.10.23.1-1OEM.670.0.0.8169922, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.670.0.0.8169922, VMware_bootbank_esx-xserver_6.7.0-0.0.8169922, VMware_bootbank_lsu-smartpqi-plugin_1.0.0-3vmw.670.1.28.10302608, HPE_bootbank_cru_670.6.7.10.14-1OEM.670.0.0.7535516, VMW_bootbank_net-nx-nic_5.0.621-5vmw.670.0.0.8169922, HPE_bootbank_conrep_6.7.0.03-03.00.10.7535516, VMW_bootbank_lsi-msgpt3_17.00.01.00-3vmw.670.2.48.13006603, VMW_bootbank_ata-pata-hpt3x
2019-06-07T13:11:41Z esxupdate: 2n_0.3.4-3vmw.670.0.0.8169922, MEL_bootbank_nmst_4.10.0.302-1OEM.650.0.0.4598673, VMW_bootbank_net-enic_2.1.2.38-2vmw.670.0.0.8169922, VMW_bootbank_sata-sata-nv_3.5-4vmw.670.0.0.8169922, VMW_bootbank_ipmi-ipmi-msghandler_39.1-5vmw.670.1.28.10302608, VMware_bootbank_esx-dvfilter-generic-fastpath_6.7.0-0.0.8169922, INT_bootbank_igbn_1.4.7-1OEM.670.0.0.8169922, VMware_bootbank_esx-ui_1.33.3-12923304, INT_bootbank_i40en_1.7.11-1OEM.670.0.0.8169922, VMW_bootbank_sfvmk_1.0.0.1003-6vmw.670.2.48.13006603, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.670.0.0.8169922, VMware_bootbank_esx-base_6.7.0-2.48.13006603, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.670.0.0.8169922, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.670.0.0.8169922, VMW_bootbank_vmw-ahci_1.2.3-1vmw.670.1.28.10302608, HPE_bootbank_scsi-hpdsa_5.5.0.66-1OEM.550.0.0.1331820, HPE_bootbank_hponcfg_6.7.0.5.4-0.3.7535516, VMware_bootbank_lsu-lsi-drivers-
2019-06-07T13:11:41Z esxupdate: plugin_1.0.0-1vmw.670.2.48.13006603, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.670.0.0.8169922, BCM_bootbank_bnxtroce_212.0.114.0-1OEM.670.0.0.8169922, VMW_bootbank_nfnic_4.0.0.17-0vmw.670.2.48.13006603, HPE_bootbank_amsd_670.11.4.0-18.7535516, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-9vmw.670.2.48.13006603, VMware_bootbank_lsu-intel-vmd-plugin_1.0.0-2vmw.670.1.28.10302608, VMW_bootbank_shim-libata-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.670.0.0.8169922, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.670.2.48.13006603, VMW_bootbank_shim-libata-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.670.0.0.8169922, EMU_bootbank_elxiscsi_12.0.1188.0-1OEM.670.0.0.8169922, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.670.0.0.8169922, HPE_bootbank_fc-enablement_670.3.40.10-7535516, VMW_bootbank_pvscsi_0.1-2vmw
2019-06-07T13:11:41Z esxupdate: .670.0.0.8169922, MEL_bootbank_nmlx5-rdma_4.17.13.8-1OEM.670.0.0.8169922, MEL_bootbank_nmlx5-core_4.17.13.8-1OEM.670.0.0.8169922, ELX_bootbank_elx-esx-libelxima-8169922.so_12.0.1188.0-03, VMW_bootbank_nvme_1.2.2.27-1vmw.670.2.48.13006603, VMW_bootbank_iser_1.0.0.0-1vmw.670.1.28.10302608, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-14vmw.670.0.0.14539518, VMW_bootbank_usbcore-usb_1.0-3vmw.670.0.0.8169922, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.670.0.0.8169922, VMW_bootbank_net-forcedeth_0.61-2vmw.670.0.0.8169922, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-13vmw.670.1.28.10302608, VMW_bootbank_char-random_1.0-3vmw.670.0.0.8169922, HPE_bootbank_testevent_6.7.0.02-00.01.12.7535516, MSCC_bootbank_smartpqi_1.0.3.2302-1OEM.670.0.0.8169922, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.670.0.0.8169922, QLC_bootbank_qfle3_1.0.77.2-1OEM.670.0.0.8169922, VMW_bootbank_vmkusb_0.1-1vmw.670.2.48.13006603, VMW
2019-06-07T13:11:41Z esxupdate: _bootbank_mtip32xx-native_3.9.8-1vmw.670.1.28.10302608, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-9vmw.670.0.0.8169922, VMW_bootbank_net-cdc-ether_1.0-3vmw.670.0.0.8169922, VMW_bootbank_nvmxnet3-ens_2.0.0.21-1vmw.670.0.0.8169922, HPE_bootbank_ssacli_3.40.3.0-6.7.0.7535516, HPE_bootbank_ilo_670.10.1.0.16-1OEM.670.0.0.7535516, VMW_bootbank_nenic_1.0.21.0-1vmw.670.1.28.10302608, HPE_bootbank_bootcfg_6.7.0.02-05.00.4.7535516, VMW_bootbank_scsi-aic79xx_3.1-6vmw.670.0.0.8169922, VMW_bootbank_sata-ata-piix_2.12-10vmw.670.0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfcoe-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_lsi-msgpt35_09.00.00.00-1vmw.670.2.48.13006603, VMware_bootbank_esx-update_6.7.0-2.48.13006603, VMW_bootbank_shim-vmklinux-9-2-3-0_6.7.0-0.0.8169922, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.670.0.0.8169922, VMW_bootbank_shim-libfc-9-2-1-0_6.7.0-0.0.8
2019-06-07T13:11:41Z esxupdate: 169922, VMW_bootbank_shim-vmklinux-9-2-1-0_6.7.0-0.0.8169922, VMW_bootbank_shim-vmklinux-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_shim-libfc-9-2-2-0_6.7.0-0.0.8169922, VMW_bootbank_sata-sata-svw_2.3-3vmw.670.0.0.8169922, QLC_bootbank_qcnic_1.0.22.0-1OEM.670.0.0.8169922, VMW_bootbank_scsi-mpt2sas_19.00.00.00-2vmw.670.0.0.8169922, VMW_bootbank_qflge_1.1.0.11-1vmw.670.0.0.8169922, QLC_bootbank_qfle3f_1.0.63.0-1OEM.670.0.0.8169922, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.670.0.0.8169922, VMW_bootbank_nmlx4-core_3.17.13.1-1vmw.670.2.48.13006603, VMW_bootbank_nmlx4-rdma_3.17.13.1-1vmw.670.2.48.13006603, VMware_bootbank_native-misc-drivers_6.7.0-2.48.13006603, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.670.0.0.8169922, VMW_bootbank_vmkata_0.1-1vmw.670.0.0.8169922, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.670.0.0.8169922, VMW_bootbank_nmlx4-en_3.17.13.1-1vmw.670.2.48.13006603, Microsemi_bootbank_nhpsa_2.0.38-1OEM.6

Looking into the log I verified that the driver was indeed installed with the customized offline bundle “HPE_bootbank_scsi-hpdsa_5.5.0.66-1OEM.550.0.0.1331820“.

So, I downloaded the new driver, extracted the VIB and then ran “esxcli software vib update -v /tmp/vibfile“. It took some time for the driver to get updated, and it requires a reboot.

Once the server rebooted, similarly to how I knew that a problem had risen, the fans went back to normal and I knew the driver fixed the issue and I was able to get my local logical volume back online.

Lesson learned: Keep breaking stuff in the lab so that you don’t have to break stuff at customers premises :-D.