HP 3PAR 10400 OOTB not responding

So, I newly had an HP 3PAR 10400 to initialize (4 controllers and 8 disk cages), as usual planned this with HP 3PAR deployment team and I installed the hardware and powered up the system and all was green and we were moving forward without any complications thankfully =).

1.  First weird thing I noticed is that when I was logging in to the nodes (any node) it took more than 2 minutes for the menu to show.
2. Starting OOTB also took a great deal of time and it was stuck showing nothing, it was not even giving me an error, so I just exited it and returned back to the main menu.
3. The next obvious option to select as the health check and that also idled as some point but when I exited the health check script some error popped:
   1. Could not check ao: Error: The cache server is waiting for the system manager to finish reporting the system configuration and status.  Please try again later.
   2. Could not check cage: System manager is still initializing – try this command later.
4. I informed the HP engineer about the error and did a little but of research and didn’t find anything promising as one of the HP forums had info about HP replacing the mid-plane which is weird because the mid-plane is a passive component and for it to break it something _merely_ impossible also HP guys immediately took out this option.
5. So HP’s support brought it in their heavy gear and we spent around 2 days troubleshooting, long story short these are the details on how we got it resolved:
   1. First I completed the Service Processor MOB (because they had an idea of connecting the SP and the InServ back to back and perform a sort of an OS re-installation via SP).
   2. We performed a de-installation for the array and waited for it to complete (took around 16 hours).
   3. After the de-installation completed successfully, we powered of all nodes and kept only node 0 up.
   4. The array was in lab mode as they had to logon to the array using lab credentials.
   5. We performed a node_prep on all nodes and rebooted each.
   6. After all nodes were up we noticed that the serial number of the array has changed (this was resolved in whack mode).
   7. At this point the system was responsive again and we didn’t see the earlier mentioned symptoms, and we’re ready for OOTB.
   8. We started OOTB but sadly it also popped some errors, so we had to run the system wipe scripts and retry OOTB.
   9. At this point we managed to complete OOTB successfully and the system was up and running.

Note that the up above should be only done with HP 3PAR support, do not attempt to perform the up above as you will most probably void support and will have to pay a lot of money to recover ;-).

I thought I would share this information with you as I couldn’t find any blog/forum posts about it, again I am extremely pleased with the 3PAR array as a system and I never get amazed by the way it get supported and how flexible it is to maintain and recover.

(Abdullah)^2