ESXi 5.5 and HP-AMS Heap globalCartel-1 already at its maximum size

Nothing better than waking to a Saturday support call, no? Don’t you just love not having a weekend and support your clients instead ;-). So long story short, it was a VMware Horizon View infrastructure and all the virtual machines that were being requested by the users were not powering on and we were getting this error:

I viewed the VMKernel warning log files (/var/log/vmkwarning) and logged the same lines that were showing in the latter screenshot, so I did a quick research and found this KB from VMware http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2085618 where they explain that this is related to HP-AMS package.

Note that those systems were installed with the customized HP ESXi 5.5 image, and was kept with the default packages at that time because no issues had risen.

At that time I couldn’t SSH to the hosts nor I did have access to the console shell because it was disabled and I couldn’t even enable it, we had to reboot each one of the hosts before being able to work on them, in addition one of the hosts didn’t shutdown properly and we had to cycle it by force.

After the hosts went up, I didn’t remove the old package I only downloaded the latest HP-AMS package from here http://h20564.www2.hp.com/hpsc/swd/public/detail?swItemId=MTX_b05d4c644fb742aa87cb5f5da1 and installed the packages directly, the package requires a reboot after its installed.

Finally it is not practical to wait for these things to happen and then troubleshoot, so I recommended that the customer would signup on this website https://h41183.www4.hp.com/sft_signup.php?country=US&language=US so that to receive updates regarding any new patches that are of relevance to the existing environment.

I hope this helps and I hope I am not repeating the same content but I felt I should share my experience.

(Abdullah)^2