ESXi Failed to open device naa.(GUID) No space left on device
So I had a late call from a friend saying that he can’t mount a datastore after expanding it from the storage, it was a weekend and next day was Monday and the wife is not happy ;-).
So that storage is a VNXe iSCSI box and what I did at first was try to replicate the scenario on a new ESXi host and everything went straight forward with my newly created LUN, I increased the size and expanded the datastore without any issues.
So at first I thought it might be related to the partition table where I spent quite sometime playing around with partedUtil without any luck and I was staring at this line 2014-11-23T21:42:52.940Z cpu1:34611 opID=e0f84d62)LVM: 11786: Failed to open device naa.GUID:1 : No space left on device
I am not an EMC guy but I had to check the storage management console, when I went to the storage pool I found out that it was already 107% over utilized :-D and from there I was able to logically conduct my analysis:
- After asking more questions it seems that the VMDKs were thin provisioned and for sure the requested allocation exceeded the available space on the datastore.
- The LUN in the storage pool was thin provisioned and it was 107% over allocated as well and even though he expanded it but in terms of availability there was no real space to allocate.
And since there no real space to be given to the datastore and ESXi thinks that the space is there it was trying to extend the sectors of the partition table but it couldn’t because as I said earlier the VNXe could not provide the required space.
We tried to decrease the LUN size from the storage but it was not possible (by design we’re allowed to increase only), I even tried to use partedUtil to delete the partition and I always ended up with the no space left of device error.
So the only solution was to increase the disks in the storage pool and then the LUN would be able to commit and provide ESXi with the storage it needs for the datastore.
I will update the post once the disks arrive and I will inform you if things are sorted afterwards.
Update 08/01/2015: Hard drives arrived and as expected once added to the array we were able to do the increase and mounted the datastores without any issues.