Case of Dead Path on ESXi

I had 8 paths go down to a dead state on an ESXi host.  The paths were MRU via Fiber Channel to a storage array.  One path worked and it was configured as RR path.

I knew this wasn’t a physical issue, it had to be a software/configuration issue on my host because there were:

  • No storage array errors
  • Additional hosts in the cluster had no problems
  • One path still worked from the HBA

Looking at the log (/var/log/vmkernel.log) I searched for one of the LUN identifiers, in my case “:L30” which was one of the dead paths.  This yielded a result showing an error with NMP plugin driver invalid command.

Next step was to figure out and verify what NMP details were and compare against a working host.

esxcli storage nmp device list |grep "Path Selection Policy:" |sort |uniq -c

I saw nothing out of the ordinary.

Apparently the storage did not like the use of RR so I removed the SATP claim rule though, so I removed it:

esxcli storage nmp satp rule remove -V IBM -M "^1746*" -P VMW_PSP_RR -s VMW_SATP_ALUA

Storage paths are happy now.

List VMFS extents in ESXi

Run the esxcli storage vmfs extent list command to generate a list of extents for each volume and mapping from device name to UUID.

You see output similar to:

Volume Name VMFS UUID Extent Number Device Name Partition
------------ ----------------------------------- ------------- ------------------------------------ ---------
esxi-local 4e0d86e1-0db6f826-6991-d8d3855ff8d6 0 mpx.vmhba2:C0:T0:L0 3
datastore1 4d4ac840-c1386fa0-9f6d-0050569300a7 0naa.6006016094602800364ce22e3825e011 1
vmfs5 4dad8f16-911648ca-d660-d8d38563e658 0naa.600601609460280052eb8621b73ae011 1

This host has no isolation addresses defined as required by vSphere HA

Had some troubles with an ESXi 5.5 host the past week and rebuilt some of the networking which included removing a vmkernel and re-adding to a standard vSwitch and then setting up VMotion and Management tags on it.

Came across the following error on my host in vSphere Client:


To resolve this issue, I needed to make sure that there is a default gateway on the port group that the VMotion vmkernel is on.

Click on the ESXi Host then navigate to Configuration > Networking and on the VMkernel Port that I’m using for VMotion, click Properties…


Next, I examine properties of the VMotion VMkernel.


Looking at the IP Settings tab, I see there is no VMkernel Default Gateway defined, so I clicked Edit…


I entered in my default gateway and clicked OK and then OK again to exit the switch properties dialog.


Finally, I right-click on the ESXi host and click Reconfigure for vSphere HA


At this point, no additional errors are reported for my ESXi host.  =)

Monitor ESXi Free using SNMP

ESXi 4.1 and 5.0 Enable SNMP

SSH to host and edit the SNMP.XML file:

vi /etc/vmware/snmp.xml

Make the following changes:


Restart management agents with the following command:

/etc/init.d/hostd restart

On ESXi 5.1 and 5.5 enable SNMP

SSH to host and run the following command:

esxcli system snmp set --communities=public --enable=yes --targets=

Test SNMP trap

vicfg-snmp --server <ESXiServerIP> --username root --password <Password> --test

Reviewing the SNMP configuration

When I look at my SNMP configuration using vCLI (once again, this is a read operation so I can use vCLI), I see the following.

vi-admin@vma:~> vicfg-snmp --server <ESXiServerIP> --username root --password <Password> -s

Current SNMP agent settings:
Enabled : 1
UDP port : 161

Communities :

Notification targets :

Options :