vMotion fails at 13 % and becomes unresponsive on VMware ESXi 5.1 host

An interesting blogpost was posted by VMware recently describing an issue experienced by a VMware customer on VMware ESXi 5.1 hosts.

Symptoms were:

• Unable to contact the MKS (mouse keyboard screen) remote console
• vMotion began failing at 13 %
• Could not SSH to host or login to console via direct access such as iDRAC

The problem was caused by SNMPD. When  enabled, the /var/spool/snmp folder is filled with Simple Network Management Protocol (SNMP) trap files. If there are many files, the ESXi host runs out of inodes. When no inodes are available anymore, new files cannot be created.

Each file on a VMware datastore has an associated inode. The inode describes  data of the file. (metadata) like ownership, access and file type. More in inode here.

The workaround is to delete SNMP logfiles so inodes become available. The procedure described in this knowledgebase article.

Check number of inodes

Taken from KB: 1007638

1. Gain root access to the command line.
   • For ESX, open a SSH session.
   • For ESXi, see Tech Support Mode for Emergency Support (1003677).
2. On the ESX host, run the command:
   [root@esx /]$ stat -f /

   
   The output appears similar to:

   File: “/”
   ID: 0 Namelen: 255 Type: ext2/ext3
   Blocks: Total: 1259079 Free: 898253 Available: 834295 Size: 4096
   Inodes: Total: 640000 Free: 580065

   
   In this example, there are 640000 inodes total and 580065 are free, so there are inodes available.
   

Free inodes

If an ESXi host has low inodes, you may have to free inodes on the host.

Taken from KB: 1008643
To free inodes on an ESXi host:

1. On the ESXi host, enter Tech Support Mode. For more information, see Tech Support Mode for Emergency Support (1003677).
2. Delete the /var/log/vmware/aam/rule directory.
3. Delete any files in the /var/core/ directory.
4. Uninstall the vpxa and aam clients with the uninstall scripts in /opt/vmware/uninstallers (VMware-vpxa-uninstall.sh and VMware-aam-uninstall.sh, respectively).
5. Restart ESXi services with the /sbin/services.sh script.
6. Reconnect the ESXi host to VirtualCenter.
7. Investigate for the recent presence of more than usual small files, such as files ending .txt or .dmp. If they are present, delete the files, then check the inodes.