vMotion fails at 13 % and becomes unresponsive on VMware ESXi 5.1 host

An interesting blogpost was posted by VMware recently describing an issue experienced by a VMware customer on VMware ESXi 5.1 hosts.

Symptoms were:

  • Unable to contact the MKS (mouse keyboard screen) remote console
  • vMotion began failing at 13 %
  • Could not SSH to host or login to console via direct access such as iDRAC

The problem was caused by SNMPD. When  enabled, the /var/spool/snmp folder is filled with Simple Network Management Protocol (SNMP) trap files. If there are many files, the ESXi host runs out of inodes. When no inodes are available anymore, new files cannot be created.

Each file on a VMware datastore has an associated inode. The inode describes  data of the file. (metadata) like ownership, access and file type. More in inode here.

The workaround is to delete SNMP logfiles so inodes become available. The procedure described in this knowledgebase article.

Check number of inodes

Taken from KB: 1007638

To determine if the ESX console file system and the VMFS file system have reached the inode or filesystem limit:
  1. Gain root access to the command line.
  2. On the ESX host, run the command:
    [root@esx /]$ stat -f /


    The output appears similar to:

    File: “/”
    ID: 0 Namelen: 255 Type: ext2/ext3
    Blocks: Total: 1259079 Free: 898253 Available: 834295 Size: 4096
    Inodes: Total: 640000 Free: 580065


    In this example, there are 640000 inodes total and 580065 are free, so there are inodes available.

Free inodes

If an ESXi host has low inodes, you may have to free inodes on the host.

Taken from KB: 1008643
To free inodes on an ESXi host:

  1. On the ESXi host, enter Tech Support Mode. For more information, see Tech Support Mode for Emergency Support (1003677).
  2. Delete the /var/log/vmware/aam/rule directory.
  3. Delete any files in the /var/core/ directory.
  4. Uninstall the vpxa and aam clients with the uninstall scripts in /opt/vmware/uninstallers (VMware-vpxa-uninstall.sh and VMware-aam-uninstall.sh, respectively).
  5. Restart ESXi services with the /sbin/services.sh script.
  6. Reconnect the ESXi host to VirtualCenter.
  7. Investigate for the recent presence of more than usual small files, such as files ending .txt or .dmp. If they are present, delete the files, then check the inodes.
About these ads

About Marcel van den Berg
I am a technical consultant with a strong focus on server virtualization, desktop virtualization, cloud computing and business continuity/disaster recovery.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 94 other followers

%d bloggers like this: