An interesting blogpost was posted by VMware recently describing an issue experienced by a VMware customer on VMware ESXi 5.1 hosts.
- Unable to contact the MKS (mouse keyboard screen) remote console
- vMotion began failing at 13 %
- Could not SSH to host or login to console via direct access such as iDRAC
The problem was caused by SNMPD. When enabled, the
/var/spool/snmp folder is filled with Simple Network Management Protocol (SNMP) trap files. If there are many files, the ESXi host runs out of inodes. When no inodes are available anymore, new files cannot be created.
Each file on a VMware datastore has an associated inode. The inode describes data of the file. (metadata) like ownership, access and file type. More in inode here.
The workaround is to delete SNMP logfiles so inodes become available. The procedure described in this knowledgebase article.
Check number of inodes
Taken from KB: 1007638
To determine if the ESX console file system and the VMFS file system have reached the inode or filesystem limit:
- Gain root access to the command line.
- On the ESX host, run the command:
[root@esx /]$ stat -f /
The output appears similar to:
ID: 0 Namelen: 255 Type: ext2/ext3
Blocks: Total: 1259079 Free: 898253 Available: 834295 Size: 4096
Inodes: Total: 640000 Free: 580065
In this example, there are 640000 inodes total and 580065 are free, so there are inodes available.
If an ESXi host has low inodes, you may have to free inodes on the host.
Taken from KB: 1008643
To free inodes on an ESXi host:
- On the ESXi host, enter Tech Support Mode. For more information, see Tech Support Mode for Emergency Support (1003677).
- Delete the /var/log/vmware/aam/rule directory.
- Delete any files in the /var/core/ directory.
- Uninstall the vpxa and aam clients with the uninstall scripts in /opt/vmware/uninstallers (VMware-vpxa-uninstall.sh and VMware-aam-uninstall.sh, respectively).
- Restart ESXi services with the /sbin/services.sh script.
- Reconnect the ESXi host to VirtualCenter.
- Investigate for the recent presence of more than usual small files, such as files ending .txt or .dmp. If they are present, delete the files, then check the inodes.