VMware vCenter Server Appliance 5.5 had a technical issue. View Manager dashboard reports ‘The service is not working properly’. The vSphere Client cannot connect to the appliance. This happened all of a sudden after months of having no issue at all.
Likely an upgrade from vCenter Server Appliance 5.5 Update 2 to vCenter Server Appliance 6.0 was done a while ago. This went all fine.
Troubleshooting the appliance is much more difficult than troubleshooting the vCenter Server running on Windows. This blogpost will provide some info. There are a couple of good VMware KB articles about the appliance.
Log in to the appliance using a SSH (Putty.exe for example) or directly into the console using root account.
This VMware KB article provides information about checking status of services. This KB article has an overview of the service names.
Type the command:
service vmware-vpxd status
service vmware-vpostgres status
All services should be running. If you notice the Postgres database is not running check the diskspace with command
If the available diskspace for the /storage/db is 0% there is your issue. Check the folder where logs are saved. This is /storage/db/vpostgres/pg_log
You see a lot of logs here. The content of the log is allmost all the same. This line saying “there is already a transaction in progress”.
That is the problem!
The problem is caused by a bug in vCenter Server Appliance 5.5 Update 2. The bug has been resolved in Update 3. However if you upgrade from Update 2 to vCenter Server Appliance 6.0 the bug will still be there!
The bug is that Postgres writes each and every warning in the logfile. The logs will grow very rapidly. So we need to adjust the level of logging.
A detailled description of how to adjust is described in this VMware KB.
This is a summary:
We change the level of logging by editing the /storage/db/vpostgres/postgresql.conf file. Make sure to make a copy of this file first.
Run this command to stop the VMware VirtualCenter Server service:
service vmware-vpxd stop
Run this command to stop the embedded vCenter Server database service:
service vmware-vpostgres stop
Then we change the log level.
using the downarrow on your keyboard go to a line saying log_min_messages = warning
Position the cursor next to the = character. Press i key for inserting text and type ‘error’. Delete the word ‘warning’.
The line now reads:
log_min_messages = error
Then press Escape key on your keyboard. Then :wq to save the file
We then need to remove the logfiles.
Remove all but the two newest log files from the /storage/db/vpostgres/pg_log file using the rm command:
Run this command to start the embedded vCenter Server database service:
service vmware-vpostgres start
Run this command to start the VMware VirtualCenter Server service:
service vmware-vpxd start
Check if all services start. You now can connect again to vCenter and View Manager should show a green square showing it can connect to vCenter.