vCAC enables automated provisioning of both virtual machines and physical servers (including the guest operating system). Provisioning of both on-premises and cloud infrastructures is supported. vCAC enables authorized end-users to quickly have access to servers without IT having to perform the provisioning. Provisioning is so reduced from days to minutes.

Agility, speed of provisioning and reducing costs are the top 3 business drivers for companies looking to deploy private and hybrid cloud infrastructures.

Typical use case scenario’s for vCAC are service providers, large organizations and organizations which do a lot of testing and development. Those environments are very dynamic with relatively many changes.
Supported resources (vCAC calls them endpoints) are VMware vSphere, Microsoft Hyper-V 2008 and Citrix XenServer. KVM support is on the roadmap for first half of 2013. Supported public clouds are all cloud providers which support VMware vCloud standard (over 150 providers worldwide) and Amazon Web Services Elastic Cloud Computing EC2.  Supported physical servers are those equipped with HP iLO, Dell iDrac and Cisco UCS.  There is no out of the box Windows Azure endpoint available. However  integration with vCAC can be provided via VMware Professional services.  For IBM bare metal server support VMware Professional services can be approached as well. Also offered is interfacing to VMware vCenter Orchestrator 5.1 and Powershell.

vCAC does lifecycle management as well. At request for a new VM the end of life of the VM can be set. At the end of the use period the VM is deleted automatically by vCAC.  Also chargeback is possible.

vCAC supports a wide range of image deployment tools. Microsoft SCCM and Citrix Provisioning Server are a few examples. See the image below for a complete overview.

For a full overview of supported systems see the vCloud Automation Center Support Matrix.

vCAC enables IT-departments to serve as a  broker service for internal cloud consumers. End-users are able to create virtual machines/physical servers from a self service portal consuming the resources that have been allocated to them by the IT-administrator. Gartner predicts that IT organizations will increasingly be assuming internal “cloud services brokerage” roles — overseeing  the provisioning and consumption of  heterogeneous and often complex cloud services for “their internal users and external business partners.”

You can think of vCAC as a abstraction layer above hypervisors, public clouds and physical servers. It’s functionality pretty much looks like vCloud Director. Only vCloud Director is limited in its support; it supports VMware infrastructures (vCenter, ESX) only. cVAC can connect to vCloud Director and to vCenter Server. vCAC will be integrated with  VMware vFabric Application Director in 2013 to be able to perform Platform as a Service.

In an IT-world in which hybrid clouds are becoming the standard it is interesting to see what the roadmap  of vCloud Director and vCAC will be. Will VMware continue develop both solutions?

vCAC is available as a standalone license and as well as part of the VMware vCloud Enterprise suite.

More information

VMware has a free online eLearning course available on vCAC. Register here.

Viktor van den Berg has a detailed posting on vCloud Automation Center here.

Dailyhypervisor.com has a series of postings on how to install vCloud Automation Center. vcacteam.info also has a lot of information.

• vCloud Automation Center – What to know before you install!
• vCloud Automation Center – vCAC Manager Installation
• vCloud Automation Cetner – DEM Installation
• vCloud Automation Center – Laying the foundation

Randy Stanley, VCDX #94 and blogger at Killerclouds.com has a series of postings on vCAC as well.

• The Unofficial vCloud Automation Center (vCAC) FAQs
• VMware vCAC 5.1 Install Process – Overview
• vCAC – Running Manager and DEM Service in Console Mode
• VMware vCAC 5.1 Prerequisite & Fix Scripts
• VMware vCloud Automation Center 5.1 – How to Get Started
• A Brief Introduction to VMware vCloud Automation Center 5.1 (vCAC)

VMware documentation

YouTube video: vCloud Automation Center Product Overview
vCloud Automation Center Reference Architecture
vCloud Automation Center Operating Guide
vCloud automation Center Frequently Asked Questions

Download via registration here.

Today Veeam announced another feature to be introduced in v7 of Veeam B&R : Virtual lab for replicas. This feature will be available for VMware only for the time being.

Veeam B&R has two methods to protect applications: backup and replication. Backup stores the data in a backup repository. Replication creates another virtual machine in another location which is ready to make operational without a restore.
It is not about the backup or replica, it is about being able to successfully restore the application/data. Since Veeam B&R v5 a feature named ‘SureBackup’  is available which verifies backup jobs. The SureBackup process starts the VM from the backup repository in an isolated network. Veeam B&R full automates verifies if the VM is able to start. The result is written to a logfile, can be sent by email etc.

Veeam B&R v7 introduces a feature named SureReplica which does the same for replicas: verify that a replica of a VM can be made operational successfully. SureReplica verifies if the VM starts, if the operating system starts and if the applications behave like expected. All is reported in a log and by email.

U-AIR for replicas is also a new feature. It allows granular item-level recovery of any application and provides wizards for SQL Server, Exchange and Active Directory. This without the need for agents. U-AIR is already available in current versions of Veeam B&R for virtual machine data stored in the backup repository.

The On demand sandbox allows administrators to test software updates on replicas before putting those on production servers. This is a very efficient, low-cost alternative to having a dedicated infrastructure for testing.

A short video explaining the new Virtual Lab for Replicas can be watched here. 
See also the Veeam blog for more information on this feature.

Error 32010. Failed to create database users. There can be several reasons for this failure.

The logfile vmMSSQLCmd.log revealed that the passwords used for the RSA_user and RSA_DBA where not complex enough.

After setting a more complex password in the screens below the installation continued without any issues.

It is beyond me why VMware does not provide a check immediately after typing in the password. There is a check on complexity of password for the vCenter Single Sign On administrator database useraccount.

This is caused because one of the requirements of vCenter Server install is missing: .NET Framework 3.5

To resolve, go to Server Manager , select the Add roles and features wizard and install .NET Framework 3.5 features.

After completion start the installation of vCenter Server again.

• Infrastructure-as-a-Service Product Line Architecture Fabric Architecture Guide (Windows Server 2012).
  This 112 pages document provides specific guidance for developing fabric architectures (compute, network, storage, and virtualization layers) of an overall private cloud solution. All of the Windows Server 2012 features are discussed.
  Very much recommended reading if your task is to design a Windows Server 2012 Hyper-V infrastructure.
  Download here.
• Infrastructure-as-a-Service Product Line Architecture Fabric Management Architecture Guide (System Center 2012)
  Fabric management is the concept of treating discrete capacity pools of servers, storage, and networks as a single fabric. The fabric is then subdivided into capacity clouds, or resource pools, that carry characteristics like delegation of access and administration, SLAs, and cost metering. Fabric management allows centralizing and automating complex management functions that can be carried out in a highly standardized, repeatable fashion to increase availability and lower operational costs.
  This document discusses sizing, databases used for the various System Center components, requirements like vCPU, RAM and storage for System Center components, ITIL and MOF. It also describes at a functional level features of System Center.
  The document also has a picture showing all System Center connections.
  Download here.

The goal of the Infrastructure-as-a-Service (IaaS) product line architecture (PLA) is to help organizations develop and implement private cloud infrastructures quickly while reducing complexity and risk. The IaaS PLA provides a reference architecture that combines Microsoft software, consolidated guidance, and validated configurations with partner technology such as compute, network, and storage architectures, in addition to value-added software components.

The private cloud model provides much of the efficiency and agility of cloud computing, with the increased control and customization that are achieved through dedicated private resources. By implementing private cloud configurations that align to the IaaS PLA, Microsoft and its hardware partners can help provide organizations the control and the flexibility that are required to reap the potential benefits of the private cloud.

The IaaS PLA utilizes the core capabilities of the Windows Server operating system, Hyper-V, and System Center to deliver a private cloud infrastructure as a service offering. These are also key software components that are used for every reference implementation.

More info at the VMware blog site.

SRM 5.1.1 release notes are here.
vSphere Replication 5.1.1 release notes are here.

What’s New

• Rights for creating, reverting, and removing snapshots: Rights for creating, reverting, and removing snapshots have been added, allowing system administrators to configure these rights for all roles.
• Allocation pool organization virtual datacenters can be elastic or non-elastic: Starting with vCloud Director 5.1.2, system administrators can configure Allocation Pool organization virtual datacenters with Single Cluster Allocation Pool (SCAP), making them non-elastic. This is a global setting that affects all Allocation Pool organization virtual datacenters. By default, Allocation Pool organization virtual datacenters have Single Cluster Allocation Pool enabled. Systems upgraded from vCloud Director 5.1 that have Allocation Pool organization virtual dataceters with virtual machines spanning multiple resource pools have Single Cluster Allocation Pool disabled by default. 

  To change the Single Cluster Allocation Pool setting go to System > Administration > General > Miscellaneous. Before enabling Single Cluster Allocation Pool, migrate any virtual machines on secondary resource pools to the organization virtual datacenter’s primary resource pool.

• vCloud Director is now supported on Red Hat Enterprise Linux 6.3
• Support for Microsoft SQL Server 2012: vCloud Director now supports Microsoft SQL Server 2012 databases.
• Additional guest operating system customization support: vCloud Director now supports customization of the following guest operating systems:
  • Microsoft Windows Server 2012

Get the release notes from here.

Download the update from here.

Storage enhancements of Update 1 are discussed in this great blogpost by Cormac Hogan.
Interesting adjustments are an increased maximum heap size. This allows to have an ESXi host a much larger amount of open VMDK files on VMFS volumes. More info on heapsize and how to increase it for vSphere 5.0 can be read in my earlier posting. 

Also it is again possible to rename VMDK files while performing a storage vMotion. Frank Denneman has written a blogposting about this.

Before performing the upgrade, make sure other VMware components are compatible with Update 1. Viktor van den Berg has a short posting about this here.

Also mind that upgrading VMware solutions needs to be done in a certain sequence. VMware has details in a post titled Update sequence for vSphere 5.1 Update 1 and its compatible VMware products (2037630)

This release  offers the following improvements:

• vCenter Server is now supported on Windows Server 2012
• Additional vCenter Server Database Support: vCenter Server now supports the following databases.
  • Microsoft SQL Server 2012
  • Microsoft SQL Server 2008 R2 SP2
• Additional Guest Operating System Customization Support -vCenter Server now supports customization of the following guest operating systems:
  • Windows 8
  • Windows Server 2012
  • Ubuntu12.04
  • RHEL 5.9
• vCenter Essentials no longer enforces vRAM usage limit of 192 GB With vSphere 5.1 Update 1, the Essentials and Essentials Plus licenses no longer restrict virtual machine power-on operations when the vRAM usage limit of 192 GB is met.
• Resolved Issues – This release delivers a number of bug fixes that have been documented in the Resolved Issues section.

More info here
Release notes here.
Download here

Windows Azure has a preview feature named Azure Backup  which offers an unlimited storage capacity for backup purposes. This post will give some insights in this feature and how to connect to it.

Windows Azure Backup  is currently in Preview. This means it does not have support and a SLA and cannot be used for production purposes.   A backup client needs to be installed on Windows Server 2012 to connect to the vault and make backups and do restores. Support for Microsoft System Center Data Protection Manager is also available although I have not tested it nor found information on this.

Windows Azure Backup has a limit of 850 GB per volume of data that can be backed up in one backup operation.

Installation
There are quite a few blogpostings on how to configure Azure Backup. The basis steps are:
1. enable the Azure Backup Vault in the Azure Management Portal
2. create a certificate. I used makecert.exe which is part of the SDK for Windows 8.
3. download and install the backup client. Add the Windows Azure backup agent in the MMC
4. register the backup server on Azure using the certificate
5. configure a backup job

The most time consuming step for me was trying to download makecert. The other steps are very easy.
Detailed instructions on how to backup to the cloud using Azure Backup:

awsomedevsigner Windows Azure Recovery Services – Now Available!
Trond’s Working! How to set up Azure Recovery Services Backup
Thomas Maurer Using Windows Azure Backup Vault (Preview)
TechMike2KX’s Blog Protecting your online and on premises assets IaaS style!: Introducing #windowsazure Online Backup
James van den Berg System Center blog Installing and Configuring #WindowsAzure Backup agent part 1 of 2 Hybrid #Cloud

Backup client
The backup client has the features listed below:

-include and exclude files and folders
-set a backup schedule (max three backups per day)
-encrypt the backup data
-configure a proxy server
-throttle bandwidth during work hours and/or non-work hours
-set the backup retention. Files deleted in the backup set will be kept available for max. 7 days.

Restore is a very easy process. Basically a next next finish job.

I played a bit with the software in my lab and found it easy and straightforward to use. Hope to give some details on upload/download speed later.

Pricing
Costs are calculated based on the average storage usage per month. Microsoft does not  charge additionally for bandwidth, storage transactions, compute, or other resources associated with providing the Backup service.

Costs depend on the contract. There is a pay as you go pricing model. Here you pay what you consume.Price=  $0.50 per GB per month

Alternatives are 6 or 12-month plans. Here you agree to consume an minimum amount of storage. Pricing per GB is a bit lower than the Pay-as-you-go model.

Windows Azure Backup is currently in preview and Microsoft  provides a 50% discount during the preview period.The first 5 GB per month is free.

Calculate the costs here.

Taken from vopendata.org

vOpenData is an open community project that grew from the question “What is the average VMDK size for deployed virtual machines?” We wanted to create an open community database that is purely driven by users submitting their virtual infrastructure configurations. Leveraging the powerful virtualization community and applying simple analytics we are able to provide various trending statistics and data for virtualized environments. This is 100% community driven and the results will be available for everyone to view and hopefully you will contribute to the overall dataset!

To contribute, you simply have to register into the website and add the infrastructures you have access to. Consultants have the possibility to upload data about their several customers, so they can increase even further the database.

Results are freely visible, even without registration, at this web page, and you can see an example down here.

Below an impression of the data! Looks very cool!

Locations are all over the USA and also in:

Amsterdam, Auckland, London, Melbourne, Ottowa, Singapore, Sydney and Toronto.

Registration , locations and dates are here. 

Archive to Tape in Veeam Backup & Replication v7 supports virtual tape libraries (VTLs), tape libraries and standalone drives.Working with VTLs, tape libraries and standalone drives is easy with v7. As long as the tape device can be seen in Windows Device Manager, it is automatically picked up by the software.

Much more information on tape support at this blog of Veeam Software.

Earlier announced feature for Version 7  is support for VMware vCloud Director Also announced is the plug-in for VMware vSphere Client.

Veeam also announced Veeam Explorer for Microsoft SharePoint

Azure Virtual Machines is an Infrastructure as a Service service model. That means the consumer of the service is responsible for managing the operating system and all software running on that operating system. So far Windows Azure was a Platform as a Service only  service model.  Developers could use the platform to create and execute applications using a fixed set of tooling like SharePoint, SQL server or .NET. If the preferred tool was not made available by Microsoft it could not be used. The consumer was not responsible for managing the operating system.

That changes with Windows Azure Virtual Machines. Customers can now create a guest VM running a selected number of supported operating systems. Now they can run any application they want on that virtual machine. Besides provisioning using an Azure  management portal, virtual machines can also be provisioned using System Center Orchestrator, using PowerShell and using System Center App Controller. Also VHD (VHDX not supported on Azure) virtual disks can be created on-premises on Hyper-V and then uploaded to Azure.

Also via the VM Depot many images are available of Linux distributions preloaded with a lot of different applications. Just a few examples are Drupal, Joomla!, Ruby Stack and WordPress.

To learn more about Azure Infrastructure Services see this short video.

Since the general availability of Windows Azure Virtual Machines some things are improved compared to the Preview:

1. Two additional sizes for virtual machines are added. Available now are 7 sizes including two Monster VM’s. Those 2 monster VM’s have 4 or 8 cores and 28 GB or 56 GB of internal memory.
2. Additional images are available. Images (or templates) contain a pre-installed operating system and software like SQL Server or SharePoint. Supported operating systems are Windows Server 2008, Windows Server 2012, SUSE Linux Enterprise Server, Ubuntu, OpenSUSE and Openlogic CentOS

3. Costs for some of the Azure services are reduced with 21 to 30 percent.

Microsoft today also announced they will follow price reductions of Amazon. So when Amazon lowers prices, Microsoft immediately will do so too.

What not has changed is the availability in the SLA. The SLA states that Microsoft agrees to deliver connectivity to the virtual machine  during 99,95% of the billing month. To meet the SLA the customer needs to have at least 2  instances of a virtual machine running which are part of the same availability set.

Suppose a customer want to runs a legacy application which does not have availability features in the application and does not support clustering. The VM the application is running on  is a single point of failure. When the virtual machine goes down (Microsoft performing scheduled maintenance on Azure involving downtime for the host, host hardware issues, networking issues) Microsoft does not violate the SLA even when 99,95% is not reached. Simply because there is no SLA for a single instance virtual machine.

An Azure Availability set ensures that virtual machines instances are not running in the same fault domain. A fault domain is the same host or the same rack. An availability set makes sure members (virtual machines) of that group never run on the same single point of failure. It is the same as anti-affinity rules in Hyper-V/SCVMM or vCenter Server.

Much more on Azure availability sets can be read here in a post titled Windows Azure Host Updates: Why, When, and How

The Service Level Agreement for virtual machines can be downloaded here.

1. “Maximum Connectivity Minutes” is the total accumulated minutes during a billing month for all Internet facing Virtual Machines that have two or more instances deployed in the same Availability Set.  Maximum Connectivity Minutes is measured from when at least two Virtual Machines in the same Availability Set have both been started resultant from action initiated by Customer to the time Customer has initiated an action that would result in stopping or deleting the Virtual Machines.

Below a nice infographic of Windows Azure. The full version can be seen here at dotnetsolutions.co.uk

A large selection of the breakout sessions presented live in Las Vegas will be available as recording free on Internet. For an overview see Channel 9. 

It is possible to download the video recordings so you can watch them offline.

Stefan Roth wrote a nice script. See his site for a detailed description on how to download.

See for another download script the Infoworks.tv site.

Update April 17: Phil Pritchett made some adjustments to the script of Stefan Roth. This script downloads also the Powerpoint presentations and makes the filename the name of the the breakout session instead of a name like AM-B301.

MyITforum has a posting about this script.

Download the scripts  here.

download MMS2013 sessions

Currently in limited Preview program  is a new service of Microsoft Windows Azure which enables Disaster Recovery Orchestration for Hyper-V. This post will give some details on this new service.

Windows Azure Hyper-V Recovery Manager helps you protect System Center Virtual Machine Manager private clouds in the event of service outage. You can build recovery orchestration plans using this feature and test your plans periodically to ensure high level of preparedness for unplanned downtime or a planned event like data center load balancing.

Introduction
Windows Server 2012 Hyper-V has a nice feature called Hyper-V Replica. It enables  replication of virtual machine disk files to another location for disaster recovery purposes. This feature is available at no additional cost, making DR a lot more cost effective compared to using third party tools. The feature is also very simple to setup and use.

One of the services currently in Limited Preview is part of ‘Azure Recovery Services’ and is named  Hyper-V Recovery Manager Vault. It can be enabled by selection of Cloud Services in the Azure Management portal. You cannot use it immediately. There is a queue at the moment.

Windows Azure Hyper-V Recovery Manager helps you protect System Center Virtual Machine Manager private clouds in the event of service outage. You can build recovery orchestration plans using this feature and test your plans periodically to ensure high level of preparedness for unplanned downtime or a planned event like data center load balancing.

This posting has a short overview of running a lab on Windows client. The conclusion is that VMware Workstation beats Client Hyper-V for nested configurations.

Windows 8 Client Hyper-V allows software developers to test their code on virtual machines they manage. It can also be used by IT Pro’s to test patches, new software or to obtain new skills.

Wouldn’t it be nice to use Windows 8 Client Hyper-V to create a virtual machine running Windows Server 2012, enable Hyper-V on that VM and run nested VM’s on that VM?

Unfortunately that is not possible when using Windows 8 Client Hyper-V. When the Hyper-V role is enabled in the guest VM you will see an error. There is a way to deploy a VM with Hyper-V role enabled using the method described at the bottom of this post. However it does not allow to start nested VM’s on the Windows Server which is a VM itself!

Bart van Keyenberg wrote a posting on how to enable Hyper-V role in a nested VM. His website is offline but the posting is still in the Google cache.

Romeo Mlinar also wrote a blogposting about this topic titled Hyper-V role inside Hyper-V virtual machine He writes:

It is possible to install Hyper-V role inside Hyper-V virtual machine! But (always that BUT) you can’t run virtual machine on virtualized Hyper-V host. You can install Hyper-V role inside Hyper-V virtual machine through PowerShell or Virtual Machine Manager. Of course, this is workaround and isn’t supported! In this blog post I’ll describe how to do that through PowerShell.

What are the alternatives?

Running nested VMware ESX on Windows 8 Client Hyper-V
With quite a bit of work it seems to be possible to run virtual machines on a ESXi virtual machine running on Windows 8 Client Hyper-V. See this posting of Kees Baggerman. Kees describes that he initially tried VMware Workstation but this was slow.

Running nested VMware ESX on VMware Workstation
It is no problem at all to run ESX as a virtual machine using VMware Workstation. It is even possible to run 64 bits guests in a nested ESXi VM. Many posts are available which describe how. See this example.

Running nested Microsoft Hyper-V on VMware Workstation
It is also possible to run Windows Server 2012 with Hyper-V role enabled on VMware workstation. running VM’s in that nested Hyper-V VM is possible. You need to make a few adjustments as described in this post How to run Hyper-V nested in VMware Workstation by Timothy Warner

More information
Vladan Seget wrote a free ebook describing how to setup a vSphere lab on a single computer. Download the e-book here. 

VMware support

VMware does not support running nested ESXi/ESX servers in production environments. This includes, but is not limited to:

• VMware ESXi/ESX running in VMware Workstation or VMware Fusion
• VMware ESXi/ESX running in VMware ESXi/ESX
• VMware ESXi/ESX running in other third-party hypervisor solutions

Issues running ESXi/ESX in a nested configuration fall outside of VMware’s Support and Service Level Agreements. If you experience issues, VMware is under no obligation to acknowledge or investigate immediately or to provide a resolution. However, VMware is interested in obtaining an easily reproducible scenario for our engineers to investigate through discussions in our VMware communities Nested Virtualization forum.

The procedure to install Hyper-V on a Windows Server 2012 VM running on Windows 8 Client Hyper-V is listed below :

However there is a way to avoid this error message and run Hyper-V in a virtual machine. The solution to this issue is the Windows assessment and deployment kit. By using the dism tool you can enable the Hyper-V feature in the installation image of Windows 2012, so that once you install WIndows 2012 Hyper-V is also immediately installed.

The precise steps to enable Hyper-V in a virtual machine running client Hyper-V are:

1. Download and install the Windows assessment and deployment kit for Windows 8 http://www.microsoft.com/en-us/download/details.aspx?id=30652
2. Mount the Windows 2012 installation DVD to your technician computer and copy the contents of this DVD to C:DeploymentWindowssetup
3. Open a command prompt and execute the command

   dism /get-ImageInfo /ImageFile:C:DeploymentWindowssetupsourcesinstall.wim

   Determine which index number your preferred installation has

4. Execute

   dism /Mount-Image /ImageFile:C:DeploymentWindowssetupsourcesinstall.wim /Index:4 /MountDir:C:Deploymentmount

5. Check the features that can be enabled by using the command

   dism /Image:C:Deploymentmount /Get-features /Format:List

6. Enable the Hyper-V feature

   dism /Image:C:Deploymentmount /Enable-Feature:Microsoft-Hyper-V

7. Commit the changes and unmount the image

   dism unmount-image /mountdir:C:Deploymentmount /commit

You now have the choice to put the content of C:DeploymentWIndowssetup on a new DVD and use this DVD as installation DVD or by installing WIndows 2012 over the network by making use of a WinPe disc. I chose the second option

1. Create a WinPe boot DVDMount the created WinPe disk to your WIndows 8 client Hyper-V VM where you will install Windows 2012 and Hyper-V on. Boot the VM from this WinPe disk

   1. On the technician computer open the Windows 8 assessment and deployment kit command-line and execute

      Copype amd64 C:Deployment

   2. Makewinpemedia /iso C:Deployment C:DeploymentMyOwnPe.iso

3. net use R: \technician_computer_nameWindowssetup

4. setup.exe

5. Install Windows 2012 by following the installation instructions

6. Once Windows 2012 is installed Hyper-V is also enabled

This posting will give the reader some insight in how to avoid those issues by adjusting settings and apply a recent published VMware patch.

An increasing number of organizations are using hosts with serious amounts of  physical memory and compute power to be able to run many VMs per host. VM’s having large amounts of memory  and virtual CPU’s are called monster VM’s.

However those monsters can easily be made unconscious when those VMs are running a couple of large VMDK’s. For instance you  might run into issues when running multiple virtual machines like Exchange Servers, file servers  or database servers with large VMDK’s on the same ESX host.

In a default installation each VMware vSphere4.1 and  5.x host can address a maximum of 8 TB of open VMDK files on VMFS-5 volumes. So in case the sum of all active VMDK’s on a given ESX host is over 8 TB you will run into issues. See later in this posting what kind of issues.

This is caused by the size of the VMFS heap in the vmkernel.

The maximum heap size in ESX4.1 /ESX5.x is 128 MB in a default installation. The maximum heap size has been further increased in ESXi 5.0 Patch, ESXi500-201303001 to 640MB, which should allow for 60TB of open virtual disk capacity on a single ESX/ESXi host. This patch was released by VMware at March 28, 2013.

What is VMFS heap size?

VMFS heap is  part of the host’s physical memory in use by the kernel which is reserved for  file handling of VMFS volumes. The heap memory contains pointers to data blocks on VMDK files on VMFS volumes. I could not find more info on heap size.

Heap size is set in the advanced settings of the ESX host. The default setting of VMFS3.MaxHeapSizeMB in vSphere 4.1 & 5.x is such that  on a given ESX host a maximum of 8 TB of VMDK files located on VMFS5 formatted volumes can be opened. This restriction does not apply to RDM of NFS.

Reported issues when running out of heap size

Various issues are reported when heap size is to low. It depends on the action performed. “Cannot allocate memory”  seems to be the most displayed error when heap size is full.

1. Errors when performing a vMotion: A general system error occurred: Source detected that destination failed to resume.
2. Virtual machines do not start. Errors shown are (for example in vmkernel.log)
   The VM failed to resume on the destination during early power on.
Reason: 0 (Cannot allocate memory).
Cannot open the disk '<<Location of the .vmdk>>' or one of the snapshot disks it depends on.
vSphere HA unsuccessfully failed over this virtual machine. vSphere HA will retry if the maximum number of attempts has not been exceeded. Reason: Cannot allocate memory.
WARNING: Heap: 2525: Heap vmfs3 already at its maximum size. Cannot expand.
   WARNING: Heap: 2900: Heap_Align(vmfs3, 2099200/2099200 bytes, 8 align) failed. caller: 0x4180368c0b90
   An unexpected error was received from the ESX host while powering on VM vm-xxx. Reason: (Cannot allocate memory)
3. Write errors reported in Windows guests when copying large amounts of data from and to large VMDK files
4. Errors running Microsoft Exchange Server Jetstress
5. Windows reporting lack of storage capacity.
6. Issues reported in vSphere Client when performing storage migrations (storage vMotion).
   A storage vmotion of a large VMDK fails at 10%. A storage vMotion of a smaller VMDK completes with no issues.
   Relocate virtual machine <virtual machine name>. A general system error occurred: Storage VMotion failed to copy one or more of the VM ‘s disks.  Please consult the VM’s log for more details, looking for lines starting with “SVMotion”.
   “Failed to create one or more destination disks. Canceling Storage vMotion.
   Storage vMotion failed to create the
   destination disk /vmfs/volumes/….
   (Cannot allocate memory). “
7. Issues while performing a backup. Backup-applications mount VMDK files. If during a backup job each time a large VMDK is added the heap size maximum could be reached.

In /var/log/vmfs/volumes/DatastoreName/VirtualMachineName/vmware.log you will see  Cannot allocate memory errors listed.

Heap size and maximum active VMDK files

The default heap size in ESXi/ESX 3.5/4.0 for VMFS-3 is set to 16 MB. This allows for a maximum of 4 TB of open virtual disk capacity on a single ESXi/ESX host if a blocksize of 1 MB is set. If a blocksize of 8 MB is set, the maximum is 32 MB

The default heap size has been increased in ESXi/ESX 4.1 and ESXi 5.x to 80 MB, which allows for 8 TB to 10 TB of open virtual disk capacity on a single ESXi/ESX host. .

In ESXi 5.x, the maximum heap size is 256 MB without hotfix. This allows 25TB to 30 TB of open files/VMDKs can be addressed by a single ESXi host

The default heap size has been further increased in ESXi 5.0 Patch, ESXi500-201303001 to 640MB, which should allow for 60TB of open virtual disk capacity on a single ESX/ESXi host.

To make an adjusted  heap size active the ESX host needs a reboot!

How to check active heap usage

The usage of the heap memory can be checked by the command below given from the CLI.

~ # vsish /> cat /system/heaps/vmfs(hit Tab key to auto fill the rest of this)/stats   (hit Enter)

For example the output might look like this.

Percent free of max size (87 in example) tells you the actual free heap size. Lowest percent free of max size ever encountered:84 . If this value is around 20 it is time to increase the heap size and.or apply the VMware hotfix ESXi500-201303001.

How to adjust the heap size

Solving or preventing issues like reported above are simple: adjust on each ESX host the VMFS3.MaxHeapSizeMB setting, reboot the host and done. For ESX(i) 4.x and 5.x the default value is 80. That means 80 MB of host memory is reserved for heap. Increase it to the maximum setting of 256 MB. This will ‘cost’  176 MB of internal memory on the host. This is a no-brainer, 176 MB of internal memory will cost nothing compared to the possible issues caused by the default value of  VMFS3.MaxHeapSizeMB.

To adjust the value:

1. Log into vCenter Server or the ESXi/ESX host using the vSphere Client or VMware Infrastructure (VI) Client. If connecting to vCenter Server, select the ESXi/ESX host from the inventory.
2. Click the Configuration tab.
3. Click Advanced Settings.
4. Select VMFS3.
5. Update the field in VMFS3.MaxHeapSizeMB.

Although the setting is named VMFS3.MaxHeapSizeMB this is still valid for VMFS5 as well!

VMware has a KB-article titled An ESX/ESXi host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 TB of virtual disk storage (1004424)

See this VMware KB article for more information on adjusting the heap size.

The future of VMFS and heap

VMware should adjust both the maximum size of a VMDK file and solve the maximum of active virtual disks per host. Competitor Hyper-V has a maximum of 64 TB per VHDX virtual disk file. I do not know if there is a maximum amount of active virtual disks per Hyper-V host.

This posting by VMware employee and storage expert Cormac Hogan hints on improvements on maximum VMDK size and improvements in the maximum amount of open VMDK files per host.

Additional information

http://blogs.vmware.com/vsphere/2012/08/vmfs-heap-considerations.html

http://bizsupport1.austin.hp.com/bizsupport/TechSupport/Document.jsp?prodSeriesId=3690376&prodTypeId=18964&objectID=c03216142

http://www.boche.net/blog/index.php/2012/09/

http://vmnick.com/2012/12/20/vmfs-5-heap-size/

http://longwhiteclouds.com/2012/09/17/the-case-for-larger-than-2tb-virtual-disks-and-the-gotcha-with-vmfs/

http://virtualkenneth.com/2011/02/15/vmfs3-heap-size-maxheapsizemb/

For an overview of blog articles explaining those new features see this post.

This poster for VMM in System Center 2012 SP1 can help you:

• Plan your VMM networks using a logical view of VM networks, logical networks, and logical switches using a variety of configurations including VLAN-based configuration, no isolation, network virtualization, external networks, and with no virtual networking.
• Configure networking in VMM using configuration steps by roles including fabric administrators, tenant administrators, or by any user.
• Understand the network object model with diagrams of objects showing the relationships between objects.
• Extend VMM with options including using a vendor network-management server with extensions, connect a VM network to other networks by configuring the VM network with a gateway, and load-balance requests to VMs that make up a VMM service tier by adding a load balancer to VMM.

download the poster here.