version 1, published March 1 2011
This posting is one of a serie of postings meant to give an understanding about design and procedures of Veeam Backup & Replication v5.
other postings are:
Know the performance impact of snapshots used for backup
Veeam Backup & Replication v5 (Veeam B&R) offers protection of VMware virtual machines. It has some unique features which can be used to automatically verify if the backup can successfully be restored to a running virtual machine (SureBackup). It also has wizards to restore Active Directory objects, SQL and Exchange Server items and has the ability to restore items from any application.
Veeam is not able to protect physical servers nor can it be used to manage a tape library. For these functions you will need additional tools like Microsoft DPM or Windows Server Backup.
This article will give design guidelines and best practices for your design for a Veeam B&R v5 installation. This posting is part of a series of two postings on design. Part 1 will focus on
- backup strategy
- licencing and pricing of Veeam B&R
- available editions
- backup modes
- the pro’s and cons of installing Veeam B&R on a physical server or virtual machine
- networking being used for backup and restore traffic
- sizing of backup storage
- SureBackup and Virtual Lab
In part two I will focus on virtual lab, reporting and application item recovery.
Information used in this posting is taken from the Veeam forums and from personal experience installing and using the software.
Step 1: Think about the backup and restore strategy
Before starting to actually double-click the executable which installs Veeam Backup & Replication some time needs to be spent on thinking about a backup strategy. Input for the strategy comes partly from the business, from the IT-department and from the knowledge and experience of the consultant implementing Veeam B&R.
Ideally the business is asked for Recovery Time Objective(RTO) and Recovery Point Objective (RPO). RTO indicates the maximum amount of time allowed to restore an application. RPO indicate the amount of data that can be lost without causes too many problems for the business. Sometimes it can be difficult to get info on RTO and RPO so the IT-staff will need to make an assumption based on their knowledge of the business needs.
Then it is important to understand the features of Veeam B&R and see if it fits with requirements. One of the interesting new features of Veeam B&R v5 is Instant Recovery. It enables to have one or more virtual machines to be run from backup storage. No need to transfer files from backup to original location (like a traditional restore). No needs to perform a regular (yearly) recovery test . Instant Recovery enables a much lower Recovery Time Objective. In the setup of the backup storage you should think about the type of disks and fault tolerance level. If your instant recovered virtual machines should deliver performance the business is used to , even during recovery, you might want to use some more expensive , faster tier1 disks for your backup storage. You can easily create several tiers of backup storage and assign different drive letters to it.
Also think about how long backup data needs to be available on backup storage, if tapes are going to be used, if replication to an offsite location is needed, the amount of growth of data etc. etc.
A solid, well thought over design will prevent having to adjust the design and the configuration shortly after it was finished.
One of the more complex features to install is the virtual lab. To be able to ping the server which is running off the backup, an isolated network needs to be created. Each network having virtual machines to be verified needs to be defined in the virtual lab. An insight in the network layout is a handy. Take your time to test and get familiar with this feature.
Make sure to read the Frequently Asked Questions document http://www.veeam.com/forums/viewtopic.php?f=2&t=5304
Available editions of Veeam B&R
Veeam Backup & Replication v5 is available in two editions. The difference in the two editions can be seen in the Veeam whitepaper published here.
One of the features missing in the standard edition is Universal Application Item Recovery. The enterprise edition has wizards to recover objects from Active Directory, SQL (still beta in Feb 2011) and Exchange Server(still in beta Feb 2011) and has the ability to recovery items from any other application. This feature is based on the ability of Veeam B&R to run a virtual machine from backup storage in an isolated network environment. Using the standard edition you are limited to extract any file from the backup, but not application items.
Also the standard edition lacks the ability to automatically verify if a backup can be successfully restored to an running virtual machine. In the enterprise edition predefined scripts can be run to ping network interfaces or probe specific network ports (like port 389 for AD). In the standard edition verification needs to be performed manually.
Licensing and price
Veeam needs to be licenced by the number of CPU sockets in the ESX hosts it protects. If your protected VM’s are running on 10 ESX hosts, each host having 2 CPU sockets, you need 20 CPU licences. Veeam has two tiers in pricing: Tier A is for CPU’s with 1 to max 6 cores. Tier B is for CPU’s with 7-12 cores. There are licences for commercial, public sector and education use.
The listprice of a single, 1-6 cores license for commercial use of the Veeam B&R standard edition is around EURO 527.
The listprice of a single, 1-6 cores license for commercial use of Veeam B&R enterprise edition is around EURO 791,-
Install Veeam B&R on a physical server or virtual server
Veeam B&R can be installed on a physical server and on a virtual server.
The advantage of installing on a physical server is that backup storage can be directly attached delivering the best throughput. Also attaching a tape library can be done. The disadvantage are additional costs for server hardware, power, cooling, maintenance and Window server license cost. All the typical costs that are reduced when using virtual machines. As virtual appliance mode is not available , you will need to present all LUNs with datastores to your Veeam server, which is a bit more work. Another disadvantage can be the risk of damaging your VMFS volumes. It could be that a system administrator not aware of the impact tries to mount the VMFS volumes.
The advantage of using a virtual machine to install Veeam on is reduced hardware and operational costs compared to a physical server. If you are using FiberChannel to connect shared storage, direct SAN access is not possible. However, the virtual appliance mode seems to deliver the same backup speed as Direct SAN access according to Veeam.
The disadvantage of a virtual machine can be it is not available when you need it. Suppose your SAN has a corrupted LUN and the Veeam virtual disks are located on that LUN. Catch 22! By installing Veeam on a physical server, or on a virtual server outside the storage it’s protecting it will have a much higher chance to be available when disaster strikes.
Scalability of using virtual machines is better than physical. For large environment where Veeam needs to protect hundreds of virtual machines it could be a single instance of Veeam is reaching it maximum. By installing additional Veeam virtual machines it might be possible to get a higher throughput. I have not seen many facts on this.
Veeam B&R is able to perform backups’ using three modes:
- Direct SAN Access. This method is available on Veeam B&R servers installed on a virtual machine (iSCSI only) and a physical server (iSCSI and FiberChannel). LUNs are presented to the Veeam backup server. Double check automount is disabled on the Veeam server or make the LUN readonly for the Veeam server!
- Virtual appliance. This method is available on Veeam B&R servers installed on a virtual machine. Virtual disks of backup sources are attached to the Veeam server using SCSI hot add. The read performance is equal to the Direct SAN access. It is the recommended mode when using Veeam on a virtual machine.
- Network mode. Data is read over the network via the ESX-host. This is the slowest mode available and should be used with caution. Read the section labeled Networks why.
Virtual appliance mode has some restrictions. A backup of an IDE disk is not possible. It a virtual machine has two virtual disk files with the same name the job will fail. For an overview see this posting on the Veeam forum
Hardware specifications of backup server
The hardware specifications for the Veeam Backup server are relatively light. 4 cores or 4 virtual CPU’s when installed on a virtual machine are recommended. A minimum of 1 GB internal memory (2 GB when using a local SQL database) and 100 MB disk storage and a 1Gbs network interface. While the amount of internal memory I/O performance. The faster the memory, the better performance on data de-duplication and compression. This is the reason why Veeam on virtual machines can outperform physical servers as memory on ESX hosts is often faster than for regular physical servers used for 1 instance of Windows Server.
I recommend to use at least two network interfaces. One interface is used for management (RDP to the server, active directory traffic, DNS etc). The other interface is dedicated for backup and restore traffic. This way, backup/restore data cannot overload the management network.
Direct SAN Access
Direct SAN access has one caveat: there is a potential risk the VMFS volume is re-signatured by Windows. To prevent this two solutions are available.
First, you could set access to the LUN’s hosting VMware datastores to readonly for the Veeam backup server. However, not all storage arrays have this feature. Dell EqualLogic and HP EVA lack this usefull feature. A LUN on these array’s can on set for read only for all hosts having access to it.
Another solution to prevent Windows server mounting and signaturing VMFS volumes is by using diskpart and set ‘automount’ to ‘disable’. Veeam B&R version 5 will automatically set automount to disabled. After that, present the VMFS LUNS to the Veeam B&R server. The disk will be shown in Windows disk management as ‘on-line’ and ‘healthy’. However, actions like Change Drive letter are not available.
Accidently setting automount to enabled is something to be prevented. Murphy will strike at some time. For example to able to install Windows Server 2008 R2 SP1, automount needs to be enabled. A Windows Server admin not knowing the effect of automount enabled could easily enable it for installing SP1. The results can be dramatic.
To prevent such a disaster, you can rename the diskpart executable to a different name. You can also display a warning at logon on the server explaining never to enable automount before disabling the connection to the SAN storage. This can be done for example by using the local security policy of the Veeam server and set ‘interactive logon: message text for users attempting to log on’
There is a limit on the number of LUN’s you can attach to the Veeam Backup server, however it is not very likely you will run into that maximum. The maximum is:
8 buses per adapter , 128 target IDs per bus, 255 LUNs per target ID for hardware maximums. Windows Server allows approximately 2,000 volumes per server to be attached.
Veeam cannot restore directly to the SAN using the SAN fabric. All restore data will leave the Veeam B&R server over a network interface, will go into the destination ESX host VMkernel interface towards the storage stack. Restores are therefore much slower than Direct SAN Access backup and Virtual appliance backup. According to Veeam restores to NFS volumes are faster than restores to datastores formatted with VMFS.
Veeam has a nice feature to solve the slow restore speed called Instant Recovery. This feature enables a virtual machine to be run from the backup storage. The backup storage is mounted to a ESX host as a NFS datastore. An instant recovery can best be compared to changing a flat tyre of a car while the car remains running.
A ESX host will have several network interfaces used for management, virtual machine traffic, vMotion traffic etc. For Network mode backup and Restore Veeam will use the network to receive and sent data. Make sure the correct network is used. It makes sense to create a dedicated network for backup and restore traffic. That way the management network cannot be overloaded. See the management network as an emergency lane of the highway. It must be kept free at all times except for maintenance and emergency purposes. The management network is used for heartbeats of VMware clusters and remote control.
To prevent Veeam using the management network for backup and restore and for mounting the NFS datastore, several steps needs to be taken.
I assume a dedicated network for backup is already in place, used to create file level backups or image level backups . For file level. each virtual machine has a dedicated virtual network adapter attached to a virtual machine portgroup. All backup traffic is routed over this adapter.
Firstly, DNS servers are probably pointing to the management IP-address of the ESX servers. To force Veeam using the IP-address of the VMkernel port dedicated for the backup network, create a hosts file on the Veeam server pointing to the IP-address of the backup network.
For an instant recovery, the backup storage of the Veeam server will be presented to a specified ESX host. You do not want NFS storage traffic to run over the management network. Initially after setup the ESX host will use the management network for this purpose. To force NFS traffic over the backup network, two actions can be taken. Firstly the host file of the ESX host used for instant recovery can be adjusted such that the netbios name of the Veeam server points to the IP-address of the network interface of the Veeam backup server.
As an alternative, you can manually remount the NFS datastore. Follow the procedure listed below:
1. start vSphere Client, select ESX host, configuration, storage, Add Storage, NFS.
2. Server: enter Veeam Backup server’s IP address used for backup network which ESX can see.
3. Folder: enter /VeeamBackup_SERVERNAME (where SERVERNAME is NetBIOS name of Veeam Backup server).
4. Do not select Mount NFS read-only checkbox.
5. Datastore name: enter VeeamBackup_SERVERNAME (no forward slash here).
The picture below shows the ports used by Veeam B&R.
Backup data created by Veeam can be stored on several storage locations.
- If using a physical server for Veeam, it can be stored on local storage attached to the Veeam server and accessible by Windows Server. It can also be stored on a Windows share and accessed over the network (less fast obviously)
- If using a virtual server, it can be stored on a virtual disk located at a datastore at local storage or shared storage (SAN/NFS)
- It can be stored on a datastore of an ESX host by using the replication feature of Veeam.
- It can be stored using a Storage Cloud provider and a virtual appliance gateway (Twinstrata)
As a number one rule I would advise to store the backup on a different storage platform than the storage platform which is being protected. Suppose a LUN on the SAN gets corrupted you do not want the backup data be unavailable as well.
Sizing of Backup storage capacity
You should make sure the capacity of backup storage is enough to meet the requirements on Recovery Time Objective and Recovery Point Objective.
A calculation taken from the Veeam website is listed below. Roughly for a retention period of 14 days, you should have 75 % of actual data usage storage available on the backup storage.
Backup capacity = C * (F*Data + R*D*Data)
Replica capacity = Data + C*R*D*Data
Data = sum of processed VMs size (actually used, not provisioned)
C = average compression/dedupe ratio (depends on too many factors, compression and dedupe can be very high, but we use 50% – worst case)
F = number of full backups in retention policy (1, unless backup mode with periodic fulls is used)
R = number of rollbacks (or increments) according to retention policy (14 by default)
D = average amount of VM disk changes between cycles in percent (we use 10% right now, but will change it to 5% in v5 based on feedback… reportedly for most VMs it is just 1-2%, but active Exchange and SQL can be up to 10-20% due to transaction logs activity – so 5% seems to be good average)
SureBackup is an automated verification of the backup. What it does is starting de-duplicated and compressed virtual machines from the backup storage. The backup storage is mounted as a NFS-datastore to an ESX host. To prevent duplicate IP-addresses and hostnames, the virtual machines are run in an isolated network. A proxy appliance installed by Veeam is used to connect the production network(s) with the isolated network(s). Veeam will use three methods to check if a virtual machine backup can be successfully restored:
- Check if heartbeat is available on the restored, running vm.
- Boot verification : Check if the host can be pinged from the Veeam backup server
- Application verification: Veeam provides a number of predefined scripts. Those scripts perform a check on an application specific network port. To test if Active Directory is running, Veeam B&R will exectute a tool called VmConnectionTester.exe which will probe port 389 for a response. It is also possible to create your own verification scripts.
The image below shows the virtual lab and the virtual appliance making sure virtual machines in the virtual lab do not interfere with production virtual machines.
So far part 1 of this serie on Veeam B&R v5. In the next part more information on performance and lots more.