HP BL460c Gen 9 with Windows Server 2012 R2 no network or intermittent loss of network connection

One of my customers requested me to have a look at their Microsoft Hyper-V Server 2012 infrastructure. They had a strange problem with the HP BL490c Gen 9 blades. The blades are equipped with a 536FLB Flex 10 network interface card based on a QLogic 57840S chipset.

The Hyper-V hosts are deployed using the bare-metal deployment in System Center Virtual Machine Manager 2012 R2.

The customer has 6 blades in a C7000 enclosure. Blade 1 could not ping to Blade 2. However blade 1 could perfectly ping to Blade 3,4,5 and 6. All blades are in the same IPv4 subnet. Another issue observed on a few other blades was that the ping would time out a few times. So ping worked, then it did not work for a couple of pings and then network was okay again.

The customer has tried many things to solve this issue:

placed the blades in a different HP C7000 enclosure.
placed the blades in different slots of the enclosure
swapped the HP switches
installed full Windows Server 2012 R2 GUI.
installed Hyper-V Server 2012 instead of R2
disabled checksum offloading , LSO, RSS, RSC and VMQ
called HP. HP was not able to solve this issue.

This all did not solve the issue. The customer was using the latest drivers for the adapter.

HP Virtual Connect 4.31 dated 2014-10-24
HP FlexFabric 10Gb 2 port 536FLB adapter drivers 7.10.39
HP software package 4.01.12

I deciced to start troubleshooting this from scratch by manually installing Hyper-V 2012 R2. The blade which initially could not ping to Blade 2 now was able to ping. Hmm, wondering why.

Next I installed Hyper-V 2012 R2 manually (no network configuration) and then manage it using SCVMM. SCVMM configured the networking of the host. Still the host was able to ping all other hosts in the same enclosure.

So for some reason when the customer is using the bare-metal deployment from SCVMM something goes wrong with networking.We are not sure what exacty. It could be related to BIOS changes. It seems that as soon as the BIOS of the blade is changed, the next bare-metal deployment will result in network issues.

I will update this post when we know more about the cause of this issue.

HP releases drivers for Emulex 10 GbE nic solving virtual machine disconnect when VMQ is enabled

Update September 15:

Someone reported that after installing this update on a HP Gen7 the problem still exists.

Hyper-V virtual machines running on servers with Emulex network interface cards can randomly disconnect from the network when VMQ has been enabled. This is experienced by many. The issue both occurs on Emulex cards as well as on OEM Emulex cards. HP servers and some others use those Emulex 10 GbE cards.

See my earlier post for more info.

Today HP released a new driver which solves this issue. The driver is included in the Service Pack for Proliant 2014.09.

However Emulex advised odd ping may still drop but no disconnects. In the near future Emulex will do a complete rewrite of the driver to also fix the dropped pings.

Tim Johnson was so kind to keep me informed about this update. He told me after installing this driver update he did not experience disconnects for the last 7 days.

5nine Cloud Security 4.0 for Hyper-V released

5nine Cloud Security for Hyper-V is the first and only agentless anti-malware and virtual firewall solution for the Windows Server Hyper-V, utilizing the flexibility of Hyper-V Extensible Switch. It offers some unique features like support for NVGRE. The solution is interesting for service providers who like to protect their customer virtual machines but do not have access to the guest operating system.

Cloud Security for Hyper-V is the new name of 5nine’s Security Datacenter for Windows Server Hyper-V. See the datasheet for detailed info.

Cloud Security is available as a free edition (with limitations) and a paid edition.
Download the free edition here.

Enterprises and cloud providers can:

Secure multi-tenant Hyper-V environments and provide VM isolation
Protect Hyper-V with light-speed agentless antivirus
Enforce PCI-DSS, HIPAA and Sarbanes-Oxley compliance

5nine Cloud Security 4.0 for Hyper-V features:

Secure multi-tenancy and VM isolation
Virtual Machine security groups and cloud tenant security
User/roles access that allows users or user groups to manage only objects associated with them
Agentless antivirus for Hyper-V hosts and real-time protection for VMs
NVGRE support
New LWF R2 vSwitch extension
Enhanced API and advanced event logging

Licensing options

Standard license is available for fewer than or equal to 10 VMs per 2 CPU.
Datacenter license is available for unlimited number of VMs per 2 CPU.
SPLA license is available for hosting providers upon request.

Thomas Maurer has written a comprehensive blogpost about 5nine here.

How to create a site-to-site VPN connection using ADSL to Windows Azure

For research on my to be released book on Windows Azure I had to create a site-to-site VPN connection from my home to Windows Azure. Untill recently I was under the impression I needed a VPN device or Windows RRAS server configured with a public facing IP-address to be able to have such a site-to-site VPN.

However, that is not the case. Using a common ADSL modem, Hyper-V manager and a virtual machine running Windows Server 2012 with RRAS I was able to setup the VPN connection.

Thanks to Christopher Keyaert who blogs at vnext.be who helped me. Read his blog which describes how to update Azure networking if your ADSL connection has a dynamic IP.

My ADSL modem is a Fritz!Box 7270. I did not have to modify the configuration of the modem. You might want to add a route in your modem pointing to your RRAS server if other servers need access to Azure VMs.

The site-to-site can be setup using a physical server with RRAS installed as well. No need for the RRAS server to have a public IP.

In my book I will publish a step by step instruction how to configure this. In this post I will provide the basic steps. There are many other posts explaining how to setup a site-to-site VPN connection. For example this one.

1. In the Azure Management Portal create a virtual network. First create a new local network. In here you configure the public IP-address which is assigned to your ADSL modem. You also specify the IP-subnet used in your home location. Mine is 192.168.178.0/24.

2. Enable ‘configure site-to-site VPN’.

3. Then create a gateway in the portal. Select dynamic routing. Creation of the gateway will take about 5 to 10 minutes.

4. After the creation has finished, select ‘Download VPN device script. Choose Windows Server RRAS and store the .cfg file on your RRAS server.

5. Rename the .cfg file to PS1. Start PowerShell and execute the .PS1 file. You might have to change the execute policy .

The PowerShell script adds a Network interface to the RRAS server. This connects to the IP-address of the Azure gateway. When the script has finished open Routing and Remote Access console. Select Network Interfaces-> then select the demand dial connection named as IP-address of the Azure gateway. Right click and select Connect.

If all goes well a VPN connection is enabled.

Make sure the Ethernet network interface of the RRAS server which connects to your internal (home) network does not have a gateway filled in for the IP-properties. Otherwise ip-traffic will not flow to and from Windows Azure.

Also make sure the firewall on the RRAS server does not block VPN-traffic.

In Windows Azure create a virtual machine and make sure it is added to the virtual network you created in the first step. After creation has finished, open an RDP connection. Then make sure the Windows Server firewall does not block VPN traffic.

That is it. You now should be able to ping or use any other connection from your home server (RRAS) to a virtual machine in Azure.

Please let me know if you have issues in setting up the S2S VPN (mvdb22 at gmail dot com )

Guide: How to sync on-premise Active Directory to Windows Azure Active Directory

Microsoft released a Test Lab Guide which explains in detail how to synchronize an organization Active Directory with Windows Azure Active Directory.

Organizations moving to a hybrid cloud want to be able to provide identity management for services running on Windows Azure whil not depending on their on-premise Active Directory.

By using Windows Active Directory Synchronization Tool (DirSync) on-premise AD can be synchronized to Windows Azure Active Directory.

This 48-pages documents explains step by step how to sign up for a free trial of Azure, how to enable WAAD and how to setup and configure DirSync.

Since November 2013 DirSync can be installed on a server with Active Directory installed. So only a single DC is needed to be able to use this Lab Guide.

More information and download of the guide here.