New HP VC Cookbook for iSCSI

Even though its been here for such a long time, iSCSI SANs still have not convinced everyone. Though I do not want to go through that discussion, every help one can get on recommendations how to set it up is helpful. HP just released its “HP Virtual Connect with iSCSI“, so here are some notes about it.

  • Array supportability: Any Array with iSCSI ports.
  • You can use Accelerated iSCSI (and NIC behaves like an iSCSI HBA), if you choose Blade NICs that support so. Max MTU size in such scenario is limited to 8342 bytes, unchangeable. Both on Windows and VMware you have no control, as this is automatically negotiated (I am still refering on Accelerated iSCSI option). On Windows Box, when TCPMSS displays 1436, the MTU negotiated is 1514, and when it displays 8260 the MTU negotiated is 8342.
  • Note that though you can directly connect Storage Host ports to Virtual Connect (except with HP P4000 or any other IP-Clustered Storage), I would definetely not do so, since it limits your options. VC does not behave as a Switch in order to prevent loops and simplify Network management, so traffic will not be forwarded out of the Enclosure (say for instance if you had another Rack Server Host wanting iSCSI). This is also the reason you cannot setup more than one box of HP P4000 to it, since you do require inter box communication for metadata and cluster formation.
  • Do not forget iSCSI basics: Keep it L2, enable 802.3x Flow control everywhere (Storage Ports, Switch ports, Virtual Connect uplink ports (enabled by default on downlink ports, and Host ports if using Software iSCSI) enable Jumbo Frame support on OS level if you are using Software iSCSI (instead of Accelerated iSCSI), configure iSCSI multipathing when possible (either by using the Storage vendor DSM, or if possible the OS own MPIO), dedicate at least an isolated VLAN or with dedicated physical devices (Switches), and finally isolate at least a FlexNIC on the Host for iSCSI traffic.


This figure illustrates a mapping of VC config from the Blade’s FlexNICs, to interior of VC (two SUS with corporate VLANs plus dedicted vNet for iSCSI VLAN, and exterior VC uplinks. For redundancy reasons and to limit contention between Downlinks and Uplinks, one would recommend using at least two Uplink Ports on each VC module dedicated to the iSCSI vNet. Whenever a VC module has a one-to-one relationship towards the upstream ToR switch (or several ToR switches when using MLAG, such as HP IRF, Cisco Catalyst VSS or Nexus vPC, etc). When configuring the iSCSI vNets, make sure you enable “Smart Link” feature, to ensure for faster failovers.

You might also want to take advantage of new VC firmware 4.01 features, which include two helpful ones. The first is flexible overprovision of Tx Bandwidth. Ensure a minimum pipeline for that iSCSI Host port, and let it fly (by assigning higher Max Bandwidth) if other pipes are empty on other FlexNICs.

Finally, the second concerns QoS. You might want to use a dedicated 802.1p class for your iSCSI traffic, which is mapped for Egress traffic.

Setup for VMware, HP Virtual Connect, and HP IRF all together

Trying to setup your ESXi hosts networking with HP Virtual Connect (VC) to work with your HP’s IRF-Clustered ToR switchs in Active-Active mode?


This setup improves performance, as you are able to load balance traffic from several VMs across both NICs, across both Virtual Connect modules, and across both ToR switchs, while in the meanwhile gaining more resilience. Convergence on your network will be much faster, either if a VC uplink fails, a whole VC module fails, or a whole ToR fails.

Here’s a couple of docs that might help you get started. First take a look on the IRF config manual of your corresponding HP’s ToR switches. Just as an example, here’s the config guide for HP’s 58Xo series. Here are some important things you should consider when setting up an IRF cluster:

  1. If you have more than two switches, choose the cluster topology.  HP IRF allows you to setup a daisy-chained cluster, or in ring-topology. Naturally the second one is the recommended for HA reasons.
  2. Choose which ports to assign to the cluster. IRF does not need specific Stacking modules and interfaces to build the cluster. For the majority of HP switchs, the only requirement is to use 10GbE ports, so you can flexibly choose whether to use cheaper CX4 connections, or native SFP+ ports (or modules) with copper Direct Attach Cables or optical transcevers. Your choice. You can assign one or more physical ports into one single IRF port. Note that the maximum ports you can assign to an IRF port varies according to the switch model.
  3. Specify IRF unique Domain for that cluster, and take into consideration that nodes will reboot in the meanwhile during the process, so this should be an offline intervention.
  4. Prevent from Split-brain Condition potential events. IRF functions by electing a single switch in the cluster as the master node (which controls Management and Control plane, while data plain works Active-Active on all nodes for performance). So if by any reason stacking links were to fail, slave nodes by default initiate new master election, which could lead into a split-brain condition. There are several mechanisms to prevent this front happening (called Multi-Active Detection (MAD) protection), such as LACP MAD or BFD MAD.

Next hop is your VC. Whether you have a Flex-Fabric or a Flex-10, in terms of Ethernet configuration everything’s pretty much the same. You setup two logically identical Shared Uplink Sets (SUS) – note that VLAN naming needs to be different though inside the VC. Next you assign uplinks from one VC module to one SUS and uplinks from another different VC module to the other SUS. It is very important that you get this right, as VC modules are not capable of doing IRF withemselves (yet).

If you have doubts on how VC works, well nothing like the VC Cookbook. As with any other documentation that I provide with direct links, I strongly recommend that you google them up just to make sure you use the latest version available online.

If you click on the figure above with the physical diagram you will be redirected to HP’s Virtual Connect and IRF Integration guide (where I got the figure in the first place).

Last hop is your VMware hosts. Though kind of outdated document (as Vmware already launched new ESX releases after 4.0), for configuration purposes the “Deploying a VMware vSphere HA Cluster with HP Virtual Connect FlexFabric” guide already allows you to get started. What you should take into consideration is that you will not be able to use the Distributed vSwitch environment with new vmware’s LACP capabilities, which I’ll cover in just a while. This guide provides indications on how to configure in a standard vSwitch or distributed setup on your ESX boxes (without using VMware’s new dynamic LAG feature).

Note that I am skipping some steps on the actual configuration, such as the Profile configuration on the VC that you need to apply to the ESX hosts. Manuals are always your best friends in such cases, and the provided ones should provide for the majority of steps.

You should make sure that you use redundant Flex-NICs if you want to configure your vSwitchs with redundant uplinks for HA reasons. Flex-NICs are present themselves to the OS (in this case ESXi kernel) as several distinct NICs, when in reallity these are simply Virtual NICs of a single physical port. Usually each half-height HP Blade comes with a single card (the LOM), whcih has two physical ports, each of which can virtualize (up to date) four virtual NICs. So just make sure that you don’t choose randomly which Flex-NIC is used for each vSwitch.

Now comes the tricky part. So you should not that you are configuring Active-Active NICs for one vSwitch.


However, in reality these NICs are not touching the same device. Each of these NICs is mapped statically to a different Virtual Connect module (port 01 to VC module 01, and port 02 to VC module 02). Therefore in a Virtual Connect environment you should note that you cannot choose the load balancing algorithm on the NICs based on IP-hash. Though you can choose any other algorithmn, HP’s recommendation is using Virtual Port ID, as you can see on the guide (this is the little piece of information missing on VC Cookbook on Active-Active config with Vmware).

Finally, so why the heck can’t you use IP-hash, nor VMware’s new LACP capabilities? Well  IP-Hash is intended when using link aggregation mechanisms to a single switch (or Switch cluster like IRF, that behaves and is seen as a single switch). As a matter a fact, IP-SRC-DST is the only algorithm supported for NIC teaming on the ESX box as well for the Switch when using port trunks/EtherChannels:

” […]

  • Supported switch Aggregation algorithm: IP-SRC-DST (short for IP-Source-Destination)
  • Supported Virtual Switch NIC Teaming mode: IP HASH
  • Note: The only load balancing option for vSwitches or vDistributed Switches that can be used with EtherChannel is IP HASH:
    • Do not use beacon probing with IP HASH load balancing.
    • Do not configure standby or unused uplinks with IP HASH load balancing.
    • VMware only supports one EtherChannel per vSwitch or vNetwork Distributed Switch (vDS).

Moreover, VMware explains in another article: ” What does this IP hash configuration has to do in this Link aggregation setup? IP hash algorithm is used to decide which packets get sent over which physical NICs of the link aggregation group (LAG). For example, if you have two NICs in the LAG then the IP hash output of Source/Destination IP and TCP port fields (in the packet) will decide if NIC1 or NIC2 will be used to send the traffic. Thus, we are relying on the variations in the source/destination IP address field of the packets and the hashing algorithm to provide us better distribution of the packets and hence better utilization of the links.

It is clear that if you don’t have any variations in the packet headers you won’t get better distribution. An example of this would be storage access to an nfs server from the virtual machines. On the other hand if you have web servers as virtual machines, you might get better distribution across the LAG. Understanding the workloads or type of traffic in your environment is an important factor to consider while you use link aggregation approach either through static or through LACP feature.

So given the fact that you are connecting your NICs to different VC modules (in a HP c7000 Enclosure), the only way you will be able to have active-active teaming is with static etherchannel, and without IP-Hash.

None-the-less please note that if you were configuring for instance a non-blade server, such as a HP Dl360, and connecting it to an IRF ToR cluster, then in that case you would be able to use vDS with LACP feature, and thus use IP-Hashing. Or if you were using a HP c3000 Enclosure, which is usually positioned for smaller enviroments. So it does not make that much sense to be using a c3000 and in the meanwhile having the required vSphere Enterprise Plus licensing.

That’s it, I suppose. Hopefully this overview should already get your ESX farm up-and-running.


Disclamer: note that these are my own notes. HP is not responsable for any of the content here provided.