Resilience measures with HP IRF: ISSU, GR and MAD (Part I)

In an effort to come down to earth and cover a topic which can be useful for the majority of now-a-days Enterprises that have HP gear, I will cover resilience features one can/should use in a HP Networking environment along with IRF.

Though I don’t argue that this is a Best Practice for all cases, clustering HP (former 3Com) switches with IRF can be a great solution to a lot of problems. How? Basically using a simple and thus effective ingredient that speaks by itself: drastic topology complexity reduction. Aggregation of devices to function as one can be an early Christmas for many cases: You get to do so many more LAGs (MLAGs preferably), your Spanning Tree is much simpler (I’m not that big of a fan of the HPN marketing papers issuing STPs death certificate by IRF, you gotta admit human mistake!), your linkstate DB gets much simpler, and you have less devices to config and manage (after you get centralized control & management planes).

Creating a Non-stop network environment
I’m not going to focus on HP’s claim on convergence time above HPs claim. That should be left for a demo by HP guys. What I want to focus on are the Software features that can be used to create an even more resilient network along with IRF, and which require human config.
These features are In-Service-Software-Upgrade (ISSU) – so yes, you get this bonus feature on standalone switches (where this feature is not natively present) when you setup an IRF cluster – Graceful Restart, and Multi-Active detection – commonly know as split brain detection.

First you have to take into consideration that the way IRF works when you do virtual clustering with standalone switches, is exactly the way chassis-based switches work: Master MPU controls Management & Control planes, synchronizes real time with standby MPU, and forwarding plane is active on all LPUs. The difference being that in standalone switches one of the switch acts as a Master and the all the rest of the nodes as standby “MPUs”.
So when your start a Software upgrade on a IRF stack, the first members to be upgraded are the standby nodes. After this job is completed, one of the standby nodes gets elected as the new master, and failover occurs. While the new elected node acts as Master, the former Master is upgraded. After this job is done, preemption-alike behavior occurs, and the former master gets reelected to the master role.
Note that the virtual cluster runs with a virtual Bridge MAC address, so L2 destination remains remains the same, and the only changes that might occur are on the link forwarding inside MLAGs. This should be neglectable if you have a solid config. Note also that the routing process (if any) running on the master will not be restarted during the service upgrade.


Comment Relative To former post “HP MSM Controllers Initial Setup Considerations”

This post is an answer to a comment relative to this former post “HP MSM Controllers Initial Setup Considerations“. (I wanted to add some drawings to the answer to make it clearer, so ended up using another post.) Thank you for your comment. Please let me know if I understood your comment well, or got it all wrong.

George P Isaac’s comment was:

As per my understanding..In option 2 we have to do following config.

1.We should tag particular VLAN in either internet port or access port
2.we should assign IP address to particular VLAN and gateway..
3.Nothing is required in VSC mapping in AP group.

then Where will I specify VLAN mapping..??

“extending the ingress interface to the egress interface “– is this option used when internet port and tunneled network in same VLAN??

OK to be fair my post could have been clearer (probably related to the fact that I’m not an expert on HP’s MSM solution). I took the next figure from a MSM Controller Config manual (section 4-30), which summarizes pretty well in my opinion what the options are for Access-Controlled traffic.

Access-controlled Flow of traffic

So let me recap: “Option 2” should actually have been Option 2 A) and B). Essentially these options are:

2. A) Having Access-Controlled Clients doing Web Auth on the HP MSM Controller, and then being ejected straight to the a default Gateway of a different interface that bypasses the Corporate Network. So in summary what the Egress VLAN is doing is defining a new default Gateway – for instance the routing device connected to the Internet – for the controller to use specifically for the clients assigned to that VLAN. The reasoning for this option might be that the Network Admin is worried that it may allow clients to access corporate resources, the Controller’s default Gateway might not have the adjusted ACLs for handling the Client traffic, or you might prefer to simply save that router CPU. In any case – the main goal is to bypass the Corporate Network.

In this option Clients still have an IP address in a different subnet from the egress VLAN subnet. This is the reason why you should enable NAT on that interface (to simplify): because clients will be placed on a subnet to which the gateway has no route to. Alternatively you can also configure a Route on that gateway to that subnet.

2. B) Having Access-Controlled Clients doing Web Auth on the HP MSM Controller, and then being ejected to a restricted VLAN, and receiving IP addresses on the egress VLAN. Hence the term extending the Egress VLAN to clients. Sort of like in non-access-controlled scenarios, but in this case the controller is actually routing on the background, though clients can’t notice it.

Please note that Option 2 A) and B) have essentially one thing in common: clients bypass Corporate resources. However in example B, clients actually receive an IP address in the VLAN where they are ejected either given by a DHCP Server resident on that VLAN, or by the Egress VLAN’s Gateway, which is implementing DHCP Relay.

then Where will I specify VLAN mapping..??” 

So in both cases, you have to alternatives to specify the VLAN Mapping: VSC level or user-based level. In VSC level you simply grab all authenticated users and forward them on the same VLAN. In user-based Egress VLANs you get more granularity by specifying customized VLAN IDs to user account profiles. Or you can implement both altogether, where user-based specifics override VSC level Egress VLAN definitions.

However in option B – where you extend the Egress VLAN IP addressing to the clients – you have some additional settings to configure, which I specified in the previous post:

  • In the global DHCP relay settings, select the checkbox for extending the ingress interface to the egress interface. After this is enabled, you will no longer be able to specify the IP address and subnet mask settings on VSCs.
  • Disable NAT on the egress VLAN IP interface.
  • Set the MSM controller’s IP address for the default gateway and DNS server (it will forward them to the correct ones).

“extending the ingress interface to the egress interface “– is this option used when internet port and tunneled network in same VLAN??

Well the Egress VLAN (whether extended or not to the client) might be implemented in either port (LAN or Internet). I would rather say that this option might make sense being used when you prefer having the “Bypassing main Corporate Network feature” + “the non-access-controlled alike behavior” all together. This solution might greatly simplify your manual setup, when you want certain clients to access some corporate resources, for instance. Having the same Subnet as those corporate resources might be advantageous for your setup.

Hope this helps. Cheers!

VMware NSX

Its VMworld show time, which means awesome stuff being announced. VMware’s Overlay Network and SDN Sagas are clearly just starting, and so VMware announced today its Network Virtualization Solution: NSX.

I haven’t been able to grasp any VMware own technical material yet, however from what I could until so far understand we are talking about the first true signs of a Nicira’s integration/incorporation into current Vmware Networking Portfolio. VMware’s Overlay Networking technology – VXLAN –  has been integrated with Nicira’s NVP platform. However, this is not mandatory, so you do get 2 additional flavors of encapsulation to choose among VXLAN: GRE and STT.

As it can be seen from NSX datasheet, key features are similar to Nicira’s NVP plataform:

• Logical Switching – Reproduce the complete L2 and L3
switching functionality in a virtual environment, decoupled
from underlying hardware
• NSX Gateway – L2 gateway for seamless connection to
physical workloads and legacy VLANs
• Logical Routing – Routing between logical switches,
providing dynamic routing within different virtual networks.
• Logical Firewall – Distributed firewall, kernel enabled line
rate performance, virtualization and identity aware, with
activity monitoring
• Logical Load Balancer – Full featured load balancer with
SSL termination.
• Logical VPN – Site-to-Site & Remote Access VPN in software
• NSX API – RESTful API for integration into any cloud
management platform”

Ivan Pepelnjak gives very interesting preview-insight on what’s under the hood. Though new, integration with Networking partners is also being leveraged. Examples are: Arista,  Brocade, Cumulus, Dell, and Juniper on the pure Networking side, and Palo Alto Networks on the security side.

Exciting time for the Networking community!


I found this Packet-Pusher’s Podcast from last week about Avaya’s Software Defined Datacenter & Fabric Connect with Paul Unbehagen really interesting. He points out some of the differences in VMware’s Overlay Network VXLAN approach against Physical Routing Switches Overlay Network’s SPB approach (which naturally Avaya is using). Some of these were different encapsulation methods – where with VXLAN the number of headers is quite more numerous – ability to support Multicasting environments (such as PIM), and most importantly, raises the central question: where do you want to control your routing and switching – the Virtual Layer or the Physical Layer. Even though I’m theoretically favorable to Virtual, arguments to keep some functions on the Physical layer still do make a lot of sense in a lot of scenarios.

The Podcast also features a lot of interesting Avaya Automation related features result of a healthy promiscuous relationship between VMware and OpenStack. Also if you want to get in more detail about SPB, Paul Unbehagen covers lots of tech details in his blog.

Overlay Virtual Networks – VXLAN

Overlay Virtual Networks (OVN) are increasingly gaining a lot of attention, whether from Virtual Networking providers, as for Physical Networking providers. Here are my notes on a specific VMware Solution: Virtual eXtensible LAN (VXLAN).


This would never be an issue if not large/huge datacenters didn’t come into place. We’re talking about some big-ass companies as well as Cloud providers, enterprises where the number of VMs scales beyond thousands. This is when the ability to scale within the Network, to rapidly change the network, and the ability to isolate different tenants is crucial. So the main motivators are:

  • First L2 communication requirement is an intransigent pusher, which drags 4k 802.1Q VLAN-tagging limitation plus L2 flooding with it
  • Ability to change configurations without burining the rest of the network, and doing it quickly – e.g. easy isolation
  • Ability to change without being “physically” constrained by Hardware limitations
  • Ability to scale large number of VMs, and being able to isolate different tenants.
  • Unlimmited Workload mobility.

These requirements demand for a change in the architecture. They demand that one is not bound to physical hardware constraints, and as such, demand an abstraction layer run by Software and which can be mutable – a virtualization layer in other words.

Network Virtualization

Professor Nick Feamster – who ended today his SDN MOOC course on Coursera – goes further and describes Network Virtualization as being the “killer app” for SDN. As a side note, here is an interesting comment from a student of this course.

Thus it is no surprise that Hyper-visor vendors were the first to push such technologies.

It is also no wonder that their approach was to treat the physical Network as a dumb Network, unaware of the Virtual Segmentation that is done within the Hyper-visor.

To conclusion, the main goal being really moving away from the dumb VLAN-aware L2 vSwitch to building a Smart Edge (Internal VM Host Networking) without having to rely on smart Datacenter Fabrics (supporting for instance EVB, etc).

Overall solutions

There is more than vendor using a OVN approach to solve the stated problems. VMware was probably one of the first Hyper-visor vendors who started with their vCloud Director Networking Infrastructure (vCDNI). MAC in MAC solution, so L2 Networks over L2. Unfortunately this wasn’t a successful attempt, and so VMware changed quickly the its solution landscape. VMware currently has two Network Virtualization solutions, namely VXLAN and more advanced Nicira NVP. Though I present these two as OVN solutions, this is actually quit an abuse, as these are quit different from each other. In this post I will restrict myself to VXLAN.

As for Microsoft, shortly after VXLAN was introduced Microsoft proposed its own Network Virtualization Solution called NVGRE. Finally Amazon uses L3 Core, which uses IP-over-IP communications.


Virtual eXtensible LAN (VXLAN) was developed by in conjunction of Cisco and VMware, and IETF launched a draft. It is supposed to have a similar encapsulation Header as in Nexus 7k OTV/LISP, allowing for Nexus 7k to act as a VXLAN Gateway.

VXLAN introduces an additional kernel Software layer between ESX vSwitches and Physical Network Card, which can either be VMware Distributed vSwitch or Cisco’s Nexus 1000v. This kernel code is able to introduce additional L2 Virtual Segments beyond the 4k 802.1Q limitation over standard IP Networks. Note that these segments run solely within the hyper-visor, which means that in order to have a physical server communicating with these VMs you will need a VXLAN Gateway.

So the VXLAN kernel is aware of Port-Groups on VM-side and intriduces a VX-segment ID (VNI), and introduces an adaptor on the NIC-side for IP communications- the VXLAN Termination Point (VTEP). VTEP has an IP address and performs encapsulation/decapsulation from L2 traffic generated by a VM and inserts VXLAN header and an UDP header plus traditional IP envelop to talk to the physical NIC. The receiving host where the destination VM resides will do the exact same reverse process.

Also note that it transforms broadcast traffic into multicast traffic for segmentation.

It is thus a transparent layer between VMs and the Network. However, since there is no centralized control plane, VXLAN used to need IP multicast on the DC core for L2 flooding. However this has changed with recent enhancements on Nexus OS.

Here’s VMware’s VXLAN Deployment Guide, and Design Guide.

Finally please do note that not everyone is pleased with VXLAN solution.


SDN Playground – getting started with OpenFlow

Most every big Networking company has announced something related to SDN. Whether simple marketing to concrete legit solutions, its a question of time until the market is filled with SDN-related products. It is thus essential to start getting familiar with it, and you know damn well there’s nothing like getting your hands dirty. So here are some helper-notes on getting started with sandboxing OpenFlow (OF) environments.

To do so I’m using Mininet – a VM created part of an OpenSource Project to emulate a whole complete environment with a Switch, an OF Controller, and even three linux hosts. Also note I’m using my desktop as a Host, with VirtualBox.

So what you’ll need:

  • If you don’t have it yet, download VirtualBox, or another PC hyper-visor Software such as VMware Player. VirtualBox has the advantage of being free for Windows, Linux and Mac.
  • Download Mininet VM OVF image.
  • After decompressing the image, import the OVF.

VB Import Applicance

  • In order to establish terminal session to your VM, you’ll need to add a Host-only Adaptor on the Mininet VM. So first (before adding the adaptor on the VM itself) go to VirtualBox > Preferences. Then select the Networking tab, and add and adaptor.


  • Next edit Vm Settings, and add an Host-only Adaptor. Save it and boot the VM.
  • User: mininet       Password: mininet
  • Type sudo dhclient eth1 (or if you haven’t added another adaptor and simply changed the default Adaptor from NAT to Host-only adaptor then type eth0 instead of eth1) to enable DHCP service on that interface.
  • Type ifconfig eth1 to get the IP address of the adaptor.
  • Establish an SSH session to the Mininet VM. Open terminal, and type ssh -X [user]@[IP-Address-Eth1], where the default mininet user is “mininet” and IP address is what you got after ifconfig. So in my case it was: ssh -X mininet@
  • Mininet has its own basics tutorial – the Walkthrough. Also interesting is the OpenFlow tutorial.

The Mininet Walkthrough is designed for less than an hour tutorial. Here are some simple shortcuts to speedup your playing around:

  • Type sudo mn –topo single,3 –mac –switch ovsk –controller remote. This will fire up the emulated environment of the switch, OF controller, and 3 linux hosts.

OF topology

  • Type nodes to confirm it. “h” stands for hosts, “s” for switch and “c” for controller. If you want, for instance, to now the addresses of a specific node such as Host2, type h2 ifconfig. If you want to establish a terminal session to the same host, type xterm h2. Note that xterm command only works if you first established ssh session by typing ssh -X

This should already get you started.

Have fun!