Hyperconvergence’s Biggest Risk

Hyperconvergence’s Biggest Risk comes from its Biggest Advantage.

IT Hyperconvergence


Hyperconvergence IT solutions are making local cloud environments a reality for any size financial institution. Let’s start with a simple definition for Hyperconvergence.  It is all of the infrastructure stack from Hypervisor down to the physical hardware needed to run a virtual environment.  This includes Compute, Storage, storage networking, replication, backups, and often a bundled or supported hypervisor all from one vendor.  And it’s that One vendor, having only One vendor that is the best thing and the riskiest thing about Hyperconvergence.




The benefits are real and it is a huge advantage to have One vendor.

  • One vendor management tool/interface for your VM & Storage & Backup environment.
  • The same vendor for storage issues
  • The same vendor for compute/host issues
  • The same vendor for storage networking
  • The same vendor for performance issues
  • The same vendor for hypervisor issues
  • The same vendor for backup and replication issues

The systems are designed, supported, and upgraded through One vendor.  The top Hyperconvergence vendors have engineers and support technicians certified and experienced in every aspect of the stack to help support you.  It is truly One vendor to call & One throat to choke should there be issues or support needed.

Which leads me to hyperconvergence’s biggest risk.  You have 1 vendor for your primary infrastructure.  While vendor due diligence and vendor management gets a bad rap because of the sometimes pointless required checklists, this is the time to use your vendor skills and know your vendor risks.  You can take some risk mitigation steps and for most institutions the benefits still outweigh the risks.

I will not cover all the risks and mitigation options but highlight just a few to get you started.  In many ways the benefits and risks are similar to cloud vendors, but you have the big advantage of still having direct physical access to your systems at your headquarters or in your colocation data center.

Here are a few risk factors to consider:

  • Your vendor is Venture Capital backed and begins running out of money:
    • They then go out of business.
    • Your vendor or their assets get acquired
  • Your vendor infringes on patents with these fast moving technology changes which impact the services they can develop or offer.
  • Service impacting events:
    • Your vendor gets acquired by a larger traditional provider changing your support & service.
    • Your vendor is extremely successful and goes through growing pains impacting your service levels.

Here are some common mitigation options:

  • Always know your vendor’s financial position, how they access money, and growth/acquisition strategy. Set an appropriate interval for reviewing basic financials & performance.
  • Know your own data migration strategy. Even if you don’t develop a full migration plan, know how data can be migrated to another system before you sign a contract.  It could be as simple as exposing an NFS share to a new environment to allow storage vmotion or it could be a more complicated process.  Even if your vendor is a rock star, chances are you’ll not be on their solution forever.
  • Know the eco system and community surrounding the vendor. User groups are a great option to find help and support for issues during a challenging situation.
  • SLAs should always be part of contracts. Companies should stand behind their claims of compression, deduplication, and service levels.   A contact commitment does not guarantee good service but does encourage a company to do the right thing for you.  It also opens the door for your migration to a new solution without any penalties and possibly with vendor assistance if issues can’t be resolved.

Two vendors I see pushing the Hyperconvergence pack are Simplivity and Nutanix.  They have similar benefits but take different approaches and have different underlying technology used to accomplish their solutions.  Even if a Hyperconvergence solution is not right for you now, they are having a significant impact on traditional SAN vendors and some of the ease of management, reporting interfaces, and benefits are starting to be built into traditional SAN vendor solutions. If you’re not following the Hyperconvergence trend, you should start today.



NetApp and Cisco LACP Setup

NetApp and Cisco LACP SetupLACP Multi Network

EtherChannel & Port Channel setup and troubleshooting.

I was assisting a client recently and thought this information would be helpful to anyone setting up a EtherChannel/Port Channel group for NetApp or other SAN and Cisco switching.

There are two basic options I’ll cover:

  1. Static Etherchannel (Multi in NetApp)
  2. Dynamic Etherchannel (LACP)

This is not a full walk though, only a reference to the specific difference that can cause issue and some basic troubleshooting.  For the setup to work correctly, the switch and SAN setup must match.

For Static Etherchannel:

In the Cisco config this is determined by the channel-group mode being set to ‘on’.  See the example config below:

interface GigabitEthernet1/1
description NetApp e0a
switchport access vlan 100
switchport mode access
flowcontrol receive on
no cdp enable
spanning-tree guard loop
channel-group 5 mode on

In NetApp the creation of VIF as Multi setups up the static Etherchannel.

vif create multi template-vif1 –b ip e0a e0b

For Dynamic (LACP) Etherchannel:

In the Cisco config this is determined by the channel-group mode being set to ‘active’.  See the example config below:

interface GigabitEthernet1/1
description NetApp e0a
switchport access vlan 100
switchport mode access
flowcontrol receive on
no cdp enable
spanning-tree guard loop
channel-group 5 mode active

In NetApp the creation of VIF as LACP setups up the dynamic Etherchannel.

vif create lacp template-vif1 –b ip e0a e0b

Basic Troubleshooting:

For NetApp

From the command line, use this command:  ifgrp status

It provides a basic summary of the ports and their status so you can identify common issues.  See the examples below:

Normal information returned (with some notes in blue)

trunked: corp_ifgrp
e7b: state up, since 26Feb2013 04:58:05 (4+19:12:27) -Your port is active "state up"
mediatype: auto-10g_sr-fd-up
flags: enabled
active aggr, aggr port: e13b - Your port is participating in the etherchannel/port group
input packets 9493, input bytes 1177132 
input lacp packets 13831, output lacp packets 415997 -You only see the lacp packets for dyanmic groups.
output packets 617618727, output bytes 2908794789546  
up indications 3, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 26Feb2013 04:58:05
consecutive 0, transitions 3
e13b: state up, since 26Feb2013 04:58:04 (4+19:12:28)
mediatype: auto-10g_sr-fd-up
flags: enabled
active aggr, aggr port: e13b
input packets 9494, input bytes 1177256
input lacp packets 13830, output lacp packets 414747 -You should see at least 1 lacp packet increase every 30 secs.
output packets 505651519, output bytes 4117402075645
up indications 3, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 26Feb2013 04:58:04
consecutive 0, transitions 3
For Cisco:
show port-channel summary
D - Down
P - Up in port-channel (members)
I - Individual
H - Hot-standby (LACP only)
s - Suspended
r - Module-removed
S - Switched
R - Routed
U - Up (port-channel)
 Group Port- Type Protocol Member Ports Channel
 1 Po1(SU) Eth NONE Eth3/4(P)  - See ref above: 'P' means up and active in the port-channel 
 2 Po2(SU) Eth NONE Eth3/2(P) Eth3/6(P)
Examples of issues:

e0b: state broken, since…   This would indicate a cable unplugged, port down, or physical port issue.  The port is down.

e0b: state lag_inactive, since…   This would indicate the port is no participating in the lacp etherchannel port group.


Resources for more information:

NetApp KB for LACP port channel

Cisco troubleshooting Port Channel groups









Manually send Data Domain autosupport files

Manually send Data Domain autosupport files:

First confirm where your autosupport files are going to be send:


Get autosupport files/information from restorer web interface:

  • There is no way to send the information from the web interface but you can view, copy, and email manually:
  • Login to the restorer using a browser.
  • Click on: “support”
  • Click on: “autosupport”
  • Select all text, and copy
  • Paste the text in an Email and forward to your support team or vendor

Send autosupport from command line interface:

  • Access the Command Line Interface (CLI) using a serial cable, telnet, or an SSH client.
  • After logging on type:
    • autosupport send
      • An autosupport should be sent to any Email address that is on the autosupport list.
    • To view the support list type:
      • autosupport show support-list
    • To add to the support list, type:
      • autosupport add <email address>
    • To send an autosupport to a specific email address, type:
      • autosupport send user@email_address.com

Where are my Data Domain autosupport support files sent?

Determine where Data Domain autosupport support files go:

EMC says: Autosupports contain basic configuration and performance data and are very useful in the trouble-shooting process.

Why might I need to determine where my support / autosupport files are going?

  1. EMC is not receiving them
  2. You are no longer under support and need diagnostic info
  3. You are using a 3rd party support vendor
  4. You need the information for competitive quotes or planned migration to another technology

How to get the support email list:

  • Access the Command Line Interface (CLI) using a serial cable, telnet, or an SSH client.
  • After logging on to the restorer, type:  autosupport show support-list 
    • This should show you the support list email addresses

To add to the support list:

  • type: autosupport add <email address>

How to manually send your autosupport files: http://www.lcfintech.com/manually-send-data-domain-autosupport-files/

Hyperconvergence Explained in One Image

Hyperconvergence Explained in One Image

Hyperconvergence Explained in one image
Hyperconvergence Explained in one image – source Simplivity Twitter post

Traditional Infrastructure Stack
  • Servers & VMware
  • Storage Switch(es)
  • HA Shared Storage
  • SSD / Flash / Cache Array
  • Backup Appliance
  • WAN Optimization
  • Cloud Gateway
  • Storage Caching
  • Backup Apps

Simplified by Hyperconvergence

  • All in One Solution – Simplivity

Privacy Policy

Privacy Policy

Your privacy is a top priority for LostCreek Fintech.  We provide this privacy notice to explain how we collect, use and safeguard your personal information.  We are committed to providing you with competitive solutions, products, & services that meet your IT and regulatory needs, which necessitates that we collect and share information about you to provide you with certain IT opportunities & services.

In order to do so, we have entered into agreements with other companies that provide either services to us or additional IT services and products for you to consider. Under these arrangements, we may disclose all of the information we collect to companies that perform marketing or other services on our behalf or to other partner companies with whom we have joint marketing and/or service agreements. To protect our customers’ privacy, we only work with companies that agree to maintain strong confidentiality protections and limit the use of information we provide. We do not permit these companies to sell to other third parties the information we provide to them.

Create and send Netapp AutoSupport file

Howto: manually create and send Netapp AutoSupport file.

This file is often needed by Netapp or 3rd Party Maintenance providers for troubleshooting or quoting hardware or support.

Create the AutoSupport file:

  1. Login into Netapp with root/admin priviledge.
  2. Run the following command:  options autosupport.doit now 'insert your custom info here'
  3. A new folder and files will be created in the /vol/vol0/etc/log/autosupport directory.  The current date and time will be used for the folder and seen in the created/modified timestamp fields.
  4. Mount that path with NFS, or access it through a CIFS share in order to copy those files onto your system
  5. Pull files from the newly created folder on the /vol/vol0/etc/log/autosupport directory to your local machine.
  6. There will be 1 text file and a number of .gz files.
  7. Open the text file and copy the information into the body of your email.   Attach the .gz files to the email to your support provider.
  8. Send your email with the AutoSupport information needed by your vendor and your done!

Reference: Netapp KB