Storage Brief – Pure Storage

Storage Brief – Pure Storage

Pure StoragePure Storage is one of the leading all flash storage provider. Competitors are XtremIO, SolidFire, Nimble, etc.

Here is a quick summary for Pure Storage:

  •  All flash performance
  • Duplication and compression that meets or beats rivals
  •  Easy of management compared to legacy storage
  • In depth reporting on hardware, software, and performance that is for customer and also goes to Pure maintenance techs for an extra extension of your team.

What is unique? Why choose Pure Storage?

  • They are as much a software company as they are a hardware storage company.  All software/licenses/upgrades are part of their storage solution.  There are not module costs/tiers.  When you buy Pure Storage you get all the software features.
  • Upgrades & expansion without downtime.  This isn’t just for storage but also storage processor replacements and full upgrades to new/higher performance SPs.
  • Database (SQL & Oracle) and VDI focused for performance and deduplication space saving.
  • Unique upgrade/replacement Business Model – their Forever flash gives upgrades for storage processors when maintenance is renewed after 3yrs.  This can provide a more stable & predictable storage cost instead of the traditional 3-5yr capital refresh to get new performance.

Contact us for more information on Pure Storage or other flash technologies.

Storage Brief – NetApp and SolidFire

 Storage Brief – NetApp and SolidFire

In Q4 of 2016 NetApp announced the acquisition of SolidFire and completed the acquisition this year, Q1 2016.

If you haven’t looked at SolidFire, here is a very briNetApp & SolidFireef overview.

  • SolidFire is all flash storage (competitors XtremIO, Pure Storage, etc).
  • It has all the things you expect from a current top tier solution:
    • Block level deduplication
    • Data protection and re-balance data after failure.
    • Easy scale-out storage
    • Simplified management compared to traditional storage
  • If it is like it’s competitors, what is unique and why buy it?
    • Performance, guaranteed performance
      • You can set QOS for I/O performance so your critical apps always have performance.  No more relying on alerts and searching through performance graphs to try and determine why your critical app’s storage is slow.

Contact us to learn more about SolidFire or other flash technologies.


Data Center & SAN Move – Tips and Tricks

Data Center & SAN Move

What to do when the timing & budget don’t allow for duplicate hardware.

Data Center

In a perfect world, you’ll always have the time and budget to stage a new environment with the storage and telecom capacity to migrate all your data to the new location for a move.  Even with my refurbished and cost reduction options this hasn’t always been possible for my clients.  Since there are only a few who live in the perfect world of large IT budgets this information is for the rest of us.  Here are some highlights from my experience that might help you move your production systems.

First and foremost, know that it can be done.

It may be a high stress time but it can be done.  There will be some downtime for applications that impact customers and communication with clients and adjusting to their needs are key to help make the move successful for everyone.

Expectation setting & communication is critical to your success:

  1. Develop a Communication Plan
    1. Identify all the parties that need to be kept up to speed
    2. Clients
    3. Internal Departments.
    4. IT teams.
    5. Management
    6. Vendors assisting with the move.
  2. Set key milestones for communication to the various groups
    1. Regular interval – current status whether the news is good or bad, send it out.
    2. Triggered by events
      1. Systems going dark
      2. Leaving old Data Center
      3. System arrived at new Data Center
      4. Systems up & testing beginning
      5. Live system & maintenance window closure

Document & Label everything:

  1. Document everything, label everything, then go back and have a different person check everything for clarity and completeness.
    1. Label & document every cable, device, shelf, and port. Make sure you label both the cable end and the port on the SAN/Server/switch.
    2. Grab configs, take screenshots from mgmt. interfaces, get a copy of everything you can.
      1. SAN, network switches, storage switches, Vmhosts and any other connected systems.
    3. Take pictures. Yes, grab your camera and take pictures from far way, close up of each rack front & back, and then even closer for the backs of storage processors, servers, or switches.
      1. If/when you need to remember what was plugged into what when the label falls off the cable or server, then you’ll at least have a picture to reference.

Expect something to go bad & prepare for the disaster:

I’ve seen things dropped, vibrate loose in shipment, or moving trucks breakdown or get in minor accidents.  Here are some things you can do.

  1. Develop & Test shutdown and startup scripts. Make sure you have a checklist with the order & dependancies of what is turned off & on in what order.
    1. This is for data safety as well as for efficiency. With many people working and as the move progresses and fatigue sets in, you need to make sure you’re addressing systems in the right order.
    2. Review this regularly before the move and when possible practice it on test systems to know everything will go smoothly. You don’t want an error in your shutdown or startup processes causing delays or data integrity issues.
    3. Make sure someone owns the orchestrator/coordinator role and have key milestones signed off on before the team continues to the next major step.
      1. If you know you need all guest VMs turned off before turning off hosts then create a ‘pause’ and have someone double check before proceeding. The same holds true with making sure all systems are off before turning off your SAN & switches.
  2. A delay is bad; loss of data or a system is much, much worse. Plan for the worst.
    1. Check Backups and test that they are good before proceeding with a move.
    2. Get full images of critical systems or applications whenever possible.
      1. That secondary SAN may not be in the budget but I bet a refurbished NAS or at the very least a few USB hard drives can be purchased if it means protecting key system data.
  3. Use Key Vendor relationships
    1. I recommend having your vendors scheduled for your move. That could mean having them participate in the shutdown/startup process milestones or at the very least on-call.
    2. In my experience it can be worth the money to have the SAN vendor on-site at the new data center location.
      1. They can quickly help address any hardware issues.
      2. I always have them come with some parts in hand.
        1. An extra power supply, hard drive of each drive type for your SAN, and possibly even a storage processor/controller if you can get them to bring one.
      3. When you’re in the 2nd half of your maintenance window, you don’t want to wait 4hrs or more for your support vendor(s) to be responding to your request.
  4. Separate production and backup
    1. After all your prep work make sure you put your backups in a different truck or transport vehicle. You don’t want to lose everything in 1 vehicle accident.
    2. I’ve had IT staff transport encrypted drives & backups in a separate car taking a separate route or delayed by 1-2 hours behind the primary moving truck.

Take care of your staff:

  1. You can’t be everywhere and check everything yourself. Take care of your staff and prepare them.
    1. Talk through scenarios, checklists, and expectations before the move.
    2. Treat movers, packers, and other laborers as part of your staff and set expectations with them. Make sure you tip them too as you’ve probably just asked them to work/drive through the day/night to help your business.
    3. Use rotating shifts for staff.
      1. Tired staff make mistakes. Whether it is management or engineers, make sure you rotate through people so they stay fresh, get rest, get food & drink, and stay healthy.

Celebrate when you are done:

  1. Celebrate your success and learn from your challenges.
    1. Do a move review a week or two later to learn from the challenges and improve processes for future moves or system maintenance.
    2. Make sure the final IT setup is well documented for the future before you close down your project.
    3. Celebrate with your staff. This can be food, company swag, a party, bonuses, or more.  The success of the move was because of your staff’s hard work.  Show your appreciation.

LostCreek Fintech handles all aspects of storage solutions. Moving a live production storage environment is not ideal but can be done with the right planning and support.  Hopefully our experience helps you through your next move.  We are here to help if you ever need storage options, support, or advice for a move or storage project.




Cloud is Not a Strategy

Cloud is Not a Strategy.

IT cloud may just be a fogThe ‘Cloud’ is talked about like it is a heavenly option saving IT but for many it is only a low lying dense fog clouding their vision.

Some institutions are wandering around blindly as they try out various software as a service offerings or charging ahead blindly with one ‘all-in’ cloud vendor hoping there is a clearing ahead that delivers on the benefits of cloud.  If you don’t have a clear IT strategy, you shouldn’t focus on the cloud or have a cloud strategy.  The cloud is not a strategy!

I’ve run into too many conversations lately where the ‘cloud’ is stated as the defining IT strategy going forward without many details around the overall IT strategy.  I’ve talked with a few CTO/CIOs who have shared the following experience.  Their CEO goes to the latest conference or has lunch with a CEO buddy and comes back stating that the bank or CU should be moving to the cloud.  It is, of course, common for a CEO to share the latest things they’ve heard that could help the business and it’s better to have an engaged CEO than an absent one.  This question & situation is not to be feared.  In all but the unhealthiest leadership cultures it is an opportunity for the CTO/CIO.

If leadership hasn’t talked about the overall IT strategy in the last few months, especially after going through the budgeting season and entering a new year, then there is a larger problem than cloud or no cloud.  This is the opportunity to open up more conversation.  The last thing needed in that situation is an independent strategy around cloud.  What is needed is an overarching IT strategy and plan.  CTO/CIOs, your plan may just need updated to address hosted and cloud solutions but what you must do is engage leaders in your institution to share your overall strategy.  Don’t get lost in the fog of only focusing on ‘cloud’.

There are many ‘how to’ articles and templates about writing strategy statements and plans so I’m not covering that here.  If you want help with that, let me know and I’ll point you to a peer who has a solid plan or vendor who can help.

I will instead suggest the following 4 steps:

  1. Find, update, or if needed develop your overall IT strategy and IT strategic plan.
  2. Know your institution’s goals, strengths, & upcoming needs. You will need to lead, support, follow, and adapt to the business units.
    1. Make sure the institution’s goals are restated in your plan for a clear connection between IT and your institution.
  3. Evaluate the strengths, weaknesses, and the resources available for updating your staff, your IT infrastructure, and your applications. These will influence your strategy.
  4. Engage your CEO, SVPs, CFO, COO, and other leaders as appropriate.
    1. Explain your plan.
    2. Listen to feedback.
    3. Healthy debate is good.
    4. Refine the plan to better serve the business & your IT info with any new information.

If you’ve done the four things above lately, then your strategy should already address cloud options and tools. If it doesn’t then you are missing an opportunity both for your institution as well as for you to communicate to your peers that you are staying relevant with current trends.  Remember the ‘cloud’ comes in many forms; Local, hybrid, public as well as SAAS, PAAS, IAAS just to throw out a few variations.  All of my customers have some aspect of cloud in their environment, even if it is just a specialized SAAS service such as HR services.  Know the tools available to you and address cloud in your IT plan and looking for vision beyond the fog.


LostCreek Fintech helps credit unions, banks, and healthcare institutions with storage solutions and large infrastructure design and implementation projects.  Contact us @ Inquiry for more information.


Top 5 Benefits of Third Party Storage Maintenance

Third / 3rd Party Maintenance

Top 5 Benefits of Third Party Storage Maintenance

  1. Tailored SLAs to meet your needs and budget

    1. 24/7/365, business hours response only, or custom schedules
    2. proactive monitoring and quick issue response
    3. onsite or remote response
  2. No forced non-human automation system

    1. You have email and phone access to humans in our support center
    2. Direct access to talk & interact with level 3 Engineers when needed.
  3. Onsite hardware kits

    1. Immediate parts available for the most common failures already at your location.
    2. Hard drives, power supplies, and other critical parts can be pre-staged at your data center & replenished if needed by the system instead of waiting for a part delivery.
  4. Remote assistance provides quicker resolution

    1. We can save you time and money by remotely troubleshooting & administering SANs & NAS devices.
    2. We work well with remote smart hands or general technicians at your data center to walk them through hardware troubleshooting & hardware replacements.
  5. Lower Cost. We just cost less than OEM hardware support.

    1. While our service & storage solutions keep our customers engaged, the budget friendly pricing is extremely attractive for our industry that often must have two or more storage solutions for regulatory and compliance reasons.
    2. Our vast inventory of parts and efficient support teams allow us to be a high value partner for the financial and healthcare industries.

Bonus.  The Gift of More Time

  1. Don’t rush into a new storage solution decision. You can create time and space to evaluate what overall solutions are the best fit for your IT & business needs by moving to 3rd party maintenance.
  2. You can keep or extend the life of storage SAN and NAS equipment to meet lower performance production needs (development & programming, archiving, backup vaults or document storage) with confidence that your hardware is being monitored and maintained.
  3. You aren’t forced into upgrades or hardware replacements because of escalating OEM maintenance pricing. You have 3rd party maintenance options that many large companies discovered years ago and are now available to credit unions, banks, and healthcare institutions to give us another tool for accomplishing our goals.

Contact us to find out more Email us


Hyperconvergence’s Biggest Risk

Hyperconvergence’s Biggest Risk comes from its Biggest Advantage.

IT Hyperconvergence


Hyperconvergence IT solutions are making local cloud environments a reality for any size financial institution. Let’s start with a simple definition for Hyperconvergence.  It is all of the infrastructure stack from Hypervisor down to the physical hardware needed to run a virtual environment.  This includes Compute, Storage, storage networking, replication, backups, and often a bundled or supported hypervisor all from one vendor.  And it’s that One vendor, having only One vendor that is the best thing and the riskiest thing about Hyperconvergence.




The benefits are real and it is a huge advantage to have One vendor.

  • One vendor management tool/interface for your VM & Storage & Backup environment.
  • The same vendor for storage issues
  • The same vendor for compute/host issues
  • The same vendor for storage networking
  • The same vendor for performance issues
  • The same vendor for hypervisor issues
  • The same vendor for backup and replication issues

The systems are designed, supported, and upgraded through One vendor.  The top Hyperconvergence vendors have engineers and support technicians certified and experienced in every aspect of the stack to help support you.  It is truly One vendor to call & One throat to choke should there be issues or support needed.

Which leads me to hyperconvergence’s biggest risk.  You have 1 vendor for your primary infrastructure.  While vendor due diligence and vendor management gets a bad rap because of the sometimes pointless required checklists, this is the time to use your vendor skills and know your vendor risks.  You can take some risk mitigation steps and for most institutions the benefits still outweigh the risks.

I will not cover all the risks and mitigation options but highlight just a few to get you started.  In many ways the benefits and risks are similar to cloud vendors, but you have the big advantage of still having direct physical access to your systems at your headquarters or in your colocation data center.

Here are a few risk factors to consider:

  • Your vendor is Venture Capital backed and begins running out of money:
    • They then go out of business.
    • Your vendor or their assets get acquired
  • Your vendor infringes on patents with these fast moving technology changes which impact the services they can develop or offer.
  • Service impacting events:
    • Your vendor gets acquired by a larger traditional provider changing your support & service.
    • Your vendor is extremely successful and goes through growing pains impacting your service levels.

Here are some common mitigation options:

  • Always know your vendor’s financial position, how they access money, and growth/acquisition strategy. Set an appropriate interval for reviewing basic financials & performance.
  • Know your own data migration strategy. Even if you don’t develop a full migration plan, know how data can be migrated to another system before you sign a contract.  It could be as simple as exposing an NFS share to a new environment to allow storage vmotion or it could be a more complicated process.  Even if your vendor is a rock star, chances are you’ll not be on their solution forever.
  • Know the eco system and community surrounding the vendor. User groups are a great option to find help and support for issues during a challenging situation.
  • SLAs should always be part of contracts. Companies should stand behind their claims of compression, deduplication, and service levels.   A contact commitment does not guarantee good service but does encourage a company to do the right thing for you.  It also opens the door for your migration to a new solution without any penalties and possibly with vendor assistance if issues can’t be resolved.

Two vendors I see pushing the Hyperconvergence pack are Simplivity and Nutanix.  They have similar benefits but take different approaches and have different underlying technology used to accomplish their solutions.  Even if a Hyperconvergence solution is not right for you now, they are having a significant impact on traditional SAN vendors and some of the ease of management, reporting interfaces, and benefits are starting to be built into traditional SAN vendor solutions. If you’re not following the Hyperconvergence trend, you should start today.



NetApp and Cisco LACP Setup

NetApp and Cisco LACP SetupLACP Multi Network

EtherChannel & Port Channel setup and troubleshooting.

I was assisting a client recently and thought this information would be helpful to anyone setting up a EtherChannel/Port Channel group for NetApp or other SAN and Cisco switching.

There are two basic options I’ll cover:

  1. Static Etherchannel (Multi in NetApp)
  2. Dynamic Etherchannel (LACP)

This is not a full walk though, only a reference to the specific difference that can cause issue and some basic troubleshooting.  For the setup to work correctly, the switch and SAN setup must match.

For Static Etherchannel:

In the Cisco config this is determined by the channel-group mode being set to ‘on’.  See the example config below:

interface GigabitEthernet1/1
description NetApp e0a
switchport access vlan 100
switchport mode access
flowcontrol receive on
no cdp enable
spanning-tree guard loop
channel-group 5 mode on

In NetApp the creation of VIF as Multi setups up the static Etherchannel.

vif create multi template-vif1 –b ip e0a e0b

For Dynamic (LACP) Etherchannel:

In the Cisco config this is determined by the channel-group mode being set to ‘active’.  See the example config below:

interface GigabitEthernet1/1
description NetApp e0a
switchport access vlan 100
switchport mode access
flowcontrol receive on
no cdp enable
spanning-tree guard loop
channel-group 5 mode active

In NetApp the creation of VIF as LACP setups up the dynamic Etherchannel.

vif create lacp template-vif1 –b ip e0a e0b

Basic Troubleshooting:

For NetApp

From the command line, use this command:  ifgrp status

It provides a basic summary of the ports and their status so you can identify common issues.  See the examples below:

Normal information returned (with some notes in blue)

trunked: corp_ifgrp
e7b: state up, since 26Feb2013 04:58:05 (4+19:12:27) -Your port is active "state up"
mediatype: auto-10g_sr-fd-up
flags: enabled
active aggr, aggr port: e13b - Your port is participating in the etherchannel/port group
input packets 9493, input bytes 1177132 
input lacp packets 13831, output lacp packets 415997 -You only see the lacp packets for dyanmic groups.
output packets 617618727, output bytes 2908794789546  
up indications 3, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 26Feb2013 04:58:05
consecutive 0, transitions 3
e13b: state up, since 26Feb2013 04:58:04 (4+19:12:28)
mediatype: auto-10g_sr-fd-up
flags: enabled
active aggr, aggr port: e13b
input packets 9494, input bytes 1177256
input lacp packets 13830, output lacp packets 414747 -You should see at least 1 lacp packet increase every 30 secs.
output packets 505651519, output bytes 4117402075645
up indications 3, broken indications 0
drops (if) 0, drops (link) 0
indication: up at 26Feb2013 04:58:04
consecutive 0, transitions 3
For Cisco:
show port-channel summary
D - Down
P - Up in port-channel (members)
I - Individual
H - Hot-standby (LACP only)
s - Suspended
r - Module-removed
S - Switched
R - Routed
U - Up (port-channel)
 Group Port- Type Protocol Member Ports Channel
 1 Po1(SU) Eth NONE Eth3/4(P)  - See ref above: 'P' means up and active in the port-channel 
 2 Po2(SU) Eth NONE Eth3/2(P) Eth3/6(P)
Examples of issues:

e0b: state broken, since…   This would indicate a cable unplugged, port down, or physical port issue.  The port is down.

e0b: state lag_inactive, since…   This would indicate the port is no participating in the lacp etherchannel port group.


Resources for more information:

NetApp KB for LACP port channel

Cisco troubleshooting Port Channel groups