Cloud Simplicity & Efficiency without ‘the cloud’

Cloud Simplicity & EEfficiencyfficiency

Labor is expensive and IT experts are VERY expensive which leaves many bank and credit union leaders with gaps and wants in their IT teams.  So how do you ensure you’re providing the support and IT services your financial institution (FI) needs?  Right now the answer being shouted from every vendor and corner of the IT world is to use the Cloud.  cloud, Cloud, CLOUD! ‘Why are you not using the cloud?’ is heard repeatedly.  There are non-public cloud options for FIs.  As FIs deal with regulators, security, and perception challenges with the public cloud, I want to share 2 options for how to get cloud simplicity and efficiency for your institution without the public cloud.  With these options you can shift or focus your IT team on more valuable business driving applications, business intelligence, and other key areas that impact your financial business.

Option 1:  Put together a managed environment with a trusted Managed Service Provider (MSP) focused on FIs.

MSPs have come a long way (to be more accurate, a few top tier MSPs) to help FIs than your traditional small/regional integrator/MSP.  The best ones are capable of:

  • Taking over all IT infrastructure for you. What does this mean?  It means they either hosting an independent environment for you in a top tier data center or in a multi-tenant environment run in a top tier data center.
  • Designing or providing IT architecture for you that meets FFIEC security, disaster recovery, and other key regulatory needs.
  • Management tools, reports, & dashboards that allow transparency, co-management, real-time and historic performance information.
  • National coverage with options for local support and local device management.
  • 24/7/365 alerting and assistance because you run an always on business and they understand your needs.
  • Expertise in multiple IT disciplines (storage, compute, data center, OS/domain admin, DB admins, security, DR/BCP, etc.) or partnerships with IT discipline experts to give you options for a fully managed environment.

 

Option 2: Use a Hyperconverged IT solution which provides a local cloud plus key backup, replication, and disaster recovery services.

There are 15+ hyperconverged solutions out there but not all are created equal nor are as mature in their services.  At this time, the traditional IT vendors like VMware, Cisco, Microsoft are all behind the curve with Hyperconvergence solutions having only recently introduced or initially stumbled and recently re-vamped their offerings.  They are becoming solid options but are not as feature rich as the top providers who were born as Hyperconvergence solutions. The top Hyperconvergence solutions right now are Simplivity and Nutanix.  I’ll highlight features below are present in both or at least one of these top Hyperconvergence solutions. They are capable of:

  • Replacing the full x86 IT stack. At a minimum this includes your storage, storage network, compute, and management of these components in a single management interface.
  • The management tools are VM object focused so you manage your VMs and don’t spend time or need expertise managing individual components supporting the VM.
  • These solutions work well with traditional Hypervisors or can include their own KVM branded hypervisor that is fully supported and integrated into your single management interface.
  • They provide snapshot and backup options for your VM. When combined with a secondary site as all CU & Banks need this can replace your backup software and be managed in the single management interface.
  • They are cluster aware and have native tools to replicate snapshots/VMs offsite and/or into other cloud services. This can in many cases replace SRM, Veeam, or Zerto and the replication is also managed in the single interface.
  • They even give you the benefit of their deduplication & compression technologies so you can put more data on the environment without adding more storage. In public and hosted clouds, the vendor often charges you for your storage and can use some of those benefits to their advantage instead of your advantage.
  • All this in 1 technology package and supported by 1 vendor who has experts in each area supporting your hyperconverged solution.

If you haven’t evaluated these options, it is time to start reading and having conversations about them.  Beyond the cloud simplicity & efficiency benefits of focusing your team on needs closer to your higher value business needs, they can also be cost savings options that help you better control and predict costs for the coming years.

While I believe the public cloud has great benefits, we can gain many of them today with the two options above.   I have helped customers explore and implement both options and would be happy to share my experience or connect you with others to gain from their experience and perspective.

 

 

RPO & RTO – Check. What about RTPO? (Return to Production)

RPO & RTO.  What about RTPO?

Don’t ignore this undervalued disaster recovery measurement

You know these:

  • RPO – recovery point objective
  • RTO – recovery time objective

But what is RTPO?

  • RTPO – Return to Production Objective

Everyone needs to consider the time & effort it will take to fail-back to their production equipment, location, and systems.  Yes I made it up, RTPO.  I didn’t make up the need though, just the abbreviation.

Unfortunately, not every solution or the way they are implemented takes into account the effort or time it will take to return to production environments.

You don’t want to go through a disaster and then turn around weeks later and do all the work over again to Return to Production systems & locations!  It’s like scheduling a second disaster.  Consider your RTPO when setting up your DR strategy.

When I help with storage solutions, Disaster Recovery (DR) is always part of the conversation so here is a chart covering at a high level the options for DR.

 

Scale 0-5 with lower being better

DR Options RPO RTO RTPO Comments

Examples

(just a sampling)

SAN replication

 

1 1 4 Replication can rarely be synchronous writes because of latency and distance between production and DR sites.  So data is never 100% up to day but even with asynchronous writes or snapshots the time to recover and recovery point are usually very quick.

 

The concern is returning to production.  Traditional solutions, depending on the type of issue or length of running out of the DR site can cause a full re-sync of data to get back to production.  Newer solutions and improved options do exist so only change deltas are synced back to the production site.  Know what your SAN can do to get back to productions!

·         EMC

·         NetApp

·         IBM

·         Pure

·         Dell

·         There are many, many SAN providers.

Hypervisor replication / DR tools

 

1 1 3 Hypervisor vendors figured out long ago how to do high availability in a shared storage environment and low latency situation like one data center.  Current day tools, with the right licensing can extend many of those capabilities to a DR site.  The tools, licensing, and exact abilities vary with each hypervisor & may also be based on your SAN/Storage replication ability.

 

Your return to production depends on the various components for your hypervisor and SAN. Be sure to know your Hypervisor and your SAN’s capabilities if using Hypervisor replication tools.

·         VMware

·         HyperV

·         Citrix Zen Server

·         KVM – various providers

VM Replication tools at the Application layer 2 1 2 Replication tools of this type usually interact at the VM or Hypervisor level.  They don’t rely on specific SAN or Hypervisor tools (some do need Hypervisor APIs) to replicate your data.  The advantage is that the secondary site & technology doesn’t need to look exactly like production.  You may be using older or slower but still capable storage & VM hosts.  You could also use a hybrid cloud solution so you don’t own any of the infrastructure at the DR site.  An extra benefit is that you should find these tools to be less complex than putting together SAN/hypervisor replication setups as they are designed to be one application for replication and recovery.

 

Return to production is done with delta changes and done on a per VM or VM group level not a full SAN or site level.

·         Veeam

·         Zerto

 

Backup software / appliances

 

4 4 4 Traditional backup solutions have the advantage of long term retention and many recovery points.  Unfortunately, they often create a longer recovery time as data has to be decrypted, copied, and/or restored from a 3rd party medium (HD or tape).  Some solutions do have rapid recovery options available that provide immediate temporary servers (read-only VMs with data change logs) or even delta restores that overwrite the production VM.  These advancements are not consistent across the vendors so know your options.

 

In general, these backup tools often take longer to restore and eventually require another restore back to production equipment if the ‘quick’ temp restore or recovery options were used.

·         StorageCraft

·         Commvault

·         Asigra

·         Acronis

·         Evault

·         Avamar

Application/DB specific tools 1 1 2 These tools are designed right into specific application or platforms that require high availability like databases and mail servers.  When DB clusters, DAG groups, or built in application replication is used for HA solutions it provides very quick recovery and by design an easy return to production environments.

 

To return to production these systems often try to first replicate data changes and only if necessary reseed a recent backup and then replicate only recent data changes.

·         SQL Always on Clusters

·         Distributed file System (DFS)

·         Exchange DAG groups

·         Oracle clusters & replication options.

I am not recommending one type of solution over another.  The ‘right’ solution or mix of solutions depends on your risk, architecture, skillsets, and budget.  I’ve helped set up all types of DR solutions and many clients have a combination of these solutions to meet their specific business needs or regulations.

I’ve experienced a RTP turning into a scheduled outage and redoing all the ‘disaster’ work just to move back to production systems.  Design and choose your DR strategy carefully & remember the Return to Production Objective is as important as RTO & RPO.

LostCreek Fintech handles all aspects of storage solutions and has partners who are experts in multiple disciplines to help our clients with other IT needs.  Contact us at info@lcfintech.com for more info.

Privacy Policy

Privacy Policy

Your privacy is a top priority for LostCreek Fintech.  We provide this privacy notice to explain how we collect, use and safeguard your personal information.  We are committed to providing you with competitive solutions, products, & services that meet your IT and regulatory needs, which necessitates that we collect and share information about you to provide you with certain IT opportunities & services.

In order to do so, we have entered into agreements with other companies that provide either services to us or additional IT services and products for you to consider. Under these arrangements, we may disclose all of the information we collect to companies that perform marketing or other services on our behalf or to other partner companies with whom we have joint marketing and/or service agreements. To protect our customers’ privacy, we only work with companies that agree to maintain strong confidentiality protections and limit the use of information we provide. We do not permit these companies to sell to other third parties the information we provide to them.

Create and send Netapp AutoSupport file

Howto: manually create and send Netapp AutoSupport file.

This file is often needed by Netapp or 3rd Party Maintenance providers for troubleshooting or quoting hardware or support.

Create the AutoSupport file:

  1. Login into Netapp with root/admin priviledge.
  2. Run the following command:  options autosupport.doit now 'insert your custom info here'
  3. A new folder and files will be created in the /vol/vol0/etc/log/autosupport directory.  The current date and time will be used for the folder and seen in the created/modified timestamp fields.
  4. Mount that path with NFS, or access it through a CIFS share in order to copy those files onto your system
  5. Pull files from the newly created folder on the /vol/vol0/etc/log/autosupport directory to your local machine.
  6. There will be 1 text file and a number of .gz files.
  7. Open the text file and copy the information into the body of your email.   Attach the .gz files to the email to your support provider.
  8. Send your email with the AutoSupport information needed by your vendor and your done!

Reference: Netapp KB

SP Collect for Clariion CX4 SAN

SP Collect for Clariion CX4 From Navisphere

Support will often ask for SP Collect (SP = Storage Processor) files.  They’ll ask for SP collect information from your EMC CLARiiON SAN / storage array. The SP Collect files will provide all the information needed to know basic status of your SAN.

Follow these steps below to create, transfer, and send SP collect on EMC SANs from Navisphere.

Navisphere Login 

You can login to navisphere from either head unit, either from SP A or SP B ip address. Make sure to enter the appropriate username and password in this screen to login.

Run SP Collect

From the Storage Tab, click on “Local Domain”, then click the serial number of the array to expand it further.

Now, select “SP A” and right mouse click, which will display a pop-up menu. In the pop-up menu click on “SP Collect”

Right click - run SP Collect

Execute SP Collect

This will show a confirmation dialogue with question “Do you want to run SP Collect script now?”. Click on “YES” to continue here.

Depending on the size of your array, and other configuration information on your array, this might take some time to finish. It may take 1 to 5 minutes.

Once it is done, it will display a success message. Click “OK” on this screen.

SP Collect feedback screens

File Transfer 

Select the SP A, and right mouse click, which will display a pop-up menu. In the pop-up menu click on “File Transfer Manager”.

File Transfer Menu - SP collect CX4 CX3

Transfer SP Collect Zip File to your PC or Laptop

This will display the following File Transfer Manager screen for SP A. The name of the SP Collect file will be in the following format:

[serial number of the array]_SPA_[date and time]_data.zip

You may see multiple zip files from you, your vendor, or others techs running prior SP collects. Make sure you select the correct file from his list. You can sort by the columns to help find the latest file.

Choose your “Destination Directory” on your local PC / Laptop or network storage before clicking transfer.

Click the “Transfer” button and the file will be moved to your PC / Laptop or network location you selected.

Once the transfer begins, you can see the status in the “Transfer Status” section.

Repeat for SP B

You should send SP Collect files from both your storage processors.  (SP A and SP B ).

So, repeat all of the above steps for SP B.  After you are done, you should now have two zip files to send to your EMC Support Partner.

Upload the files to your EMC Support vendor

Your provider should have a file transfer upload/receiving option such as a secure website or SFTP site for uploading the files.   Just upload the files to your EMC vendor and they will be able to read them and determine any next steps needed.