Data Center & SAN Move – Tips and Tricks

Data Center & SAN Move

What to do when the timing & budget don’t allow for duplicate hardware.

Data Center

In a perfect world, you’ll always have the time and budget to stage a new environment with the storage and telecom capacity to migrate all your data to the new location for a move.  Even with my refurbished and cost reduction options this hasn’t always been possible for my clients.  Since there are only a few who live in the perfect world of large IT budgets this information is for the rest of us.  Here are some highlights from my experience that might help you move your production systems.

First and foremost, know that it can be done.

It may be a high stress time but it can be done.  There will be some downtime for applications that impact customers and communication with clients and adjusting to their needs are key to help make the move successful for everyone.

Expectation setting & communication is critical to your success:

  1. Develop a Communication Plan
    1. Identify all the parties that need to be kept up to speed
    2. Clients
    3. Internal Departments.
    4. IT teams.
    5. Management
    6. Vendors assisting with the move.
  2. Set key milestones for communication to the various groups
    1. Regular interval – current status whether the news is good or bad, send it out.
    2. Triggered by events
      1. Systems going dark
      2. Leaving old Data Center
      3. System arrived at new Data Center
      4. Systems up & testing beginning
      5. Live system & maintenance window closure

Document & Label everything:

  1. Document everything, label everything, then go back and have a different person check everything for clarity and completeness.
    1. Label & document every cable, device, shelf, and port. Make sure you label both the cable end and the port on the SAN/Server/switch.
    2. Grab configs, take screenshots from mgmt. interfaces, get a copy of everything you can.
      1. SAN, network switches, storage switches, Vmhosts and any other connected systems.
    3. Take pictures. Yes, grab your camera and take pictures from far way, close up of each rack front & back, and then even closer for the backs of storage processors, servers, or switches.
      1. If/when you need to remember what was plugged into what when the label falls off the cable or server, then you’ll at least have a picture to reference.

Expect something to go bad & prepare for the disaster:

I’ve seen things dropped, vibrate loose in shipment, or moving trucks breakdown or get in minor accidents.  Here are some things you can do.

  1. Develop & Test shutdown and startup scripts. Make sure you have a checklist with the order & dependancies of what is turned off & on in what order.
    1. This is for data safety as well as for efficiency. With many people working and as the move progresses and fatigue sets in, you need to make sure you’re addressing systems in the right order.
    2. Review this regularly before the move and when possible practice it on test systems to know everything will go smoothly. You don’t want an error in your shutdown or startup processes causing delays or data integrity issues.
    3. Make sure someone owns the orchestrator/coordinator role and have key milestones signed off on before the team continues to the next major step.
      1. If you know you need all guest VMs turned off before turning off hosts then create a ‘pause’ and have someone double check before proceeding. The same holds true with making sure all systems are off before turning off your SAN & switches.
  2. A delay is bad; loss of data or a system is much, much worse. Plan for the worst.
    1. Check Backups and test that they are good before proceeding with a move.
    2. Get full images of critical systems or applications whenever possible.
      1. That secondary SAN may not be in the budget but I bet a refurbished NAS or at the very least a few USB hard drives can be purchased if it means protecting key system data.
  3. Use Key Vendor relationships
    1. I recommend having your vendors scheduled for your move. That could mean having them participate in the shutdown/startup process milestones or at the very least on-call.
    2. In my experience it can be worth the money to have the SAN vendor on-site at the new data center location.
      1. They can quickly help address any hardware issues.
      2. I always have them come with some parts in hand.
        1. An extra power supply, hard drive of each drive type for your SAN, and possibly even a storage processor/controller if you can get them to bring one.
      3. When you’re in the 2nd half of your maintenance window, you don’t want to wait 4hrs or more for your support vendor(s) to be responding to your request.
  4. Separate production and backup
    1. After all your prep work make sure you put your backups in a different truck or transport vehicle. You don’t want to lose everything in 1 vehicle accident.
    2. I’ve had IT staff transport encrypted drives & backups in a separate car taking a separate route or delayed by 1-2 hours behind the primary moving truck.

Take care of your staff:

  1. You can’t be everywhere and check everything yourself. Take care of your staff and prepare them.
    1. Talk through scenarios, checklists, and expectations before the move.
    2. Treat movers, packers, and other laborers as part of your staff and set expectations with them. Make sure you tip them too as you’ve probably just asked them to work/drive through the day/night to help your business.
    3. Use rotating shifts for staff.
      1. Tired staff make mistakes. Whether it is management or engineers, make sure you rotate through people so they stay fresh, get rest, get food & drink, and stay healthy.

Celebrate when you are done:

  1. Celebrate your success and learn from your challenges.
    1. Do a move review a week or two later to learn from the challenges and improve processes for future moves or system maintenance.
    2. Make sure the final IT setup is well documented for the future before you close down your project.
    3. Celebrate with your staff. This can be food, company swag, a party, bonuses, or more.  The success of the move was because of your staff’s hard work.  Show your appreciation.

LostCreek Fintech handles all aspects of storage solutions. Moving a live production storage environment is not ideal but can be done with the right planning and support.  Hopefully our experience helps you through your next move.  We are here to help if you ever need storage options, support, or advice for a move or storage project.

 

 

 

Manually send Data Domain autosupport files

Manually send Data Domain autosupport files:

First confirm where your autosupport files are going to be send:

http://www.lcfintech.com/where-are-my-data-domain-autosupport-support-files-sent/

Get autosupport files/information from restorer web interface:

  • There is no way to send the information from the web interface but you can view, copy, and email manually:
  • Login to the restorer using a browser.
  • Click on: “support”
  • Click on: “autosupport”
  • Select all text, and copy
  • Paste the text in an Email and forward to your support team or vendor

Send autosupport from command line interface:

  • Access the Command Line Interface (CLI) using a serial cable, telnet, or an SSH client.
  • After logging on type:
    • autosupport send
      • An autosupport should be sent to any Email address that is on the autosupport list.
    • To view the support list type:
      • autosupport show support-list
    • To add to the support list, type:
      • autosupport add <email address>
    • To send an autosupport to a specific email address, type:
      • autosupport send user@email_address.com

Where are my Data Domain autosupport support files sent?

Determine where Data Domain autosupport support files go:

EMC says: Autosupports contain basic configuration and performance data and are very useful in the trouble-shooting process.

Why might I need to determine where my support / autosupport files are going?

  1. EMC is not receiving them
  2. You are no longer under support and need diagnostic info
  3. You are using a 3rd party support vendor
  4. You need the information for competitive quotes or planned migration to another technology

How to get the support email list:

  • Access the Command Line Interface (CLI) using a serial cable, telnet, or an SSH client.
  • After logging on to the restorer, type:  autosupport show support-list 
    • This should show you the support list email addresses

To add to the support list:

  • type: autosupport add <email address>

How to manually send your autosupport files: http://www.lcfintech.com/manually-send-data-domain-autosupport-files/

SP Collect for Clariion CX3 SAN

SP Collect for Clariion CX3 From Navisphere

Support will often ask for SP Collect (SP = Storage Processor) files.  They’ll ask for SP collect information from your EMC CLARiiON SAN / storage array. The SP Collect files will provide all the information needed to know basic status of your SAN.

Follow these steps below to create, transfer, and send SP collect on EMC SANs from Navisphere.

Navisphere Login 

You can login to navisphere from either head unit, either from SP A or SP B ip address. Make sure to enter the appropriate username and password in this screen to login.

Run SP Collect

From the Storage Tab, click on “Local Domain”, then click the serial number of the array to expand it further.

Now, select “SP A” and right mouse click, which will display a pop-up menu. In the pop-up menu click on “SP Collect”

Right click - run SP Collect

Execute SP Collect

This will show a confirmation dialogue with question “Do you want to run SP Collect script now?”. Click on “YES” to continue here.

Depending on the size of your array, and other configuration information on your array, this might take some time to finish. It may take 1 to 5 minutes.

Once it is done, it will display a success message. Click “OK” on this screen.

SP Collect feedback screens

File Transfer 

Select the SP A, and right mouse click, which will display a pop-up menu. In the pop-up menu click on “File Transfer Manager”.

File Transfer Menu - SP collect CX4 CX3

Transfer SP Collect Zip File to your PC or Laptop

This will display the following File Transfer Manager screen for SP A. The name of the SP Collect file will be in the following format:

[serial number of the array]_SPA_[date and time]_data.zip

 

You may see multiple zip files from you, your vendor, or others techs running prior SP collects. Make sure you select the correct file from his list. You can sort by the columns to help find the latest file.

Choose your “Destination Directory” on your local PC / Laptop or network storage before clicking transfer.

Click the “Transfer” button and the file will be moved to your PC / Laptop or network location you selected.

Once the transfer begins, you can see the status in the “Transfer Status” section.

Repeat for SP B

You should send SP Collect files from both your storage processors.  (SP A and SP B ).

So, repeat all of the above steps for SP B.  After you are done, you should now have two zip files to send to your EMC Support Partner.

Upload the files to your EMC Support vendor

Your provider should have a file transfer upload/receiving option such as a secure website or SFTP site for uploading the files.   Just upload the files to your EMC vendor and they will be able to read them and determine any next steps needed.