Data Center & SAN Move
What to do when the timing & budget don’t allow for duplicate hardware.
In a perfect world, you’ll always have the time and budget to stage a new environment with the storage and telecom capacity to migrate all your data to the new location for a move. Even with my refurbished and cost reduction options this hasn’t always been possible for my clients. Since there are only a few who live in the perfect world of large IT budgets this information is for the rest of us. Here are some highlights from my experience that might help you move your production systems.
First and foremost, know that it can be done.
It may be a high stress time but it can be done. There will be some downtime for applications that impact customers and communication with clients and adjusting to their needs are key to help make the move successful for everyone.
Expectation setting & communication is critical to your success:
- Develop a Communication Plan
- Identify all the parties that need to be kept up to speed
- Internal Departments.
- IT teams.
- Vendors assisting with the move.
- Set key milestones for communication to the various groups
- Regular interval – current status whether the news is good or bad, send it out.
- Triggered by events
- Systems going dark
- Leaving old Data Center
- System arrived at new Data Center
- Systems up & testing beginning
- Live system & maintenance window closure
Document & Label everything:
- Document everything, label everything, then go back and have a different person check everything for clarity and completeness.
- Label & document every cable, device, shelf, and port. Make sure you label both the cable end and the port on the SAN/Server/switch.
- Grab configs, take screenshots from mgmt. interfaces, get a copy of everything you can.
- SAN, network switches, storage switches, Vmhosts and any other connected systems.
- Take pictures. Yes, grab your camera and take pictures from far way, close up of each rack front & back, and then even closer for the backs of storage processors, servers, or switches.
- If/when you need to remember what was plugged into what when the label falls off the cable or server, then you’ll at least have a picture to reference.
Expect something to go bad & prepare for the disaster:
I’ve seen things dropped, vibrate loose in shipment, or moving trucks breakdown or get in minor accidents. Here are some things you can do.
- Develop & Test shutdown and startup scripts. Make sure you have a checklist with the order & dependancies of what is turned off & on in what order.
- This is for data safety as well as for efficiency. With many people working and as the move progresses and fatigue sets in, you need to make sure you’re addressing systems in the right order.
- Review this regularly before the move and when possible practice it on test systems to know everything will go smoothly. You don’t want an error in your shutdown or startup processes causing delays or data integrity issues.
- Make sure someone owns the orchestrator/coordinator role and have key milestones signed off on before the team continues to the next major step.
- If you know you need all guest VMs turned off before turning off hosts then create a ‘pause’ and have someone double check before proceeding. The same holds true with making sure all systems are off before turning off your SAN & switches.
- A delay is bad; loss of data or a system is much, much worse. Plan for the worst.
- Check Backups and test that they are good before proceeding with a move.
- Get full images of critical systems or applications whenever possible.
- That secondary SAN may not be in the budget but I bet a refurbished NAS or at the very least a few USB hard drives can be purchased if it means protecting key system data.
- Use Key Vendor relationships
- I recommend having your vendors scheduled for your move. That could mean having them participate in the shutdown/startup process milestones or at the very least on-call.
- In my experience it can be worth the money to have the SAN vendor on-site at the new data center location.
- They can quickly help address any hardware issues.
- I always have them come with some parts in hand.
- An extra power supply, hard drive of each drive type for your SAN, and possibly even a storage processor/controller if you can get them to bring one.
- When you’re in the 2nd half of your maintenance window, you don’t want to wait 4hrs or more for your support vendor(s) to be responding to your request.
- Separate production and backup
- After all your prep work make sure you put your backups in a different truck or transport vehicle. You don’t want to lose everything in 1 vehicle accident.
- I’ve had IT staff transport encrypted drives & backups in a separate car taking a separate route or delayed by 1-2 hours behind the primary moving truck.
Take care of your staff:
- You can’t be everywhere and check everything yourself. Take care of your staff and prepare them.
- Talk through scenarios, checklists, and expectations before the move.
- Treat movers, packers, and other laborers as part of your staff and set expectations with them. Make sure you tip them too as you’ve probably just asked them to work/drive through the day/night to help your business.
- Use rotating shifts for staff.
- Tired staff make mistakes. Whether it is management or engineers, make sure you rotate through people so they stay fresh, get rest, get food & drink, and stay healthy.
Celebrate when you are done:
- Celebrate your success and learn from your challenges.
- Do a move review a week or two later to learn from the challenges and improve processes for future moves or system maintenance.
- Make sure the final IT setup is well documented for the future before you close down your project.
- Celebrate with your staff. This can be food, company swag, a party, bonuses, or more. The success of the move was because of your staff’s hard work. Show your appreciation.
LostCreek Fintech handles all aspects of storage solutions. Moving a live production storage environment is not ideal but can be done with the right planning and support. Hopefully our experience helps you through your next move. We are here to help if you ever need storage options, support, or advice for a move or storage project.