Data Migrations

There are more ways to shave this yak than I care to write about, but the main methodologies I’ve used time and time again with maximum success and low risk boil down to a select few.

• Array based technologies such as replication or virtualization
• Host based LVM copies
• For VMware, Storage vMotion.

There are emerging technologies that are quite interesting but I’ll go into those at another time. Over the past year I have managed several large-scale data migrations, and I find that the same questions come up each time, specifically:

1. My database is very large, how are we going to move from array A to array B with minimal downtime?
2. Downtime is difficult to schedule, what steps can we take to eliminate it?
3. I have small LUNs today, but as the system keeps growing it becomes onerous to manage. How can we move to larger LUNs?

While Lumenate has a consistent migration process, the answer to these questions is unique to the customer’s environment. This post is meant as a starting point in the process and help you make informed decisions for your technology refresh or data migration.

Let's start with the easiest case first. VMware changes the game in many ways by abstracting physical resources from running guests. Migrations, be it compute or storage resources, can be done with relative ease. The majority of the time outages are not necessary, and services are never interrupted. There are times when storage vMotion (virtual machine migration from one datastore to another datastore) is not possible. This may be due to raw device mappings, full datastores, or other reasons, but these are relatively rare. Outside of these situations Lumenate always recommends storage vMotion. The steps are pretty much the same as above. With proper planning and staging, the migration is typically quick and easily managed from the familiar vSphere client.

Of course not every host is virtualized, so storage vMotion isn't always an option.

Whether structured or unstructured in nature, moving large amounts of data quickly and with minimal service interruption can be difficult. Typically, consistent and repeatable results are found by leveraging array technologies in these scenarios. In cases where I can leverage HDS technology, specifically the enterprise line, virtualization has provided the best results. A typical work flow goes something like this:

Planning

• Determine what LDEVs need migrated, then document, document, document!

Staging

• Initiate connectivity between storage arrays (could be direct attached or through switches)
• Prepare connectivity between target array and host (build zones, but do not activate)
• Create new LUNs on the target array
• If using array-to-array replication, this is a good time to configure and sync pairs.
• Configure LUN security on the target array

Outage Window

• Typically plan for a two hour event, but normally takes under an hour
• Bring applications down, file systems un-mount and as a safety precaution, shut down the host
• Deactivate old array zones, activate the target array zones
• If using array virtualization, discover LUNs on target array, and configure LUN security to the host. (Normally, LUNs are presented in same order as found on the old array)

Finalizing

• Boot host, discover storage and bring services back online
• If utilizing HDS virtualization, configure Tiered Storage Manager and migrate storage from old array to new array.

Back-out plan

• If utilizing array to array replication, you will have a point in time copy of the data on your old storage to fall back to
• In the case of HDS Tiered Storage Manager, if you do not select "wipe data" when building your migration plans, the old data will remain on the old LDEV. It is important to note that TsM doesn't operate like replication, there are no consistency groups. While the data is there it may not be crash consistent like you would expect from traditional replication.

When the business cannot take an outage at all your choices become more limited. Logical volume management at the host level to copy data between arrays tends to be the standard answer in this scenario. There are advantages to this over array based technologies. For starters, we generally avoid an outage requirement as storage can be mapped to modern operating systems while on-line. Additionally you begin to shift the focus of migration related activities more toward the system administrators and off of the storage team; a great benefit if you are the storage lone ranger in your shop. Should you need to resize your volumes or LUNs there is no better way to do it than with LVM (unless you just need to grow them, then most modern arrays can do the job just fine).

Planning

• As above, document the environment

Staging

• Prepare connectivity between target array and host (build zones, but do not activate)
• Create new LUNs on the target array
• If using array-to-array replication, this is a good time to configure and sync pairs
• Configure LUN security on the target array

Migration

• Activate new fabric zones
• Discover LUNs on the host, format, and bring under LVM control
• Mirror LUNs

Finalizing

• Verify Mirror status
• Break the mirror by removing the old array LUNs. (If you are looking for a consistent point in time “snapshot” of your data on the old array, consider taking applications and file systems off-line for this step.)
• De-allocate old storage

Back-out plan

• When breaking the LVM mirrors, consider taking applications and file systems off-line to provide a consistent point-in-time copy of data
• Keep this extra copy of data for some determined length of time, in case of regression.

Data Migrations