Migrating volumes with Linux LVM

Client has existing data on a LUN provisioned from an old array that is to be decommissioned. The data must be migrated online to a new LUN. However, the old LUN must be retained intact so as to facilitate a rollback in case the new array doesn't work right.

This situation isn't complex at all, except for that second requirement...to keep the original LUN intact and consistent. That means a simple pvmove operation is out of consideration. So let's see what else we can do.

The orginal situation:

growler / # df -h /mnt/prod
/dev/mapper/prodVG-prodLV 2.0G 73M 1.9G 4% /mnt/prod

growler / # ls /mnt/prod/
rtv10n1.pdf rtv2n3.pdf rtv4n2.pdf rtv6n1.pdf rtv8n21.pdf
rtv1n1.pdf rtv2n4.pdf rtv4n3.pdf rtv6n2.pdf rtv9n1_HR.pdf
rtv1n2.pdf rtv3n1.pdf rtv4n4.pdf rtv6n4.pdf rtv9n2 LR-web.pdf
rtv1n3.pdf rtv3n2.pdf rtv5n1.pdf rtv7n2.pdf RTVOL6N3.pdf
rtv1n4.pdf rtv3n3.pdf rtv5n2.pdf tv7n3v10.pdf
rtv2n1.pdf rtv3n4.pdf rtv5n3.pdf rtv7n4.pdf
rtv2n2.pdf rtv4n1.pdf rtv5n4.pdf rtv8n11_web.pdf

growler prod/ # md5sum * > /root/orig.md5sum

growler / # lvm
lvm> lvdisplay -m
--- Logical volume ---
LV Name /dev/prodVG/prodLV
VG Name prodVG
LV UUID K1ni9A-Q1qU-8xD1-Po4g-i0Y9-tYQN-C34riv
LV Write Access read/write
LV Status available
# open 1
LV Size 1.95 GiB
Current LE 499
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
--- Segments ---
Logical extent 0 to 498:
Type linear
Physical volume /dev/loop1
Physical extents 0 to 498

lvm> vgdisplay
--- Volume group ---
VG Name prodVG
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.95 GiB
PE Size 4.00 MiB
Total PE 499
Alloc PE / Size 499 / 1.95 GiB
Free PE / Size 0 / 0
VG UUID toXyOv-6YX2-te1E-28fS-1kFh-KtGo-XEGI5S

lvm> pvdisplay
--- Physical volume ---
PV Name /dev/loop1
VG Name prodVG
PV Size 1.95 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 499
Free PE 0
Allocated PE 499
PV UUID 3aOE5g-FyCf-z38s-NV7W-XY29-6E5X-8v7N0g
"/dev/loop2" is a new physical volume of "1.95 GiB"
--- NEW Physical volume ---
PV Name /dev/loop2
VG Name
PV Size 1.95 GiB
Allocatable NO
PE Size 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID GqZgUS-D0wj-6DHj-2lFs-YTqV-MJLy-JmlP4x

Demonstration of Solution

/dev/loop1 is playing the part of the old LUN that is being decommissioned. /dev/loop2 is playing the role of the LUN provisioned from the new array. I'm using loopback devices for two reasons: first, growler doesn't have any extra spindles and second, using loopback devices will hopefully protect anyone who is using this blog as a recipe from inadvertently destroying their live volumes.
The next step is to extend the existing volume group onto the new loop2 device:

lvm> vgextend prodVG /dev/loop2
Volume group "prodVG" successfully extended

With that done, now we can convert the existing logical volume prodLV into a mirror with two legs, one of which is on the new disk. LVM gives you a helpful progress report. Note that I'm using a memory-based sync log (corelog option) instead of a bitmap logging volume. Since this isn't intended to be a long term mirror, it's safe and faster to use a memory log instead of the more traditional logging volume.

lvm> lvconvert -m1 --corelog prodVG/prodLV /dev/loop2
prodVG/prodLV: Converted: 31.3%
prodVG/prodLV: Converted: 35.1%
prodVG/prodLV: Converted: 38.3%
prodVG/prodLV: Converted: 42.3%
prodVG/prodLV: Converted: 45.3%
prodVG/prodLV: Converted: 47.5%
prodVG/prodLV: Converted: 49.3%
prodVG/prodLV: Converted: 51.5%
prodVG/prodLV: Converted: 54.5%
prodVG/prodLV: Converted: 58.5%
prodVG/prodLV: Converted: 62.3%
prodVG/prodLV: Converted: 65.7%
prodVG/prodLV: Converted: 70.1%
prodVG/prodLV: Converted: 73.3%
prodVG/prodLV: Converted: 77.2%
prodVG/prodLV: Converted: 80.8%
prodVG/prodLV: Converted: 84.4%
prodVG/prodLV: Converted: 88.0%
prodVG/prodLV: Converted: 92.6%
prodVG/prodLV: Converted: 97.4%
prodVG/prodLV: Converted: 100.0%
Logical volume prodLV converted.

And now we can see prodLV is mirrored across the two disks:

lvm> lvdisplay -m
--- Logical volume ---
LV Name /dev/prodVG/prodLV
VG Name prodVG
LV UUID K1ni9A-Q1qU-8xD1-Po4g-i0Y9-tYQN-C34riv
LV Write Access read/write
LV Status available
# open 1
LV Size 1.95 GiB
Current LE 499
Mirrored volumes 2
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
--- Segments ---
Logical extent 0 to 498:
Type mirror
Mirrors 2
Mirror size 499
Mirror region size 512.00 KiB
Mirror original:
Logical volume prodLV_mimage_0
Logical extents 0 to 498
Mirror destinations:
Logical volume prodLV_mimage_1
Logical extents 0 to 498

This next step turns off the mirror leg that is writing to the original LUN. I've chosen to do this 'hot' with an active filesystem. Whether this is wise or not depends on the type of filesystem and type of workload. Even here, it creates a couple extra steps. In a perfect world, I'd take a 10 second outage to dismount the volume, split the mirror, and remount the volume.

lvm> lvconvert -m0 prodVG/prodLV /dev/loop1
LV prodVG/prodLV_mimage_0 in use: not deactivating

Now a cleanup step. LVM won't automatically remove the mirror image devices because the volume was active when I split it. So I get to do it manually. It'd be a good idea to do a lvdisplay -am before you do the lvremove, to ensure all the extents are where they're supposed to be, and that prodVG/prodVL is showing mapped exclusively to loop2.

lvm> lvremove -f prodVG/prodLV_mimage_0
Do you really want to remove active logical volume prodLV_mimage_0? [y/n]: y
Logical volume "prodLV_mimage_0" successfully removed
lvm> lvremove -f prodVG/prodLV_mimage_1
Do you really want to remove active logical volume prodLV_mimage_1? [y/n]: y
Logical volume "prodLV_mimage_1" successfully removed

OK, now we can split the volume group apart. Afterwards, we recreate the logical volume in it's new volume group, on exactly the same extent boundaries. Notice the -l499 parameter. 499 was the "Current LE" size in the very first lvdisplay that I got.

lvm> vgsplit prodVG backoutVG /dev/loop1
New volume group "backoutVG" successfully split from "prodVG"
lvm> lvcreate -l499 -n backoutLV backoutVG
Logical volume "backoutLV" created

Now the last step is to fsck the backout device. This step is only necessary because I split the mirror while it was mounted.

growler linux # fsck /dev/backoutVG/backoutLV
fsck from util-linux-ng 2.17.2
fsck.jfs version 1.1.14, 06-Apr-2009
processing started: 7/23/2010 15.27.53
Using default parameter: -p
The current device is: /dev/mapper/backoutVG-backoutLV
Block size in bytes: 4096
Filesystem size in blocks: 510976
**Phase 0 - Replay Journal Log
Filesystem is clean.

And just to prove everything is copesetic, I will mount the backout volume and compare the md5sums of it's contents, and the current production volume contents, with the checksum file I took at the beginning of this exercise:

growler linux # mount -t jfs /dev/backoutVG/backoutLV /mnt/oldLun
growler prod # cd /mnt/oldLun/
growler oldLun # md5sum --quiet -c /root/orig.md5sum
growler oldLun # cd ../prod
growler prod # md5sum --quiet -c /root/orig.md5sum

Any bit errors would have complained. But I got no errors, and LVM is happy. Eventually, I'll tear down backoutLV and backoutVG and remove the loop1 device from LVM. Then at long last, I can disconnect and decommission the old array.

Migrating volumes with Linux LVM