To be sure that any changes in production make their way to the DR site, the customer is replicating not only the volumes associated with the database, but the root/boot device as well.
In addition, all devices are multipathed using Linux native multipathing.
In preparation for the initial DR test we split the TrueCopy pairs and tried to boot the system at the DR site off of the replicated device. Unfortunately the /boot filesystem failed to mount, causing the boot process to fail and the root device to be mounted read-only. We were able to successfully mount the devices using "mount -o remount,rw" against the underlying non-multipathed devices, but could not get the system to configure a multipath device for root at boot.
Frustratingly, even trying to force multipathd to create an entry for the root device failed. Eventually we discovered that initrd includes a multipath configuration file (etc/multipath.conf) and that it blacklists everything except the root device. More specifically everything except the root device's wwid. Here's an example from a VirtualBox Guest:
I plan to go into more detail on wwids in a future post, but for now it's enough to note that they are (typically) derived from a combination of the array's serial number and the LUN number (LDEV in HDS speak). Since we are replicating to a different array the wwid doesn't match and as a result the root device at the DR site is blacklisted.
The fix was to extract initrd at the primary site, update multipath.conf to include the wwids for both sites and then recreate initrd.
NOTE: This was originally observed on RHEL 5.6. The listing above was taken from a CentOS 6 box, which uses initramfs instead of initrd, but the issue (and fix) are the same.
No comments:
Post a Comment