Storage Meat

HDS Hitachi Content Platform HCP

2012-01-05T06:31:00.000-08:00

Back in 2007, HDS purchased a small software company named Archivas. Archivas was formed by some ex-EMC folks. They set out to create a product that was better than EMC's Centerra archive platform. The product became known as the HDS HCAP platform after the HDS acquisition. It was specifically designed to be an archive platform for fixed content. It answered the need to comply with recent regulations requiring certain industries to retain immutable copies of their data. The HCAP platform provided client access via standard network attached file system APIs such as CIFS and NFS.

HDS realized that the market desired more than just a strict compliance platform. Decreasing storage cost in the SATA and nearline disk technologies provided the cost justification for organizations to utilize online archive/repositories rather than cutting tapes. With the maturation of virtualization technology, why purchase multiple silos or platforms. Hence, HDS removed the "A" from HCAP and created HCP.

HCP can still provide the strict compliance archive functionality of its predecessor, but the new version adds the capability to have virtual entities for sharing the platform across business units, groups and even allows for service provisioning. The virtual entities are comprised of two levels: Tenants and Name Spaces. The Tenants provide an administration domain for its Name Spaces and the Name Spaces provide data space for authenticated users.

The supreme user for the platform is the Service Console Administrator which manages physical attributes of the platform and creates Tenants. Upon Tenant creation, an Admin user is specified along with a quota of storage space. From there, the administrator for the tenant can create Name Spaces with quotas carved from the Tenant quota. The Tenant administrator manages Data Access Accounts which are used to authenticate users for file manipulation in the name space. The Data Access Accounts are the actual end user accounts for storing and retrieving data from the Name Space. Quotas and access rights are inherited from the parent and can be reduced by the administrator at each level. So each administration level can tweak the permissions that it allows for its children. Note the Service Console Administrator does not have access to data in the Name Spaces, so security is maintained within a Tenant.

Two different operating modes are available for Tenants. They can be run in a strict Compliance mode or the default Enterprise mode. Compliance mode offers similar functionality to the original HCAP. The Enterprise mode offers more flexibility in a repository type implementation. In Enterprise mode a "Privileged Delete" function can be passed down to allow a Tenant Administrator to delete files that are under retention. All activities are recorded and the privileged delete function requires that a description of the activity be entered into the log. And on that note, the log is never truncated or deleted. So a permanent record exists for all administrative activities.

In order for HDS to provide the virtual functionality, they had to make a tradeoff on the APIs that would be available. Traditional CIFS and NFS access are provided for the "Default Name Space," which mimics the initial HCAP implementation. As for the Tenant name spaces, access is limited to the REST API interface via HTTP or HTTPS. REST (Representational State Transfer) is a software architecture that defines communications between clients and servers via HTTP(S). HDS provides a few utilities with the platform that can migrate and copy data to HCP via the REST interface. Otherwise, you need an application that can communicate with HCP. Many popular backup and archive packages support communication with HCP. In addition, HDS provides a product named Hitachi Data Ingestor (HDI) which provides a traditional CIFS/NFS interface, but communicates to the HCP via HTTPS/REST. The HDI appliance serves as a cache and can be used for remote site to central site archiving.

So how does HCP store data and ensure its integrity? In general, two types of HCP appliances are available: HCP300 & HCP500. Each utilizes Hitachi's CR220 – 2U rackmount server. The HCP300 uses the internal drive bays on the server for the repository data and appliance boot. In contrast, the HCP500 has no internal disks and uses a Hitachi disk array for both boot and data repository storage. The HCP500 Appliance version comes with an AMS2000 series array and the diskless version relies on customer provided Hitachi storage. Of course, the HCP500 is a little more cumbersome to setup, but provides more scaling and flexibility. Both versions checksum the data and keep two copies of the checksum. In the event of a disagreement, the data or checksum are repaired if possible. In replication configurations (one HCP replicates to another local or remote) corrupt data blocks can be recovered/repaired from the remote HCP system.

In order to locate and manage data, HCP provides a search function via Search Nodes. Search Nodes can be added to the cluster as necessary to increase search index scalability. The search function allows a Data Access Account user with search permission to locate files using a variety of metadata attributes and perform subsequent operations on the search set.

As for scaling, the HCP300 scales horizontally by adding nodes, whereas, the HCP500 can scale by adding nodes or mapping additional storage to existing nodes. The HCP300 requires that two copies of all data be stored, whereas, the HCP500 requires only one copy due to RAID technology in the AMS2000 array. Up to four copies of data can be stored depending on user requirements. Both configurations are configured as "clusters." Failure of a single node in either configuration will allow continued data access. Even failure of multiple nodes without loss of data access is possible, unless it is the wrong two nodes. In the HCP300 most likely some data will be inaccessible for a multiple node failure situation. Unaffected data will remain available. In the case of the HCP500, multiple node failures can bring the entire HCP "cluster" down due to loss of quorum. It depends on the number of nodes in the configuration.

So, what about the CLOUD? As discussed previously, the HCP300 or HCP500 can provide a central repository for remote locations utilizing HDI as a gateway. In addition to being just a gateway, the Hitachi HDI appliance provides local caching, compression and encryption. When we put all the pieces together, we have an internal storage cloud that is secure, searchable and maintains necessary compliance attributes.

Storage MBps Performance Concepts

2011-10-28T09:00:00.000-07:00

We recently had a reader post a question about the other aspect of storage performance – MBps. In most cases my clients are concerned with IOPS more so than transfer rate so it doesn’t come up that often. When it does it normally gets back to economics, applications such as streaming video, audio or even backups tend to be sequential and require a lot of capacity. As with IOPS the higher RPM drives tend to outperform the lower RPM drives but the difference in not nearly as substantial as it is with IOPS.

Here are some data transfer performance numbers of a few modern disk drives. I’m using Seagate drives as a reference point simply because their website is user friendly. Note that this information is not listed directly on the website but it can be found in the product manual that you can download once you click on the specifications tab. Be aware that each manual includes the internal data rate as well as the sustained data rate. The internal data rate is often called the burst rate and represents the rate to / from the buffer cache on the drive. The sustained rate represents the actual transfer rates from the drive media itself. The sustained transfer rate is the number you should be focusing on.

Seagate specifies the internal data rate numbers on their 15K and 10K drives in Gbps while their 7200RPM drives are listed in MBps. I’ve listed them all in Megabytes per second to make it easier to understand.

Cheetah 15K.7 SAS 6-Gb/s 600-GB Hard Drive Model # ST3600057SS
· Internal data rate (Burst Rate) 190 -303MB/s
· Sustained Transfer Rate 122 – 204 MB/s
· Areal Density 225 Gbits/inch2

Cheetah NS.2 SAS 6-GB/s 600-GB Hard Drive Model # ST3600002SS (10K RPM)
· Internal data rate (Burst Rate) 129.28 – 235.52MB/s
· Sustained Transfer Rate 82 – 150 MB/s
· Areal Density 225 Gbits/inch2

Seagate® Constellation® ES.2 SAS 6Gb/s 3-TB Hard Drive Model # ST33000650SS (7200 RPM)
· Internal data rate (Burst Rate) 113 to 239 MB/s
· Sustained Transfer Rate Sustained Transfer Rate 82 – 150 MB/s
· Areal density 444Gb/in2

A couple of things to note.

· The 15K and 10K drives listed above are identical with the exception of RPMs. The RPMs on the 15K drive make it outperform the 10K drive. This is as expected.

· What may be surprising is that the sustained transfer rates of the 10K drive and the 7200 RPM drive are identical. How can that be? The answer is that the 7200 RPM in this example has a higher recording density, or more correctly, areal density. Areal density describes how many tracks are on a platter and how many sectors are on each track. The number of tracks isn’t relevant to performance but the number of sectors per tracks is. Since the data is more tightly packed on the 7200 RPM drive it reads more sectors, per rotation than the 10K drive and therefore evens out the total transfer rates. By contrast the 15K drive operates at over twice the speed of the 7200 RPM drive and is able to maintain its performance advantage even with a smaller areal density.

The third element to consider is which tracks are being accessed during the IO. Outer tracks provide the highest performance and performance declines as you move towards the center of the drive.

· The outer tracks contain more sectors than the innermost tracks. This concept is referred to zone bit recording, where the drive is divided into zones and each zone has X number of tracks with Y number of sectors per track. The results it that in a single rotation you can read more data from an outer track. To read the equivalent amount of data from the center of the drive would require accessing multiple tracks and you would experience the performance penalty associated with the extra read / write head positioning.

This is why the sustained transfer rates are always provided in a range, because it depends on which tracks are involved in the IO.

So that’s the logic behind it, as for the math, you would need to look at the RPM of the drive and the sectors per track, which as we mentioned would vary depending on the zone. Here is an example using the Seagate 15K.7 drive listed above.

Keep in mind that it is not always easy to find documentation that will provide the zoning information that will allow you to do this exercise. From a practical standpoint I deploy storage for a living and I’ve never had the occasion to calculate the MBps for a drive since the manufacturer provides this number as long as you know where to look, meaning the sustained transfer rate rather than the internal transfer rate. That being said I personally found this to be a very interesting exercise. When I first started writing these performance entries I envisioned a few simple posts that would cover the most common scenarios and questions we were asked. As more questions came in there always seemed to be another area to explore. As an example, from a practical standpoint I work with arrays not individual disk drives and I know that most storage arrays do not provide the type of MBps performance you would expect if you simply added up the capabilities of each drive in the array. This is a given and frankly I’ve never spent much time trying to determine why this is. I know some things that impact it and I have some suspicions but I’ve never satisfied the question in my mind. As organizations begin to shift from tape to disk based backups the MBps capabilities of an array will become more important. I guess I need to look into that question as well.

Cisco MDS switch interoperability with Brocade Access Gateway and Qlogic Adapters

2011-10-24T08:58:00.000-07:00

During a recent SAN switch migration from McData directors to Cisco MDS 9509 switches, I came across two interoperability issues. The first issue involves certain Qlogic adapters connected to the Cisco MDS switches where the area portion of the Fiber Channel ID (FCID) is the same for the initiator and target. The second issue has to do with internal routing in the Brocade 4Gb switches running in "Access Gateway" mode. In my case the Brocades were embedded in blade chassis, but they could be stand alone. Note, the 8Gb version of the Brocade switches did not exhibit the routing issue.

Common to both issues are particular FCID combinations. The FCID is comprised of 3 bytes, 0xDDAAPP, where the letters stand for Domain, Area and Port numbers. The domain portion is unique between switches or VSANs in a fabric, but the same for all ports on a particular switch or VSAN. The area and port numbers are generated by the switch for each WWPN that logs into the switch. Each area can have 256 ports. Cisco begins with area 00 and sequentially assigns port numbers.

The first problem occurs with particular Qlogic adapters if the area portion is the same for the HBA and a zoned target. In this case, the adapter will not recognize the target. Cisco rectifies this issue by assigning a unique area for all Qlogic adapters that exhibit this issue. In our case, the WWPN begins with 21:00:00:1b:32… Each manufacturer device that is identified in the switches Organizationally Unique Identifier (OUI) table will be allocated a unique area for each WWPN. Each of the affected Qlogic adapters receives a unique area and the port number 0x00. So the area will never be the same as the target, but the port number portion will always be 0x00, which leads to the second issue.

First, the 4Gb Brocade switches in Access Gateway mode will not route to a WWPN with the port portion of the FCID being 0x00. Second it will not route to two or more ports with the same port number on the same switch. So Cisco's solution to the first issue is not compatible with routing on these switches. Brocade outlines 4 work arounds for the combined issue. Each recommendation depends on your configuration. The first two options are only available if you don't and will not have target devices connected to the Cisco MDS switch in question. If this is the case, you can undo Cisco's fix by deleting specific entries from the OUI table or putting the switch in flat-fcid allocation mode, which essentially ignores the table. It is probably best to avoid these two work a rounds in order to prevent future problems.

The third option is to basically swap the OUI entries of HBAs and targets. In other words, remove the HBA OUI IDs and add the target device OUI IDs. The HBAs will get sequential port numbers in the same area and the target devices will get unique area IDs resolving both issues. The overall problem with this solution is that the devices need to be connected after the OUI table manipulation. Devices connected before the table update will already have FCIDs assigned. If persistent FCID allocation is enabled, the FCIDs will not change without disconnecting the WWPN and manually deleting them. Otherwise, the VSAN needs to be restarted.

The last option is the option that we chose to implement. Basically, we manually assigned FCIDs to each of the affected HBAs connected via the Brocade switches in Access Gateway mode. We kept track of manually assigned FCIDs in a spreadsheet to ensure that no port numbers were duplicated on a switch. Actually, the switch will not allow you to assign duplicate FCIDs. An FCID is allocated the first time the HBA logs into the switch. The static FCID assignment can be performed in advance if the WWPNs for each HBA are known prior to connection. Otherwise, the host needs to be shutdown so that the FCID can be deleted and the static assignment performed. In my case, after the hosts were shutdown, I had to shut/no shut the interface to release the FCID. Prior, we did verify multipathing on other hosts accessing storage via the same FC port.

In the example below, first I list the free and allocated FCIDs in the VSAN in order to choose a range to assign. Next I check the FCIDs in use on the port that my Qlogic HBA is connected. If they are all the affected Qlogic HBAs, I can tell if they are in use by whether the FCID has been manually assigned. Otherwise, I need to consult the active zoneset. If another server is actively using SAN storage, I check the multipath software on that server. Now, I check to see if the initiator is logged into the switch. In my example, it is not logged in. Next, I proceed to shut the port as if the FCID was in use. Finally, I can delete the existing FCID and add a new one. Afterward, I bring the port back up, verify the new FCID and save the configuration. At this point, I am free to move onto creating aliases and zoning.

sh fcdomain address-allocation vsan 5

Free FCIDs: 0xc90028 to 0xc9002a

0xc90066 to 0xc900e3

0xc900e5 to 0xc900ee

0xc900f0 to 0xc900ff

0xc90200 to 0xc902e3

0xc902e5 to 0xc902ee

0xc902f0 to 0xc903ee

0xc903f0 to 0xc904ee

0xc904f0 to 0xc905ee

0xc905f0 to 0xc905ff

0xc90e00 to 0xc90e01

0xc90e03 to 0xc90eee

0xc90ef0 to 0xc90eff

0xc91400 to 0xc914ee

0xc914f0 to 0xc915ee

0xc915f0 to 0xc916ee

0xc916f0 to 0xc919ff

0xc91d00 to 0xc91dee

0xc91df0 to 0xc91eee

0xc91ef0 to 0xc92000

0xc92002

0xc9201c to 0xc920ff

0xc92800 to 0xc929ff

0xc92b00 to 0xc92bff

0xc92d00 to 0xc92eff

0xc93000 to 0xc930ff

0xc93a00 to 0xc93aff

0xc93c00 to 0xc93cff

0xc93f00 to 0xc93fff

0xc94400 to 0xc944ff

0xc94b00 to 0xc9fffe

Assigned FCIDs: 0xc90000 to 0xc90027

0xc9002b to 0xc90065

0xc900e4

0xc900ef

0xc90100 to 0xc901ff

0xc902e4

0xc902ef

0xc903ef

0xc904ef

0xc905ef

0xc90600 to 0xc90dff

0xc90e02

0xc90eef

0xc90f00 to 0xc913ff

0xc914ef

0xc915ef

0xc916ef

0xc91a00 to 0xc91cff

0xc91def

0xc91eef

0xc92001

0xc92003 to 0xc9201b

0xc92100 to 0xc927ff

0xc92a00 to 0xc92aff

0xc92c00 to 0xc92cff

0xc92f00 to 0xc92fff

0xc93100 to 0xc939ff

0xc93b00 to 0xc93bff

0xc93d00 to 0xc93eff

0xc94000 to 0xc943ff

0xc94500 to 0xc94aff

0xc9ffff

Reserved FCIDs: 0xc9ffff

Number of free FCIDs: 52852

Number of assigned FCIDs: 12684

Number of reserved FCIDs: 1

MDS# show flogi database interface fc1/3

--------------------------------------------------------------------------------

INTERFACE VSAN FCID PORT NAME NODE NAME

--------------------------------------------------------------------------------

fc1/3 5 0xc90100 21:00:00:1b:32:06:f8:e7 20:00:00:1b:32:06:f8:e7

fc1/3 5 0xc92c00 21:00:00:1b:32:94:99:da 20:00:00:1b:32:94:99:da

Total number of flogi = 2.

MDS# show fcdomain fcid persistent unused vsan 5

Total entries 1.

Persistent FCIDs table contents:

VSAN WWN FCID Mask Used Assignment

---- ----------------------- -------- ----------- ---- ----------

5 21:00:00:1b:32:06:f8:e7 0xc90100 ENTIRE AREA NO DYNAMIC

MDS# conf t

MDS(config)# interface fc1/3

MDS(config-if)# shut

MDS
(config-fcid-db)# fcdomain fcid database

MDS
(config-fcid-db)# no vsan 5 wwn 21:00:00:1b:32:06:f8:e7 fcid 0xc90100

MDS
(config-fcid-db)# vsan 5 wwn 21:00:00:1b:32:06:f8:e7 fcid 0xc92002

MDS
(config-fcid-db)# interface fc1/3

MDS(config-if)# no shut

MDS(config-if)# exit

MDS(config)# exit

MDS# show fcdomain fcid persistent vsan 5

Total entries 188.

Persistent FCIDs table contents:

VSAN WWN FCID Mask Used Assignment

---- ----------------------- -------- ----------- ---- ----------

…

5 21:00:00:1b:32:06:f8:e7 0xc92002 SINGLE FCID YES STATIC

…

MDS# show flogi database interface fc1/3

--------------------------------------------------------------------------------

INTERFACE VSAN FCID PORT NAME NODE NAME

--------------------------------------------------------------------------------

fc1/3 5 0xc92002 21:00:00:1b:32:06:f8:e7 20:00:00:1b:32:06:f8:e7

fc1/3 5 0xc92c00 21:00:00:1b:32:94:99:da 20:00:00:1b:32:94:99:da

Total number of flogi = 2.

MDS# copy running-config startup-config

Recovering an NTFS Boot Sector in Symantec Storage Foundation for Windows

2011-09-19T10:45:00.000-07:00

One evening, I received a call from a customer who ran into an issue with Storage Foundation for Windows. They ran into a SFW bug while trying to shrink volumes for a disk space recovery project. FYI, unlike UNIX, SF volumes in Windows need to be offline before shrinking the volume. Long story short, the shrink process ended up corrupting the NTFS boot record on a 1.3TB volume. Even though the vxprint output showed the volumes ENABLED and ACTIVE with the correct volume boundaries, the volume showed up RAW, not NTFS from both VEA console and DISKPART.

From the UNIX perspective, this would have been no big deal. Running a vxmksdpart would recreate the partition table. This process is not as easy with Windows. You need to locate the backup NTFS boot sector which is located at the end of the volume and copy it over the corrupted one. You then run an fsck of the file system (which can take FOR-E-VER!!), cross your fingers and pray. I emphasize that this is a LAST RESORT measure before recovering from backups. There is NO, I repeat, NO GUARANTEE that you will be free from data corruption. But, if you’re going to have to rebuild the file system and recover the data from tape/disk, you might as well give it a shot. As in my customer’s case, they were fortunate enough to recover the volume on the first try. When this issue appeared again, they ended up losing ACL’s to all files in the volume. However, recovery of the ACLs was still much faster recovering all the data!

Here’s the detailed steps on the process of recovering the NTFS boot sector. I used the vxtool from Symantec to locate the backup boot sector and to perform the recovery. This tool can be downloaded from ftp://ftp.veritas.com/pub/support/vmtools.

1) Determine the location of the backup NTFS boot sector.

a) Make a note of the harddisk number that contains the RAW volume.

b) In VEA, select the volume with the corrupt NTFS boot sector. In the right panel, select the Subdisks, tab. Highlight the affected subdisk, right-click and select “Properties”. Note the size of the volume in sectors and disk offset.

c) Jot down the Size and Disk offset.

d) Add the size in sectors to the disk offset in sectors minus 1. This is the location of the backup boot sector.

Size=409600 Disk offset=65570

Backup boot sector = 409600 + 65570 – 1 = 475169

2) From a command window, use the vxtool utility to scan the range near the end of the problem volume.

Hard disk size (sectors) = 10485760

Command: vxtool disk sectorscan harddisk<#>

From a command window:

vxtool disk sectorscan harddisk1 0 10485760

Output:

The Disk Offset number (65570) is the location of the original NTFS boot sector. The backup NTFS boot sector is the number calculated from Step 1d). Verify that this block is noted as “magic NTFS”.

3) Now, to restore the corrupted NTFS boot sector using the backup copy using the vxtool utility. The source and target disks may be different if the volume spans multiple disks.

Command:

vxtool disk copyblock sourcedisk offset targetdisk offset block

Example:

vxtool disk copyblock harddisk1 65570 harddisk1 475169 1

4) Perform a rescan from VEA and DISKPART.

The volume should show up as an NTFS volume. Cross your fingers and run a chkdsk to ensure file system data integrity.

Here are some reference articles from Symantec that addresses the data corruption issues with volume shrinks. A patch has now been provided as of 06/01/2011.

Tech Alert

http://www.symantec.com/docs/AL1130

Tech Article
http://www.symantec.com/docs/TECH160393

Patch
https://sort.symantec.com/patch/detail/5022

Integrating DB2 Advanced Copy Services with NetApp Snapshots

2011-09-12T09:00:00.000-07:00

IBM provides integration with NetApp snapshots via their Advanced Copy Services (ACS). ACS is available for DB2 on release 9.5 and newer. ACS supports NAS or SAN attached NetApp subsystems when running under AIX. Only NAS attachment is supported under Linux. ACS allows the DBA to issue backup & restore directives from the DB2 environment without involving the storage administrator. These backup and restore directives manipulate snaphots for the appropriate volumes on the NetApp filer. One caveat to using ACS is the 2 snapshot (backup) limit if you do not own a full TSM license. A full TSM license can be very pricy. Due to the 2 snapshot limit, I decided to use a hybrid approach where ACS provides two daily backups for quick recovery in conjunction with the traditional file based backup for longer retention periods.

Setup of ACS is fairly easy in either a Linux or AIX environment. ACS uses RSH to communicate with the NetApp filer, so the RSH service must be enabled on the filer (options rsh.enable on). The database server can be included in the /etc/hosts.equiv file on the filer so that a password is not needed. ACS is intelligent enough to correlate the database and log file systems/volume groups on the server to the corresponding volumes/LUNs on the filer.

During testing under AIX, I found an issue with the setup.sh script where the acscim program caused the script to fail. The acscim module is used to communicate to IBM storage such as the DS series and requires supporting libraries that were not available on my system. I commented out the acscim check and the setup script completed normally. The setup script needs to be executed as root, but from the database instance user's home directory. The database instance user in the example below is db2int2.

ACS is installed under the database instance user's home directory. Go into the ACS directory and edit the setup.sh script to comment out the acscim binary check.

bash-3.00# cd /home/db2int2/sqllib/acs

bash-3.00# grep cim setup.sh

checkbin ${INST_DIR}/acs/acscim

enableSUID ${INST_DIR}/acs/acscim

bash-3.00# vi setup.sh

bash-3.00# grep cim setup.sh

# checkbin ${INST_DIR}/acs/acscim

enableSUID ${INST_DIR}/acs/acscim

Execute the setup.sh script to provide the necessary parameters to configure ACS. I chose defaults for most of the questions. The ACS_REPOSITORY needs to be set to desired directory path which will be created by the script. The COPYSERVICES_HARDWARE_TYPE is either NAS_NSERIES or SAN_NSERIES under AIX or NAS_NSERIES for Linux.

bash-3.00# pwd

/home/db2int2/sqllib/acs

bash-3.00# ./setup.sh

checking /home/db2int2/sqllib/acs/acsnnas ...

OK

checking /home/db2int2/sqllib/acs/acsnsan ...

OK

Do you have a full TSM license to enable all features of TSM for ACS ?[y/n]

n

****** Profile parameters for section GLOBAL: ******

ACS_DIR [/home/db2int2/sqllib/acs ]

ACSD [57329 ] 57328

TRACE [NO ]

****** Profile parameters for section ACSD: ******

ACS_REPOSITORY [/home/db2int2/sqllib/acs/acsrepository ]

****** Profile parameters for section CLIENT: ******

TSM_BACKUP [NO ]

MAX_VERSIONS [2 ]

LVM_FREEZE_THAW [YES ]

DEVICE_CLASS [STANDARD ]

****** Profile parameters for section STANDARD: ******

COPYSERVICES_HARDWARE_TYPE [SAN_NSERIES]

COPYSERVICES_PRIMARY_SERVERNAME [netappcntl1 ]

COPYSERVICES_USERNAME [root ]

======================================================================

The profile has beeen successfully created.

Do you want to continue by specifying passwords for the defined devices? [y/n]

y

Please specify the passwords for the following profile sections:

STANDARD

master

Creating password file at /home/db2int2/sqllib/acs/shared/pwd.acsd.

A copy of this file needs to be available to all components that connect to acsd.

BKI1555I: Profile successfully created. Performing additional checks. Make sure to restart all ACS components to reload the profile.

After setup is complete, check to see if the daemons are configured to start in /etc/inittab. Note: acsnnas is for NetApp NAS volumes and acsnsan is for NetApp SAN volumes.

bash-3.00# grep acs /etc/inittab

ac00:2345:respawn /home/db2int2sqllib/acs/acsd

ac00:2345:respawn /home/db2int2sqllib/acs/acsnnas –D

OR

ac00:2345:respawn /home/db2int2sqllib/acs/acsnsan –D

Check to see if the daemons are running:

bash-3.00# ps -ef | grep acs

root 12255442 6225980 0 16:25:07 pts/2 0:00 grep acs

db2int2 12451872 1 0 16:24:50 - 0:00 /home/db2int2/sqllib/acs/acsd

db2int2 12451873 1 0 16:26:35 - 0:00 /home/db2int2/sqllib/acs/acsnsan -D

bash-3.00#

Now that ACS is configured, we can perform snapshot backups and restores. As the database instance user execute the following commands to take backups, list backups or restore the database.

Execute the following to take an offline backup:

bash-3.00$ db2 backup db mydb use snapshot

You can specify the "online" parameter to take an online backup of the database:

bash-3.00$ db2 backup db mydb online use snapshot

To list the backups of the database as follows:

bash-3.00$ db2acsutil query

To restore the latest backup:

bash-3.00$ db2 restore db mydb use snapshot

"Unfortunately this has not been documented very well." Fun with VERITAS Cluster Server

2011-09-05T09:00:00.000-07:00

So Justin and I are wrapping up a large refresh project for a client where we're moving them from an existing configuration running Oracle on Sun 6800s with VCS over to new M5000s. As you'd expect this includes a migration to Solaris 10 as well as upgrades to VERITAS Volume Manager, File System, and NetBackup (that's Symantec Storage Foundation and NetBackup to some of you).

The application team went through their testing over the last month or so and we completed our VCS test matrix in preparation for cutover. During the cutover, though, we noticed the following message in the alert log:

WARNING:Oracle instance running on a system with low open file descriptor
limit. Tune your system to increase this limit to avoid
severe performance degradation.

Thinking that we'd missed a resource control setting somewhere we went through the process of validating those settings. Then, seeing that they looked correct, we asked the DBA to stop and restart the database manually only to find that the error message above didn't appear. Using VCS to stop and start the database would generate this error every time, though.

We opened a case with Symantec and started to troubleshoot. Thankfully we found that in VCS 5.1 SP1 Symantec added a file called vcsenv that hardcodes limits for CPU time, core file size, data segment size, file size, and the number of open file descriptors before we ran out of window for the cutover.

The location and contents of the file are shown below, including where we set the number of file descriptors to 8192.

bash-3.00# cd /opt/VRTSvcs/bin

bash-3.00# more vcsenv

# $Id: vcsenv,v 2.8 2010/09/30 05:45:29 ptyagi Exp $ #

# THIS SOFTWARE CONTAINS CONFIDENTIAL INFORMATION AND TRADE SECRETS OF

# SYMANTEC CORPORATION. USE, DISCLOSURE OR REPRODUCTION IS PROHIBITED

# WITHOUT THE PRIOR EXPRESS WRITTEN PERMISSION OF SYMANTEC CORPORATION.

# The Licensed Software and Documentation are deemed to be commercial

# computer software as defined in FAR 12.212 and subject to restricted

# rights as defined in FAR Section 52.227-19 "Commercial Computer

# Software - Restricted Rights" and DFARS 227.7202, "Rights in

# Commercial Computer Software or Commercial Computer Software

# Documentation", as applicable, and any successor regulations. Any use,

# modification, reproduction release, performance, display or disclosure

# of the Licensed Software and Documentation by the U.S. Government

# shall be solely in accordance with the terms of this Agreement. $

# This is just a sample as to how you can specify various environment

# variables you need to set. Uncomment/add/modify the values as per

# your requirement.

# Specify the default language in which you want to bring up the

# VCS agents.

# LANG=C; export LANG

VCSHOME="${VCS_HOME:-/opt/VRTSvcs}"

#This is required for agents which use dynamic VCSAPI libraries

LD_LIBRARY_PATH="$VCSHOME/lib:$LD_LIBRARY_PATH"

export LD_LIBRARY_PATH

# Setting ulimit.

# Common For Linux, HP-UX, SunOS & AIX

ulimit -t unlimited # CPU Time

ulimit -c unlimited # Core File Size

ulimit -d unlimited # Data Seg Size

ulimit -f unlimited # File Size

ulimit -n 8192 # File Descriptor

if [ `uname` = "AIX" ];then

RT_GRQ=ON

export RT_GRQ

Working with NetBackup CTIME Timestamps in Excel

2011-08-29T09:00:00.000-07:00

NetBackup makes extensive use of timestamps for backup job start and stop times, expiration dates, etc. NetBackup appends a timestamp to the client name to create a backup id (myserver_1303936891). Don’t know about you, but I don’t natively understand CTIME timestamps.
To convert between the timestamp and “human readable” time, Symantec provides the bpdbm command with the -ctime flag to convert from NetBackup’s timestamp to the current time zone. For example:

bash-3.00# bpdbm -ctime 1303936891
1303936891 = Wed Apr 27 15:41:31 2011

Some of my favorite NetBackup reports output job start/stop times and other data using CTIME timestamps:

bpmedialist shows the expiration date for tapes
bpimagelist shows the start and stop times for backup images in the catalog, as well as image expiration dates

It’s just not feasible to run bpdbm on every timestamp for output that may contain 10K lines. Excel makes a far better tool.
To convert from CTIME to “Excel” time you need the following information:

The NetBackup timestamp (the number of seconds since 1/1/1970, in GMT)
The number of seconds per day (86,400)
The Excel serial number for 1/1/1970 (25,569)
Your current timezone offset in hours (for US/Central this is currently -5)

Excel date/time stamps are real numbers where the integer portion represents the number of days since 1/1/1900 (where 1/1/1900 is day 1). The decimal portion of an Excel date/time value represents the portion of a single day (ie, 0.5 is 12 hours). Therefore, to convert from CTIME to Excel time use the following formula:

timestamp/86400+25569+(-5/24)

How to Mount Cloned Volume Groups in AIX

2011-08-22T09:00:00.000-07:00

Today, most SAN storage vendors provide some kind of volume or LUN cloning capability. The name and underlying mechanics for each vendor differ, but the end result is pretty much the same. They take a primary volume or LUN and create an exact copy at some point in time. NetApp's name for this technology is FlexClone.

Typically, creating a clone of a LUN and mounting the file system on the original server is a trivial process. The process becomes more complex if volume management is involved. Server based volume management software provides many benefits, but complicates matters where LUN clones are used. In the case of IBM's Logical Volume Management (LVM), mounting clones on the same server results in duplicate volume group information. Luckily, AIX allows LVM to have duplicate physical volume IDs (PVID) for a "short period" of time without crashing the system. Not sure exactly what a "short period" of time equates too, but in my testing I didn't experience a crash.

The process to "import" a cloned volume group for the first time is disruptive in that the original volume group must be exported. It is necessary to have the original volume group exported so that the physical volume IDs (PVIDs) on the cloned LUNs can be regenerated. The recreatevg command is used to generate new PVIDs and to rename the volume names in the cloned volume group. Note that the /etc/filesystem entries need to be manually updated because the recreatevg command prepends /fs to the original mount point names for the clones. Once the /etc/filesystem file is updated, the original volume group can be re-imported with importvg.

Subsequent refreshes of previously imported clones can be accomplished without exporting the original because ODM remembers the previous PVID to hdisk# association. It does not reread the actual PVID from the disk until an operation is performed against the volume group. The recreatevg command will change the PVIDs and volume names on the cloned volume group without affecting the source volume group.

Process for initial import of cloned volume group:

Clone the LUNs comprising the volume group
1. Make sure to clone in a consistent state
Unmount and export original volume groups
1. Use df to associate file systems to volumes
2. Unmount file systems
3. Use lsvg to list the volume groups
4. Issue varoffvg to each affected volume group
5. Use lspv to view the PVIDs for each disk associated with the volume groups
6. Remember the volume group names and which disks belong to each VG that will be exported
7. Use varyoffvg to offline each VG
8. Use exportvg to export the VGs
Bring in the new VG
1. Execute cfgmgr to discover new disks
2. Use lspv to identify the duplicate PVIDs
3. Execute recreatevg on each new VG listing all disks associated with the volume group and –y option to name the VG
4. Use lspv to verify no duplicate PVIDs
Import the original volume groups
1. Execute importvg with the name of one member hdisk and the –y option with the original name
2. Mount the original file systems.
Mount the cloned file systems
1. Make mount point directories for the cloned file systems
2. Edit /etc/filesystems to update the mount points for the cloned VG file systems
3. Use mount command to mount the cloned file systems

The subsequent import of a cloned volume group differs in that only the cloned volume group needs to be unmounted and varied offline prior to the clone refresh. Remember the hdisk numbers involved in each clone volume group that is to be refreshed. Once refreshed use exportvg to export the volume group. Afterward, the recreatevg command is issued naming each hdisk associated with the volume group and its previous name. Now the volumes and file systems are available. Prior to mounting, the /etc/filesystem entries need to be updated to correct the mount points.

Process to refresh cloned volume group:

Unmount and vary off the cloned volume groups to be refreshed
1. Execute umount on associated file systems
2. Use varyoffvg to offline each target VG
Refresh the clones on the storage system
Bring in the refreshed clone VGs
1. Execute cfgmgr
  1. Use lspv and notice that ODM remembers the hdisk/PVID and volume group associations
2. Use exportvg to export the VGs noting the hdisk numbers for each VG
3. Execute recreatevg on each refreshed VG naming all disks associated with the volume group and –y option to name the VG to its original name
4. Now lspv displays new unique PVIDs for each hdisk
Mounting the refreshed clone file systems
1. Edit /etc/filesystem to correct the mount points for each volume
2. Issue mount command to mount the refreshed clones

See the example below for a first time import of two cloned volume groups, logvg2 and datavg2, consisting of 2 and 4 disks respectively:

bash-3.00# df

Filesystem 512-blocks Free %Used Iused %Iused Mounted on

/dev/hd4 1048576 594456 44% 13034 17% /

/dev/hd2 20971520 5376744 75% 49070 8% /usr

/dev/hd9var 2097152 689152 68% 11373 13% /var

/dev/hd3 2097152 1919664 9% 455 1% /tmp

/dev/hd1 1048576 42032 96% 631 12% /home

/dev/hd11admin 524288 523488 1% 5 1% /admin

/proc - - - - - /proc

/dev/hd10opt 4194304 3453936 18% 9152 3% /opt

/dev/livedump 524288 523552 1% 4 1% /var/adm/ras/livedump

/dev/pocdbbacklv 626524160 578596720 8% 8 1% /proddbback

/dev/fspoclv 1254359040 1033501496 18% 2064 1% /cl3data

/dev/fspocdbloglv 206438400 193491536 7% 110 1% /cl3logs

/dev/poclv 1254359040 1033501480 18% 2064 1% /proddb

/dev/pocdbloglv 206438400 193158824 7% 115 1% /proddblog

/dev/datalv2 836239360 615477152 27% 2064 1% /datatest2

/dev/loglv2 208404480 195088848 7% 118 1% /logtest2

bash-3.00#

bash-3.00$ umount /datatest2/

bash-3.00# umount /logtest2/

bash-3.00# lsvg

rootvg

pocdbbackvg

dataclvg

logsclvg

pocvg

pocdblogvg

datavg2

logvg2

bash-3.00# varyoffvg datavg2

NOTE: remember the hdisk and vg names for the exported vg's.

bash-3.00# lspv

hdisk0 00f62aa942cec382 rootvg active

hdisk1 none None

hdisk2 00f62aa997091888 pocvg active

hdisk3 00f62aa9a608de30 dataclvg active

hdisk4 00f62aa9a60970fc logsclvg active

hdisk10 00f62aa9972063c0 pocdblogvg active

hdisk11 00f62aa997435bfa pocdbbackvg active

hdisk5 00f62aa9a6798a0c datavg2

hdisk6 00f62aa9a6798acf datavg2

hdisk7 00f62aa9a6798b86 datavg2

hdisk8 00f62aa9a6798c36 datavg2

hdisk9 00f62aa9a67d6c9c logvg2 active

hdisk12 00f62aa9a67d6d51 logvg2 active

bash-3.00# varyoffvg logvg2

bash-3.00# lsvg

rootvg

pocdbbackvg

dataclvg

logsclvg

pocvg

pocdblogvg

datavg2

logvg2

bash-3.00# exportvg datavg2

bash-3.00# exportvg logvg2

bash-3.00#

bash-3.00# exportvg datavg2

bash-3.00# exportvg logvg2

bash-3.00# cfgmgr

bash-3.00# lspv

hdisk0 00f62aa942cec382 rootvg active

hdisk1 none None

hdisk2 00f62aa997091888 pocvg active

hdisk3 00f62aa9a608de30 dataclvg active

hdisk4 00f62aa9a60970fc logsclvg active

hdisk10 00f62aa9972063c0 pocdblogvg active

hdisk11 00f62aa997435bfa pocdbbackvg active

hdisk5 00f62aa9a6798a0c None

hdisk6 00f62aa9a6798acf None

hdisk7 00f62aa9a6798b86 None

hdisk8 00f62aa9a6798c36 None

hdisk13 00f62aa9a6798a0c None

hdisk14 00f62aa9a6798acf None

hdisk15 00f62aa9a6798b86 None

hdisk9 00f62aa9a67d6c9c None

hdisk12 00f62aa9a67d6d51 None

hdisk16 00f62aa9a6798c36 None

hdisk17 00f62aa9a67d6c9c None

hdisk18 00f62aa9a67d6d51 None

bash-3.00#

Notice the duplicate PVIDs. Use the recreatevg command naming all of the new disks in each volume group of the newly mapped clones.

bash-3.00# recreatevg -y dataclvg2 hdisk13 hdisk14 hdisk15 hdisk16

dataclvg2

bash-3.00# recreatevg -y logclvg2 hdisk17 hdisk18

logclvg2

bash-3.00# importvg -y datavg2 hdisk5

datavg2

bash-3.00# importvg -y logvg2 hdisk9

logvg2

bash-3.00# lspv

hdisk0 00f62aa942cec382 rootvg active

hdisk1 none None

hdisk2 00f62aa997091888 pocvg active

hdisk3 00f62aa9a608de30 dataclvg active

hdisk4 00f62aa9a60970fc logsclvg active

hdisk10 00f62aa9972063c0 pocdblogvg active

hdisk11 00f62aa997435bfa pocdbbackvg active

hdisk5 00f62aa9a6798a0c datavg2 active

hdisk6 00f62aa9a6798acf datavg2 active

hdisk7 00f62aa9a6798b86 datavg2 active

hdisk8 00f62aa9a6798c36 datavg2 active

hdisk13 00f62aa9c63a5ec2 dataclvg2 active

hdisk14 00f62aa9c63a5f9b dataclvg2 active

hdisk15 00f62aa9c63a6070 dataclvg2 active

hdisk9 00f62aa9a67d6c9c logvg2 active

hdisk12 00f62aa9a67d6d51 logvg2 active

hdisk16 00f62aa9c63a6150 dataclvg2 active

hdisk17 00f62aa9c63bf6b2 logclvg2 active

hdisk18 00f62aa9c63bf784 logclvg2 active

bash-3.00#

Notice the PVID numbers are all unique now.

remount original file systems

bash-3.00# mount /datatest2

bash-3.00# mount /logtest2

bash-3.00#

create new mount points and edit /etc/filesystems

bash-3.00# mkdir /dataclone1test2

bash-3.00# mkdir /logclone1test2

bash-3.00# cat /etc/filesystems

…

/fs/datatest2:

dev = /dev/fsdatalv2

vfs = jfs2

log = /dev/fsloglv03

mount = true

check = false

options = rw

account = false

/fs/logtest2:

dev = /dev/fsloglv2

vfs = jfs2

log = /dev/fsloglv04

mount = true

check = false

options = rw

account = false

/datatest2:

dev = /dev/datalv2

vfs = jfs2

log = /dev/loglv03

mount = true

check = false

options = rw

account = false

/logtest2:

dev = /dev/loglv2

vfs = jfs2

log = /dev/loglv04

mount = true

check = false

options = rw

account = false

bash-3.00#

Notice the cloned duplicates are prefixed with /fs on the mount point by the recreatevg command. Also the volume names were changed to prevent duplicate entries in /dev. Update /etc/filesysems with the mount points created previously.

bash-3.00# mount /dataclone1test2

Replaying log for /dev/fsdatalv2.

bash-3.00# mount /logclone1test2

Replaying log for /dev/fsloglv2.

bash-3.00# df

Filesystem 512-blocks Free %Used Iused %Iused Mounted on

/dev/hd4 1048576 594248 44% 13064 17% /

/dev/hd2 20971520 5376744 75% 49070 8% /usr

/dev/hd9var 2097152 688232 68% 11373 13% /var

/dev/hd3 2097152 1919664 9% 455 1% /tmp

/dev/hd1 1048576 42032 96% 631 12% /home

/dev/hd11admin 524288 523488 1% 5 1% /admin

/proc - - - - - /proc

/dev/hd10opt 4194304 3453936 18% 9152 3% /opt

/dev/livedump 524288 523552 1% 4 1% /var/adm/ras/livedump

/dev/pocdbbacklv 626524160 578596720 8% 8 1% /proddbback

/dev/fspoclv 1254359040 1033501496 18% 2064 1% /cl3data

/dev/fspocdbloglv 206438400 193491536 7% 110 1% /cl3logs

/dev/poclv 1254359040 1033501480 18% 2064 1% /proddb

/dev/pocdbloglv 206438400 193158824 7% 115 1% /proddblog

/dev/datalv2 836239360 615477152 27% 2064 1% /datatest2

/dev/loglv2 208404480 195088848 7% 118 1% /logtest2

/dev/fsdatalv2 836239360 615477160 27% 2064 1% /dataclone1test2

/dev/fsloglv2 208404480 195744288 7% 114 1% /logclone1test2

bash-3.00#

Virtual Machine Migration Fails

2011-08-15T09:00:00.000-07:00

After recently upgrading a customer to vSphere 4.1 Update 1, I received a call from the customer because one of their guest VMs could not be migrated to a different VMFS datastore. The somewhat troubling, “unable to access file” error message indicated a problem with the VMDK for a specific snapshot.

Through an SSH session I validated the snapshot VMDK and VMDK flat file existed in the proper directory. The VMDK flat file had a non-zero size, suggesting to me this might be a configuration problem.

Out of curiosity I examined the snapshot’s VMDK configuration file and noticed the parentFileNameHint entry contained the full pathname and used the VMFS GUID value. Hmm. Is that the proper GUID? No it wasn’t.

Since the VM had other snapshots I reviewed those configuration files as well and noticed they used relative path names for the parentFileNameHint. Could it be that simple?
I edited the snapshot VMDK configuration file and removed the full path qualification.

Problem solved.

In my example above (which was reproduced in my lab), I changed:
parentFileNameHint=”/vmfs/volumes/4ab3ebbc-46f3c941-7c14-00144fe69d58/retro-000001.vmdk"
To:
parentFileNameHint="retro-000001.vmdk"

FCIP considerations for 10 GigE on Brocade FX8-24

2011-08-08T09:00:00.001-07:00

While working with a client to architect a new FCIP solution, there were a number of considerations that needed to be addressed. With this particular implementation we are leveraging the FX8-24 blades in DCX chassis and attaching the 10Gbe xge links to the network core.

With the current version of FOS (6.4.1a in this case), the 10Gbe interface is maximized by using multiple “circuits” which are combined in a single FCIP tunnel. Each circuit has a maximum bandwidth of 1Gb. To aggregate the multiple circuits you need the Advanced Extension License. Each of these circuits needs an IP address on either end of the tunnel. Additionally, there are two 10Gbe xge ports on each FX8-24 card; and they require placement in separate VLANs. Be sure to discuss and plan these requirements with your network team.

There are other considerations as well, such as utilizing Virtual Fabrics to isolate and allow the FCIP fabrics to merge between switches or utilizing the Integrated Routing feature (additional licensing) to configure the FCIP tunnels with VEX ports without the requirement of merging the fabrics.

Regardless of the architecture (Virtual fabric vs integrated routing) you will need to configure the 1Gbe circuits appropriately. You will want to understand the maximum bandwidth your link can sustain, and configure the FCIP tunnel in such a way that you consume just under the maximum bandwidth to prevent TCP/IP congestion and sliding window ramp up from slowing down your overall throughput.

In our example, we want to consume about 6 Gbe of bandwidth between their locations. We will need to configure six circuits within the FCIP tunnel, each configured just under the 1Gbe maximum bandwidth setting.

Storage Performance Concepts Entry 6

2011-08-01T09:00:00.000-07:00

The Impact of Cache

Although the cache within an array can be used for a number of purposes such as storing configuration information or tables for snapshots and replication its primary purpose is to serve as a high speed buffer area between the hosts and the backend disk. The caching algorithms used by the manufacturers are their “secret sauce” and they aren’t always forthcoming with exactly how they work but at a basic level they all provide the same types of functions.

In our previous entries on storage performance we highlighted the performance differences of RAID 10 versus RAID 5. RAID 10 provides better performance than RAID 5 in particular for random IO because the RAID penalty is lower and given the same amount of usable capacity it will have more underlying RAID Groups and total disk drives.

Despite the performance benefits we deploy RAID 5 much more frequently than RAID 10. The first reason why is obvious, RAID 5 is much less expensive. The second reason is that cache greatly improves performance to the point that RAID 5 is acceptable.

All host write requests are satisfied by cache. Once the IO is stored in cache the array sends an acknowledgement back to the host that initiated the IO to let it know that it was received successfully and that it can now send its next IO request. In the background the array will write or flush the pending write IOs to disk.

How and when these write IOs are flushed depends on the cache management algorithms of the array as well as the current cache workload. Most arrays including the HDS AMS 2000 will attempt to hold off flushing to disk to see if it can perform a full stripe write. This is particularly important in RAID 5 configurations because it means you can eliminate the read and modify operations associated with performing write IO. Details of Read-Modify-Write can be found at the following link.

http://en.wikipedia.org/wiki/Standard_RAID_levels

Although the array is not always able to do full stripe writes at least on the HDS arrays we are often able to use RAID 5 in scenarios where you would typically be looking at RAID 10. This is not to say that RAID 5 is faster than RAID 10 just that with effective cache algorithms RAID 5 may perform better than expected.

It’s interesting that if you google array cache performance you will find a number of articles and blog entries that talk about how cache can negatively impact performance. This is a bit misleading. These entries are focused on very specific use cases and typically involve a single application with a dedicated storage device rather than a shared storage environment like the ones used in most environments. Let’s breakdown how cache is used and the impact associated with various workloads.

How Cache Works

Exactly how cache operates is dependent on the storage array and the particular step by step processes involved are outside the scope of this entry. In any case, what we are interested in is not so much the cache operations of a specific array but rather the general impact of cache in real world workloads. I cannot speak for every array but in the case of HDS enterprise systems their Theory of Operation Guide goes into excruciating detail on cache. If you are looking for this type of detail I suggest you get a copy of this guide or similar information from your array manufacturer.

The performance benefits of cache are based on cache hits. A cache hit occurs when the IO request can be satisfied from cache rather than from disk. The array manufacturers develop algorithms to make the best use of the available cache and to increase the number of cache hits. This is done in a number of ways.

Least Recently Used – Most arrays employ a least recently used algorithm to manage the content of cache. In short the most recent data is left in cache and the least recently used data is removed, freeing up capacity for additional IO.

Locality of Reference – When a read request is issued the data is read from disk and placed into cache. In addition to the specific data requested, additional data said to be nearby is also loaded into cache. This is based on the principal that data exists in clusters and that there is a high degree of likelihood that nearby data will also be accessed.

Prefetching – The array will attempt to recognize sequential read operations and prefetch data into cache.

Applying this to workloads

As we discussed in our previous entry Cache can be thought of as very wide but not very deep, it provides a lot of IOs but doesn’t store a lot of data. This can come into play when you try and determine how to address a performance issues with a storage array. Here is an illustration that should help.

You have an array with 16GB of cache that you are frequently seeing performance issues. You look at performance monitor and determine that the cache has entered write pending mode, meaning that 70% of the user cache has pending write data and the array has begun throttling back the attached hosts.

Should you add more cache? Let’s take a look.

· During the performance degradation the array is receiving 400MBps of write requests

· The array has 16GB of cache and we are considering upgrading it to 32GB.

o 16GB of installed cache results in

§ 8GB Mirrored

§ 70% limit = 5.734GB

o 32GB of installed cache results in

§ 16GB Mirrored

§ 70% Limit = 11.2GB Usable

o Writing data at 400MBps fills up the 16GB configuration in ~15 Seconds

o Writing data at 400MBps fills up the 32GB configuration in ~29 Seconds

Assuming you pay around $1,000 per GB of cache it will cost you $16,000 for the array to slow down to a crawl in 29 seconds rather than 15. This is probably not the solution you or management is hoping for. What may be needed is a combination of approaches.

1. You may still need cache, if the bursts are not sustained the extra cache may be good enough.

2. You may need to partition the cache so that the offending application doesn’t consume all available cache and impact the other hosts sharing the array. The way this works is you dedicate a portion of cache to the LUNs that are receiving the heavy IO, leaving the rest of the cache available for other applications. This will not speed up the application that is performing all of the writes but it will at least keep all of the other applications from suffering.

3. You may need to speed up the disk – the bottom of the funnel in our previous diagram. This can be done by using something like Dynamic Provisioning to distribute the IO across more disks. If Dynamic Provisioning is already in use you may need to add more disk to the pool, each new disk provides additional IOPS.

You may need a combination of all three approaches; more cache, Dynamic Provisioning and more disks. The answer depends on the workload. Cache is an advantage primarily when the IO profile consists of short bursts of random IO. This is most common in OLTP environments. Cache does not help and may even hinder performance in the cases of long sequential IO.

Demonstration of Hitachi Dynamic Tiering

2011-07-25T09:00:00.001-07:00

On the latest model of the Hitachi Data Systems enterprise storage array, the Virtual Storage Platform (VSP), HDS has included the capability to dynamically tier data at a sub-LUN (page) level. Hitachi Data Systems Dynamic Tiering (HDT) is a technology that enables optimization of an HDS Dynamically Provisioned (HDP) Pool by allocating highly referenced pages to higher tiers of storage. HDT will periodically move pages up or down the tiers depending upon access patterns.

For the purpose of this exercise, we will create a multi-tiered HDP pool. Within the HDP pool, there will be two tiers, one SAS and one SATA. From this pool, we present three volumes to a Windows server and populate each of these volumes with static data. We will initially populate all three volumes with static data, filling Tier1 (SAS) first, with the remainder of data spilling over to Tier2 (SATA). We will confirm our Tier1 is filled with static data by viewing the Tier Properties. Next, we will create what we dub, “active data” with IOMeter and generate I/O to test files on each of the three volumes. After several HDT cycles, we will revisit the Tier Properties to observe the affect that HDT has on highly referenced pages.

Storage tiering is not a new concept and there are a variety of ways to tier storage. Prior to HDT, HDS provided the capability to tier at the LUN level with Tiered Storage Manager. Other ways of tiering include file archiving and virtualizing storage. However, HDT can automatically tier data at a finer granularity without the need to set up policies or classify the data.

Hitachi Dynamic Provisioning (HDP) in practice

2011-07-18T09:00:00.000-07:00

We've talked about HDP on the blog a few times before (here and here, for example). And with the advent of the VSP, we've moved into a world where all LUN provisioning from HDS arrays should be done using HDP.

In brief, HDP brings three different things to the table:

Wide striping - data from each LUN in an HDP pool is evenly distributed across the drives in the pool.
Thin provisioning - space is only consumed from an HDP pool when data is written from the host. In addition, through Zero Page Reclaim (ZPR), you can recover unused capacity.
Faster allocation - In a non-HDP environment there were two options. You could either have predetermined LUN sizes and format the array ahead of time, or you could create custom LUNs on-demand and wait for the format. With HDP you are able to create custom-sized LUNs and begin using them immediately.

Most of our customers move to HDP as part of an array refresh. Whether it's going from an AMS 1000 to an AMS 2500 or a USP-V to a VSP, they get the benefits of both HDP and newer technology. While this is great from an overall performance perspective it makes it difficult to quantify how much of the performance gain is from HDP vs. how much is from using newer hardware.

We do have one customer with a USP-VM that moved from non-HDP over to HDP, though, and I thought it was worth sharing a couple of performance metrics both pre- and post-HDP. Full disclosure - the HDP pool does have more disks than the traditional layout did (80 vs. 64), and we added 8 GB of data cache as well. So, it's not apples-to-apples, but is as close as I've been able to get.

First we have Parity Group utilization:

As you can see, back-end utilization completely changed on 12/26 when we did the cut over. Prior to the move to HDP parity group utilization was uneven, with groups 1-1 and 1-3 being especially busy. After the move utilization across the groups is even and the average utilization is greatly reduced.

Second we have Write Pending - this metric represents data in cache that needs to be written to disk:

Here you see results similar to the parity group utilization. From cutover on 12/26 until 1/9 write pending is basically negligible. From 1/9 to 1/16 there was monthly processing, corresponding to the peak from 12/12 to 12/19 in the previous month, but as you can see write pending is greatly reduced.

The peak in write pending between 12/19 and 12/26 is due to the migration from non-HDP volumes to HDP volumes. In this case we were also changing LUN sizes, and used VERITAS Volume Manager to perform that piece of the migration.

The difference pre- and post-HDP is compelling, especially when you consider that it's the same workload against the same array. If you're on an array that doesn't support wide striping, or if you're just not using it today then there's an opportunity to "do more with less."

ESX Site Recovery Manager with NetApp Storage

2011-07-11T09:00:00.000-07:00

I'm happy to report that configuring SRM using NetApp storage and SnapMirror is a relatively straightforward operation. That is to say, not any surprises and things pretty much work like you'd expect. The nifty thing about NetApp is that it doesn't require identical arrays at each site; in fact, you can have small regional VM farms (running on something like a small workgroup FAS2040) and SRM those back to your core datacenter running a big-dog FAS6280. I don't have that kind of horsepower to play with in the lab, but down below I'll show you how I protected a FAS270 from yester-year up to a larger 3020 array. And for fun, I did it across a T-1 line. Your mileage may vary, and your co-workers will likely be peeved when you hog up all the bandwidth during that initial sync (I know mine were). Incremental sync jobs afterwards didn't produce hardly any complaints, by the way.The video doesn't detail the initial software installation. Suffice to say, you'll need the SRM installalable from VMware and the NetApp SRA, which is helpfully called the NetApp Disaster Recovery Adapter (NOW login required to download) if you're searching for it. Both should be installed on a dedicated SRM systems, one at the primary site and one at the recovery site.
Other things you will need:

A NetApp head at each site, running at least OnTAP 7.2.4

A SnapMirror license installed at each site

A SnapMirror relationship defined and established for your primary datastore

A FlexClone license (required only to enable the test failover function, as demonstrated in the video)

There's a couple of 'gotchas' when planning this configuration too, at least with 1.4 version of the SRA:

The datastore FlexVols can only have a single SnapMirror relationship, which is to the secondary location. No daisy-chains. This also limits the ability to have multiple recovery sites for a single primary site.

Replication should be done with plain-old Volume SnapMirror. (Qtree-SnapMirror might work and isn't explicitly unsupported, but would be an unwise plan).

SyncMirror however is explicitly unsupported in conjunction with SRM. That should present less of an issue. If you're lucky enough to have SyncMirror, your single ESX cluster should probably span the two sites. So no SRM required. You can still run regular SnapMirror along with SyncMirror to get the VMs off to a third more distant location.

Storage Performance Concepts Entry 5

2011-07-04T09:00:00.000-07:00

Cache Architecture - Part 1

The last area we want to cover is cache. Cache or Cache Memory is just that – memory / DIMMs installed in an array to serve as a high speed buffer between the disks and the hosts. Most storage arrays are what are referred to as cache centric, meaning that all reads and writes are done through cache not directly to disk. In addition to user data / host IO, cache can be used to store configuration information and tables for snapshots, replication or other advanced features that need a high speed storage location. The data in cache must be protected and this is most commonly done with mirroring. In some cases all of the data is mirrored, in others the write IOs are mirrored while reads are not, since the read IOs already exist on the disk.

A common question is “in an array with 16GB of cache how much is really available for user / host IO?”

The exact details depend on the array and the configuration you have but the following concepts should be fairly constant. For example I am using an HDS AMS 2000 Series array.

· 16GB of Cache (8GB per controller)

· A percentage of cache is dedicated to the system area. This varies depending on the hardware configuration and whether or not features that use cache such as replication or Copy on Write Snapshot are enabled. Assuming that replication and Copy on write are not in use 3,370MB total or 1,452MB per controller will be dedicated to the system area leaving 13,480MB or 6,740MB per controller.

· Next each controller mirrors its cache to its partner, 13,480MB becomes 6,740MB or 3,370MB per controller.

· The last calculation depends on the type of IO. All arrays deploy some mechanism to keep the cache from being overrun with write IO requests. How this works is, a threshold is set that when met tells the array to begin throttling back incoming host write requests. In the case of the AMS 2000 series that threshold is 70%. Note that this is for write IO not reads. In a worst case scenario when performing 100% writes the available cache is limited to 70% of the 6,740MB number - 4,718MB total or 2,359MB per controller.

Looking at these numbers many are initially surprised by how little cache is actually available for user IO. It’s interesting to note that we rarely have cache related performance issues with these arrays. The reason has to do with the way that cache operates in a modern storage array. The following diagram created by our CTO and illustrates the relationship between cache and the physical disk drives.

Cache to Disk Relationship

The cache is wide, it provides a lot of IOPS but it is shallow – there isn’t much capacity. The disks are deep, they hold a lot of capacity but individually they aren’t particularly fast. What seems to be impacting the relationship most significantly is wide striping. Wide Striping allows you to pool multiple array groups and more effectively distribute IO across more disks. The result is that writes in cache are flushed to disk more quickly, keeping cache available for incoming IOs. Referring back to our funnel diagram we are essentially widening the bottom of the funnel. Wide striping is not unique to HDS, it is a common feature on many arrays and it just one example of how storage vendors attempt to balance an array. In our next entry we will take a look at the role of cache with various workloads.

Performing a CommVault CommServe Recovery

2011-06-27T09:00:00.000-07:00

Prior to the release of Simpana 9 SP1, upgrades from previous versions could get a little hairy. Sure, CommVault had its upgrade tool (on Maintenance Advantage) to verify that the CommServe could be upgraded. However, this really didn’t cover the critical portion of the upgrade, the MS SQL 2005 to 2008 upgrade. I’ve always recommended testing the upgrade in a lab environment first…just in case! This is where the DR process comes in handy.

For the purpose of this exercise, I will be recovering a version 7 CommServe. After installing the OS and the Simpana 7 CommServe on the VM, a copy of the DR backup set will be loaded into the test CommServe. I have outlined the instructions below.

Perform the following on your Production CommServe:

1) From the CommCell GUI, veriify that the CommServe has the latest updates installed.

2) Take a manual DR backup. Right click on the CommServe, and select “Run DR Backup”

3) Determine the location of the DR backups. In the control panel, double-click on DR Backups.

The DR backup location is located under the “Export Settings Tab”.

4) In the DR backup directory, you will see the DR backups (which include SQL dump, registry settings, etc.) The SET_### consist of a single DR backup. Make a copy of the latest SET_### directory and copy it to the test CommServe VM.

Perform the following on your Test CommServe:

1) Install the CommServe software. A 30 day temporary license will be granted with plenty of time to complete testing. I would highly recommend purchasing the CommServe DR License. This license allows your CommServe two different IP addresses, one for the production site and one for a DR CommServe. The price for this license is miniscule compared to the benefits you gain by having an extra DR CommServe.

2) Verify the location of the DR backup copied from the production CommServe

3) Run the CommServe DR Recovery GUI

\\software_install_directory\base\CommServeDisasterRecoveryGUI.exe

Enter the following parameters in the DR Recovery GUI

SQL Restore Tab

a) Restore file: Select the commserv__FULL.dmp file to restore.

b) Restore path: Verify that the directory where you wish to restore the SQL DB’s.In my case, I installed the default CommServe DB on the C: drive, therefore, I can restore to the original path.

Disregard the following error message. You will be overwriting the existing default CommServe database.

Please note, CommVault Services will need to be restarted.

c) (Optional) Enter valid Mail Server/Recipient information to receive mail notification

d) Click "OK" and verify that the recovery is complete.

Name/License Change Tab

a) Check the CommServe Name Change to the hostname of your Test server. The CS DR GUI will automatically populate this information.

b) Click “OK”. The message window at the bottom will state the Hostname change is complete. Also, notice that the “Commserve Name Change” is no longer checked.

Post Recovery Tab

This is an optional piece of the DR recovery process. If your CommServe was also your Media Agent, you may want to disable schedules and media agent processes. I typically select the following items

a) Check Perform PostRecovery Operations

b) Check “Delete all Active Jobs”

c) Check “Disable Media Agent”

d) Check “Disable schedules"

e) Click "OK"

4) Verify that all CommVault Services have started.

5) Run the CommCell Console. Keep in mind that the cvadmin password will be the password you used when installing the CommServe. If you have AD authentication turned on, then you should be able to use these credentials as well.

Now you can test your CommServe upgrade without impacting production!!

Please note that this process does not take into account any media agent or other iDA installations that may have been installed on the production CommServe.

Storage Tiering Considerations

2011-06-08T09:54:00.000-07:00

The concept of storage tiering is relatively straightforward, rather than deploying one class of storage you implement multiple classes so that the cost of the storage can be aligned with the performance, availability and functional requirements of the applications and services it will support.

An example might be to use high capacity SATA for backups, high speed SAS in a RAID 10 configuration for databases and RAID 5 for application servers. This would be considered basic storage tiering and most organizations have already embraced this strategy. Although this alone certainly offered some cost savings it did have some limitations.

When you architect a solution like this you are often making a bet on what the requirements for a given application will be. Availability and functional requirements usually remain constant but performance requirements often change. You may initially deploy an application to RAID 5 but later determine that it does not provide the necessary performance and that RAID 10 would have been more appropriate. The reverse is also true, the initial application requirements may indicate that RAID 10 is necessary but upon implementation it is discovered that the performance demands were not nearly as high as anticipated and that RAID 5 would have worked fine. Essentially you have a tiered storage environment but you don’t necessarily have a good method for aligning the tiers with the applications.

The earliest method for solving this was to incorporate some type of data mobility solution that allowed LUNs or volumes to be migrated between tiers without an outage. If an application’s storage performance requirements change or the initial design was incorrect you have the ability to make adjustments without much difficulty or disruption to the application environment.

When these types of solutions initially hit the market they were positioned pretty aggressively. A common concept was that you could move data between tiers as part of standard operating processes. For example the accounting databases could reside on RAID 5 most of the time but for end of month processing they could be migrated to RAID 10.

The main challenge with this approach is a lack of granularity. LUNs and volumes have continued to increase in size and the larger they get the longer they take to migrate between tiers. These larger capacities also impact the performance of the array overall since migration operations consume IOPS just like any other IO. This lead us to the latest evolution – sub LUN level tiering.

Sub LUN level tiering expands the tiering concept in two key ways.

- Each vendor’s implementation differs but in each case some unit of capacity smaller than a LUN is defined and allowed to migrate between tiers. The unit of capacity may be fixed or variable in size. Where new data is written also varies, some arrays always put new data to the highest performing tiers and then migrate down based on access patterns while others work in the reverse direction. In either case the result is that smaller amounts of data can be migrated which takes less time than migrating complete LUNs and requires less overall IOPS to do so. This solution should also require less high performance storage and cost less.

- The second change is that the migrations are now most frequently automated. Rather than the administrator selecting LUNs or volumes to move, the array migrates data automatically based on its algorithms and defined policies. The most frequently accessed data is moved to the highest performing tier while infrequently accessed data trickles down to the lower performing tiers.

This is a significant improvement and makes real time or at least near real time migrations possible. One of the most common use cases for this approach is for adding Solid State Disk (SSD) to an existing storage architecture. Solid State Drives are incredibly fast but they are also expensive when viewed from a capacity standpoint. For example, consider a 4TB database that needs 10,000 IOPS. The IOPS requirement can be met with only a few Solid State Drives but in order to meet the capacity requirements you have to purchase multiple SSD array groups, driving up the cost and resulting in tens of thousands of unused IOPS. Remember that just because the storage may be able to handle 100K IOPS doesn’t mean you are going to use them. Sub LUN tiering can address this problem by allowing you to purchase a smaller amount of SSDs combined with traditional spinning disk and migrating the data between the tiers.

This is a great tool and has the potential to lower costs significantly while maintaining or even improving performance. That being said, since this is the shiny new feature available in many vendor offerings it is frequently being positioned too aggressively and without enough consideration as to how it will actually work in a given environment. Here are some things to consider.

- If the performance and capacity requirements can be met with traditional allocation methods sub LUN tiering may not offer a lot of advantages. Taking the same 4TB database example from above but with 5K IOPS in performance requirements rather than 10K illustrates this point.

o Using 450GB drives in a RAID 10 4+4 configuration we get 1,250 IOPS per array group and 1.6TB of capacity. Using 4 array groups we get 6.4TB usable and 5K IOPS. Assuming we pay $1,000 per drive the solution costs $32,000.

o Solving the same problem with sub LUN level tiering might look as follows. Using 450GB drives in an 8+1 RAID 5 configuration we need 2 array groups to meet the capacity requirements. This provides 6.5 terabytes of capacity but only provides 2,130 IOPS. We add a tier of RAID 5 2+1 400GB SSD to this providing another 768GB of capacity and another 10K IOPS. This solution now has a total of 7.2TB of capacity and 12,130 IOPS. Using the 1K per drive cost times 18 drives equals 18K plus 10K per SSD equals $48,000. In this scenario the sub LUN tiering approach costs 50% more to meet the same requirements.

Some may look at my comparison here and say “well, that’s not a very good example – you created a scenario to intentionally make the standard solution look better”. Well that’s true and it is the point I was trying to illustrate. In some cases you may not want to use sub LUN tiering because you don’t gain anything. The ability of a storage array to do sub LUN tiering doesn’t mean that you should ignore the application requirements or discontinue looking at traditional allocation methods.

In addition to the cost differences indicated in the above example you need to understand that some amount of read IOs will be fulfilled by the lower performing tiers. If that number is small the solution will probably work fine but if it occurs too often the overall performance will suffer. This is similar to what we see in file archiving solutions. File archiving is a great way to lower costs by migrating infrequently accessed files to a lower cost storage tier but if your archiving policies are too aggressive the archive can get overworked and negatively impact the user experience. This is the same with sub LUN level tiering - your ratio of high performance versus high capacity storage must be correct for your workload.

Unfortunately many organizations do not have a good understanding of their performance requirements much less how much of the data is hot – accessed frequently versus cold – accessd infrequently. With unstructured data such as traditional file servers this is fairly easy to determine simply by looking at the last access times. Within databases it becomes a bit more complicated. Vendors that offer sub LUN level tiering are actively working on tools to help assess the requirements but they are application specific and not available for every scenario.

You also need to be cognizant of activities that may interfere with the algorithms used to determine where data should reside. For example backup operations may look to the array like standard access and be factored in. To avoid this some array manufacturers suggest that you omit monitoring during the backup windows. Depending on your backup environment this can be challenging.

None of this it to suggest that sub LUN level tiering is bad but rather that it is a tool just like any other and you need to put some thought into how and where you will use it. We are currently working on a solution that will put roughly 400TB of database capacity into a sub LUN tiering solution. We’ve collected performance data on all of the databases involved over a reasonable sample period. In architecting the solution we have added a significant high performance buffer to ensure we have the IOPS we need. When it comes time to implement we will be moving the databases over one at a time and monitoring the performance closely.

This 400TB represents about one quarter of the total capacity in the environment. We are not using sub LUN tiering in the rest of the environment because the IO profile wouldn’t benefit from it and performance would likely decrease.

As we work through the actual implementation will post what we find.

vSphere 4.1 VAAI integration with Hitachi's USP-V/VM, AMS 2000 series, and VSP

2011-04-29T10:47:00.000-07:00

Tuesday this week, HDS released microcode on the USP-V and USP-VM (Microcode 60-08-01-00/00) that includes VMware VAAI integration. The AMS2000 series and VSP already support VAAI integration.

This means that users of the USP-V/VM subsystem whom are already on vSphere 4.1, or are planning to upgrade to it, can now take full advantage of VAAI support. The main feature of VAAI that excites me is the Hardware Accelerated Locking; which relieves SCSI reserve locking pressure on the array by locking only a portion of the VMFS data store vs. locking the entire VMFS LUN upon physical host I/O. This can be a huge improvement for customers that use LUSEs, or that have large LUNs and a high number of VMs per LUN. Other VAAI features include Full Copy, and Block Zeroing which help with VM guest creation and cloning.

While this is the first release of VAAI integration code for the USP-V/VM, HDS has implemented these features in their AMS and VSP line of subsystems already. Typically storage managers do not like being on the first rev of a microcode release, however in this case the benefit is very much worth the low risk.

Solaris 10 dfstab bug

2011-02-20T09:48:00.000-08:00

I'll admit, it's been awhile since I have had to present NFS shares on Solaris 10 that were not ZFS file systems. ZFS makes this pretty easy, but hey, it's not like it was terribly difficult to edit the /etc/dfs/dfstab file. But after a recent head banging experience where my dfstab file would get magically overwritten during reboots, and my shares commented out; I started to wonder if I really need to hang up this IT thing and open the donut shop.

It turns out that Oracle has a little bug in Solaris 10 (6941744) which basically says that libsharecore.c has some incorrect code that improperly affects Solaris 10 systems.

Based on this thread it doesn't sound like a fix is here yet, so I put this together quickly to get past the issue. While I have no reason to suspect this script will cause issues, I provide it to you with the understanding that you will ensure it is the right solution for your environment. I placed this script at /etc/init.d/local, and linked it to rc2.d and rc3.d as necessary.

#!/usr/bin/bash

# This is a cleanup script due to an issue in Solaris 10 update 9 and previous

# SUN/Oracle bug 6941744

# From /etc/dfs/dfstab

# --------------------

# Use the sharemgr(1m) command for all share management

# This file is reconstructed and only maintained for backward

# compatibility. Configuration lines could be lost.

# --------------------

# created by Justin Richards, Lumenate

# Version 1.0 Date 02202011

CAT="/usr/bin/cat"

SED="/usr/bin/sed"

CP="/usr/bin/cp"

SHARE="/usr/sbin/shareall"

case "$1" in

start|restart)

$CAT /etc/dfs/dfstab | $SED '/^# Error: Syntax: /s/# Error: Syntax: //' > /etc/dfs/dfstab.tmp ;\

$CP /etc/dfs/dfstab.tmp /etc/dfs/dfstab ; $SHARE

;;

echo "Usage: $0 {start|restart}"

exit 1

;;

esac

That Looks Bad

2011-02-17T08:57:00.001-08:00

Ever been caught in a situation like this (see more)? Last year we documented various recovery procedures for one of our clients to assist them with full system recovery without the assistance of Symantec’s Bare Metal Restore (BMR).

We took care to only document procedures that a systems administrator could follow without the aid of the NetBackup admins, so I told the admin, “hey, since you are the target audience for these procedures, I’ll just look over your shoulders as you follow them.”

Enter Mr. Murphy. Obviously, one critical step in the restore process is restoring the C:\ drive. When it came time to select the recovery point in time, the NetBackup Backup, Archive, and Restore (BAR) application pops up the following dialog:

“That’s funny. I don’t find that on my procedures,” said the admin with a grin. Yeah, I know.

I knew the server had good backups, so why was NetBackup giving me the stiff arm? I checked the policy configuration on NetBackup and realized NetBackup 7 no longer has an “MS-Windows-NT” policy, but “MS-Windows” instead.

Let’s face it, I understand why Symantec changed the name of this policy type. One could argue this change should’ve been made a long time ago, and it’s a great example of how seemingly innocent changes can have unintended consequences.

Since BAR can restore from any policy type, we attempted to configure it for “MS-Windows” instead of “MS-Windows-NT", but BAR knew nothing of “MS-Windows” because it was a NetBackup 6.5.x client.

Fortunately, there was a way out. We completed our restore procedures using server-directed restores (thanks to Frank for the great suggestion). I simply initiated the restore from the Java admin GUI. Now I’ve got some screen shots with which to update the documentation.

Here were the key components for a successful restore:

Install NetBackup to a drive other than C:\
Remove the system from a domain (if this applies) prior to attempting the System_State restore.
Add the system to DNS or ensure forward/reverse name lookups function
I found some invalid SERVER entries in the client’s SERVER list that prevented a successful restore.
Check available disk space before the restore.
Delete the Windows page file (or reduce it greatly) prior to restoring the system state.

Chicken Little Messages in Storage Navigator

2011-01-03T22:41:00.000-08:00

In preparation for a ShadowImage server migration, I was formatting a couple dozen LDEVs of a plethora of odd-ball sizes. Fortunately, Storage Navigator allowed me to define all of the new custom volumes I needed (so long as I defined all new volumes within a RAID group at the same time).

Once I hit apply, SN asked me if I wanted to use quick format, which is sort of a stupid question to ask for two reasons. The first reason this is a stupid question is who wants to use the slow option?? "When someone asks you if you’re a god, you say YES!" right? Anyway, I said "YES"…

I had forgotten the system disk requirement for quick format on the USP-V, which is exactly the second reason the above dialog box is a stupid question, since the array should know full well the system disk requirements are not met.

Anyway, after saying, "Yes, I want to apply these changes" SN put up a dialog box suggesting the format was proceeding.

This of course was total baloney because not too long after this it says the formats failed:

At this point, I was pining about all my wasted time I spent defining the new LDEVs. I thought to myself, "I should’ve been honest and said I wasn’t a god." But as it turns out, SN did actually define my LDEVs, but they were blockaded.

During the process of defining the new LDEVs I realized I was going to have to volume-to-space (V2S) one more LDEV in order to assign it a lower CU:LDEV number so it could be used as the head volume of a LUSE (LUSE requires all sub-volumes to be added in numerical order). I figured since my formats failed, I would V2S the LDEV, redefine it, and then format everything all at once.

So I hit Apply on the V2S operation, locked my computer screen and stepped away momentarily, expecting the operation to be completed by the time I got back. Only, when I got back I saw Storage Navigator reporting that the sky was falling:

Had I been more prescient in the moment, I would’ve considered the blocked LDEVs as the most likely culprit. But I had actually just told my wife, "I’ve got one more thing I want to do so I don’t have to do it Monday… shouldn’t take long at all." Haha. Thank you Mr. Murphy.

Turns out my failed formats prevented the V2S operation because the LDEVs were blockaded. Completing the formats first allowed me to proceed with the V2S operation.

Data replication options - Part Two (Application-based replication)

2010-12-13T08:00:00.000-08:00

At the top of the IT stack lies the application. This is what the end-user interfaces with and what you're most likely to hear "is down" in the event of an outage. Since the application is what you're actually trying to protect against a disaster it makes a great deal of sense to leverage any built-in options for data replication.

And luckily applications have added built-in replication as the importance of disaster preparedness increased. A few brief examples:

Oracle offers Data Guard for their RDBMS. While this solution could have been considered simply log shipping in the past, today they position it as a complete disaster recovery solution.

For the popular Exchange Server, Microsoft offers Cluster Continuous Replication (CCR), while for SQL Server there are options both for log shipping and transactional replication. A more detailed discussion of the SQL Server options is available here.

If you define "application" broadly enough to include infrastructure services, then there are typically options there as well. DNS, LDAP, and Active Directory are all architecturally designed so that they may be deployed in a fashion that is redundant across multiple sites - the key is to recognize the need for this redundancy, deploy appropriately, and test.

Given that the application is what we're trying to protect, then why doesn't everyone just rely on application-based replication? Well, there are a couple of considerations:

First of all, most environments have multiple applications that they're trying to protect against a disaster. In the same way that the real test of a backup is whether or not you can restore, the real test of a disaster recovery solution is whether or not you can recover. When you use an application-based replication solution then when a disaster happens you have to have a person available at the remote site who knows enough about the application to perform the necessary steps to bring it online. If you're only running one application (if you're a Software as a Service provide, for example) then that's great. You have the necessary resources for that one application. As you start putting more and more applications into the mix, though, the probability that you won't have the right resource available in the event of an emergency increases.

A second reason is that leveraging application-based recovery couples your disaster recovery solution with the support matrix of the application. This means that as time progresses and the application move through its life cycle you have to include the replication in your considerations.

To summarize:

Pros:

Protects the environment at an easily understandable level.
Typically cost-effective.
No concerns around application support (as it is part of the application).
Often includes testing for logical corruption (which other approaches cannot).

Cons:

Increased complexity in environments with multiple applications, decreasing the probability of successful recovery of an entire environment in a disaster.
Couples replication to the application, meaning that application maintenance and upgrades must include testing and validation of replication.

Storage Performance Concepts Entry 4 - The Real World

2010-12-06T09:00:00.000-08:00

In the previous three entries on this topic we discussed several key storage performance concepts.

The Physical Disk Capabilities. Fibre Channel and SAS drives can handle more IOPS than SATA drives, making them a good choice for applications that generate a lot of random IO. From a MBPS standpoint SATA isn’t quite as fast as Fibre Channel and SAS but the delta is much smaller, making SATA acceptable for workloads that mainly generate sequential IO.

The common RAID implementations and their impact on storage performance. RAID 10 provides the best performance for random workloads. RAID 5 and 6 provide good performance for sequential workloads in some cases RAID 5 may actually be faster than RAID 10 – although this isn’t the norm.

The workload, the mix of; random, sequential, reads and writes has a major impact on performance, with writes putting the biggest load on disk drives.

We also showed how this formula could be used to determine the number of array groups that would be needed to meet an applications IOPS based on the disk drives and RAID level you choose.

(TOTAL IOps × % READ) + ((TOTAL IOps × % WRITE) ×RAID Penalty)

We left off pointing out that while this information is valuable it leaves off some of the challenges we face when architecting solutions in the real world. Two factors we have not considered are capacity and cost. The majority of the time we start building our solution based on the capacity requirements.

For example, an organization might need 10TB of capacity to support a new application with a random workload consisting of 75% reads and 25% writes with a peak IOPS load of 2,500. The capacity will be added to an existing array that supports both SAS and SATA drives.

Since this is a random workload we will be recommending SAS drives but aren’t yet sure if this needs to be a RAID 10 or RAID 5 configuration. We could use RAID 6 but since we will be using 450GB drives and our array has multiple hot spares we think that RAID 5 will provide suitable protection for the data.

First we will look at the capacity requirements for each RAID level.

Capacity	RAID 5	RAID 10
RAID Group Size	8+1	4+4
Usable Capacity	3,600	1,800
Required RAID Groups	3	6
Total Capacity	10,800	10,800

Using 450GB LUNs we need twice as many RAID 10 groups as RAID 5 groups to meet the capacity requirements.

Now we will take a look at the cost. We are using the same size drives for each configuration so the cost per drive is constant but we need almost twice as many drives for the RAID 10 configuration. In addition the number of drives in the RAID 10 configuration will require additional drive trays to be added to the array. In our case we are assuming that the drives are $1,500 a piece and that each tray holds 15 drives at a cost of $10,000 per tray.

Cost	RAID 5	RAID 10
RAID Group Size	8+1	4+4
Required RAID Groups	3	6
Total Disks Required	27	48
Trays Required	2	4
Cost of disks	$40,500	$72,000
Cost or trays	$20,000	$40,000
Total Cost	$60,500	$112,000

As you would expect the cost of RAID 10 is almost twice as high as RAID 5. What may not be as obvious are the performance differences between the two configurations. In the past we focused on comparing a single RAID group of each type, keeping the number of drives constant. In this case two things have changed.

1. I’m using an 8+1 array group rather than a 7+1. 8+1 is the RAID 5 configuration recommended by the manufacturer because of the way it aligns with the caching mechanisms of the array. In addition an 8+1 provides sufficient availability and rebuild times while making better use of the raw space.

2. In this real world configuration I have twice as many RAID 10 groups and therefore a lot more disk, hence raw IOPS.

Using 185 IOPS per drive we find that the two configurations have the following characteristics.

Performance	RAID 5	RAID 10
IOPS Per Drive	185	185
# of Drives	27	48
Raw IOPS	4995	8880

We can now use our formula to determine if either solution will meet our requirement.

Performance	RAID 5	RAID 10
Required IOPS	2500	2500
Percent Read	75%	75%
Percent Write	25%	25%
RAID Penalty	4	2
Adjusted IOPS	4375	3125

In our example both RAID 5 and RAID 10 meet the performance requirement. Although the RAID 5 configuration is tighter, it has a reasonable amount of headroom. Given the major difference in cost it is probably reasonable to proceed with the RAID 5 configuration.

Looking at our results you may say “Well the RAID 5 configuration may work but wouldn’t the RAID 10 design be a lot faster?” Well, not necessarily. If the speed limit is 55 and you must drive the speed limit a Ford F150 and a Ferrari will both get you there in the same amount of time. This is the same with storage, just because one configuration could run faster doesn’t mean it will – you have to be able to drive higher IOPS from the host.

The area that we will explore in our next entry is cache. While cache improves performance in general it is particularly beneficial for parity based RAID configurations.

Data replication options - Part One (Overview)

2010-12-01T08:00:00.000-08:00

As mentioned previously, I'll be going over different approaches to data replication between now and the end of the year. In this post I'll outline the different points in the data path where replication can occur and then follow up with a post per point (say that fast three times!) outlining pros and cons.

Before touching on the different points it is important to define some terminology.

Using a broad brush, all replication solutions may be divided into either synchronous or asynchronous. Synchronous replication means that the write must be received at the remote location before the local host receives acknowledgement. Asynchronous replication means that the local host receives acknowledgement once the write is complete at the local site, and replication of the write is handled, well, asynchronously.

Two quick comments:

Using asynchronous replication means that there will be some data loss in the event of a disaster. How much depends on several factors including the replication technology in use, the bandwidth between the sites, the change rate of the application(s) in question, and so on and so forth.
In practice, most people use some form of asynchronous replication. The biggest reason? Synchronous replication introduces latency into every write - it has to traverse the link between locations and then an acknowledgement has to be sent back. If the sites are close enough and the budget will support it, then synchronous is possible - but it's expensive.

Replication solutions can be further divided into continuous or discontinuous. Continuous means that for every write generated at the primary site the same write is performed at the remote site. Synchronous replication is, by definition, continuous. Asynchronous replication may be either continuous or discontinuous. Snapshot-based replication is an example of a discontinuous approach. Discontinuous replication can lead to significant cost savings in the bandwidth required between sites if the application in question writes to the same location repeatedly, since only data at the scheduled time is replicated and the intermediate writes can be disregarded for replication purposes.

One last bit of exposition - I'll refer to replication as being write-order consistent (or "crash" consistent) throughout these posts - this basically means that writes are applied in-order for the solution. This is a requirement for the successful replication of databases and other applications.

With that out of the way - here are the five places on where replication can occur, going from the highest level to the lowest level.

The Application - most enterprise applications in use today have some form of replication baked in (possibly at the cost of an additional licensing fee). Oracle Data Guard, Exchange CCR, MicroSoft SQL log shipping - all have some method of getting the data from point A to point B in a usable form.
The Host - from something as simple to a write-splitter to something as involved as replacing the native volume manager, there are more host-based options for data replication than you can shake a stick at.
The Switch - Several years ago there was a movement to make all storage intelligence (including replication and virtualization) built into the switches. That vision didn't really pan out, but it is still possible to do replication from the switch.
An Appliance - begrudgingly added against my initial biases there are a number of appliances on the market that can perform virtualization. Some involve virtualization of the storage in question, and could arguably be classified as array-based, while still others involve the installation of write-splitters on the hosts and could be classified as host-based, but I digress.
The Array - Once the domain of only monolithic storage arrays, today most midrange arrays offer data replication solutions that are sufficient for typical needs.

I believe that all replication solutions can be grouped into one of the five categories above. Coming up next - Application-based replication.

Storage Performance Concepts Entry 3

2010-11-24T09:00:00.000-08:00

So far we have covered the performance characteristics of the physical drives as well as the impact of various RAID levels. In this entry we will take a look at the next level in the stack – the workload.

As we mentioned in the previous entries, most applications do a mix of IO; reads, writes, sequential and random. We’ve also touched on IOPS versus MBPS. These concepts are related in that with random IO you are typically most interested in IOPS while with sequential IO you are normally interested in MBps. We pointed out that while SAS and FC drives can handle significantly more random IO than SATA, for sequential operations the MBps capabilities are comparable.

We can use this information when architecting solutions. Operations such as backup to disk are typically sequential in nature meaning that we are more concerned with MBPS and that SATA may be a good choice. OLTP applications perform primarily random IO with a mix of reads and writes, making SAS or Fibre Channel a better fit.

There are typically two scenarios we are looking at when designing storage solutions, either A) this is a new application not yet implemented, or B) this is an existing application that we are looking to move to a new storage platform.

When architecting the storage for a new application we may not have as much information to work with. Since we cannot collect any real world statistics we have to rely on the application provider to give us insight into the expected workload. This can be a challenge because in some cases they just don’t know. The workload characteristics are heavily dependent on the modules in use, the overall size and how your organization will actually use the product. This is in part why many software vendors and in particular database vendors recommend RAID 10. RAID 10 is the fastest for these workloads and therefore the safest recommendation. Regardless you will be best served by getting as much information as possible on the actual IO requirements in terms of IOPS, MBPS, reads versus writes and the mix of sequential versus random IO.

In situations where the application is already in place and you are looking to move to a new storage platform the challenge is a bit easier since you have real data to look at. There are two ways to collect the workload information we will need.

If the existing storage array has the capabilities, you can collect the information directly from the array. If you use this approach make sure you pay attention to where the data is being collected from and what it is telling you. For example if you collect the physical IOPS from the backend disk drives or RAID groups, the numbers you are looking may include the impact of the RAID configuration - the actual data being written as well as parity or mirroring operations. In this scenario you may also miss IOs that are being handled in cache. Ideally what you would like to collect are the peak IOPS per logical device (referred to as LUN, LDEV or Volume depending on the nomenclature used by the array manufacturer) broken down by reads versus writes, sequential and random.

The same is true for MBps if that is what your application calls for. If the storage device provides this capability and has a decent interface this might be the easiest way to go. You can collect the information in one place and filter or sort the data to get exactly what you are looking for.

The second option is to collect the data from the application hosts themselves. Being a storage guy I typically don’t like to use stats collected from within the application. The primary reason is that I’m not always as familiar with them and therefore less confident about what they are telling me. Are these disk IOs or are some of the IOs coming from memory or the network? For this reason I typically recommend collecting the information using the native Operating System utilities such as performance monitor on Windows or sar or iostat for Unix. Make sure you understand the correct Syntax for your flavor of Unix.

The following link provides a basic overview and some examples of how to use these utilities.

IO Collection Utilities

One more thought on collecting performance data. It is very helpful to understand the access patterns from an application point of view, when are the busiest times, when do operations such as backups or data loads occur. You want to make sure you are capturing peak loads and understand any anomalies.

At this point you understand the performance characteristics of the disk drives themselves, the impact of RAID levels and the IO patterns associated with your workload. The next step is to combine this information to determine which approach will best meet your needs.

The simplest formula I have found for doing this is.

(TOTAL IOps × % READ) + ((TOTAL IOps × % WRITE) ×RAID Penalty)

So for example if your application generates 1000 random IOPS and a mix 70% reads and 30% writes a RAID 5 solution would work as follows.

(1000 x .70) + ((1000 x .30) x 4) = 700 + 1200 = 1900 IOPS

Using 15K SAS drives each capable of 180 Random IOPS in a 7+1 configuration =1,225 IOPS. Based on this I would need two 7+1 RAID 5 groups.

Let’s look at the same example using RAID 10

(1000 x .70) + ((1000 x .30) x 2) = 700 + 600 = 1200 IOPS

In this scenario, using the same drives types I would need one 4+4 RAID 10 Group.

A full overview of this formula and some real world examples can be found at the following link.

Workload Calculation Formula

So there you have it, the backend disk performance, the impact of the RAID configuration and workload considerations. So what does that leave? Well, in order to properly design a storage solution we have to understand these concepts but in a lot of ways that is what they are, concepts. What these concepts do not take into account are cache and cash! Cache has a significant impact on the performance you experience and cash impacts what you will actually be able to purchase. While working with clients we often start by looking at a solution based purely on RAID 10, and then we see the price and all have a good laugh before getting to work on the real solution. Applying what we have learned so far to real world designs is the focus of our next entry.

Of strategy, tools, and trade-offs

2010-11-24T08:00:00.000-08:00

I'm going to take some time over the remainder of this year to talk about different solutions for data replication. It's a complicated discussion, and I'll be limiting the scope to only cover solutions that are write-order consistent (sorry rsync fans).

While collecting my thoughts, though, I thought it made sense to talk a little philosophy.

First, I want to touch on strategy. A few years ago I was called into a brainstorming meeting with a customer to discuss their IT strategy and make suggestions. It was a very brief meeting, as their strategy was "We use vendor X." Confused, I tried to clarify what, exactly, that meant. "You mean you use them to fulfill a piece of your strategy?" I asked. "No, whatever X tells us to do, we do." Six months later, X was bought by Y, and not too many years later Y was bought by Z. The customer, meanwhile, was left high and dry.

Strategy does not equal products. Strategy does not equal vendors. A strategy is a plan. A plan identifies direction not details. Customers (should) have a strategy. Vendors (should) have a strategy. Sometimes the two align and there can be a mutually beneficial relationship. If a customer cedes control of their strategy to a vendor, though, then they are basically saying that IT does not matter to their business. While that may be the case it should be considered and made as a conscious decision.

Second, it's important to realize for all the passion around specific products and approaches they are ultimately tools. Products are neither good nor bad - they are either fit for a purpose or unfit. Just because it's made by a specific vendor doesn't make it bad, especially in the acquisition-crazy world we're living in. Just because it's not the approach you're used to doesn't make it bad, either. The real question is does it do the job I need done, at a reasonable cost (both hard and soft dollars) without forcing me to fundamentally alter the way I do business today.

And that ties into the third point, trade-offs. There is no one perfect solution for every problem - and thank God because that would be a boring world. Sometimes you're willing to trade an hour's worth of data loss in the event of a disaster for the (substantial) cost savings. Sometimes you're willing to single-path development hosts. The key is to understand what trade-offs are available, and again make a conscious decision about what is right for your environment.

With that as the groundwork, I'll continue next week with the common approaches to data replication.

Storage Performance Concepts Entry 2

2010-11-22T09:00:00.000-08:00

Our first entry on storage performance focused on IOPS and MBps from a physical disk drive perspective. This entry will take a closer look at performance as it relates to various RAID levels. Since this entry is intended to be practical we are not going to focus on all of the RAID levels or proprietary approaches, but rather the most common options you might be considering as you design a storage solution. In general these are RAID 10, RAID 5 and RAID 6. As a recap, here is how these levels are defined as per the SNIA Dictionary.

RAID 5

A form of parity RAID in which the disks operate independently, the data stripe size is no smaller than the exported block size, and parity check data is distributed across the RAID array's disks.

RAID 6

Any form of RAID that can continue to execute read and write requests to all of a RAID array's virtual disks in the presence of any two concurrent disk failures. Several methods, including dual check data computations (parity and Reed Solomon), orthogonal dual parity check data and diagonal parity have been used to implement RAID Level 6.

RAID 10

RAID is referred to as a nested or hybrid approach and is therefore excluded from the SNIA Dictionary. Technically RAID 10 is just a combination of RAID 1 and RAID 0 – mirroring and then striping. Years ago I would hear debates about which arrays did RAID 10 versus those that did RAID 0+1 but I haven’t heard any grumblings in a while so we will stick with RAID 10 to keep it simple.

Putting disk drives into RAID groups does not fundamentally change how many IOPS they can handle, the raw IOPS capabilities of a drive is constant. The reason we experience differences in performance have to do with how the various RAID levels handle IO and in particular any additional IOs generated to protect the data.

· A random write operation to a RAID 10 configuration results in 2 disk IOs, one to each drive in the mirror. The same single write request to a RAID 5 set would generate 4 IOs; read the data, read the parity, write the data, write the parity. RAID 6 adds two additional parity operations to each write.

· A sequential write IO to RAID 10 works the same as a random write, one to each disk in the mirror. For RAID 5 and 6 sequential operations are a bit more efficient than random IO since there is no existing data or parity to be read; write the data, write the parity.

· Read Operations are a bit different since neither parity nor mirroring come into play, it’s simply a matter of how many drives can you read data from concurrently. Assuming that each RAID configuration has the same number of drives, RAID 10 is slightly faster than parity RAID.

Here is a chart that makes this a little clearer.

Note: The chart is based on sending 4 IOs to the RAID group. I could have used a single IO as the base for the comparison but this would not demonstrate the differences in sequential and random IO for the parity based RAID configurations.

	RAID 10	RAID 5	RAID 6
Random Read	4	4	4
Random Write	8	16	24
Sequential Read	4	4	4
Sequential Write	8	5	6

As you can see the differences are pretty significant. By combining the information in our first entry about the performance capabilities of the various drive types with the characteristics of the most common RAID levels you can get an accurate picture of the overall backend performance that the system is capable of. Of course in practice workloads don’t typically fall neatly into these categories, but rather perform a combination of different IO types.

Understanding the workloads and how they impact performance is the subject of our next entry.

NetBackup Catalog Backups Fail With Status Code 2

2010-11-15T08:00:00.000-08:00

During a routine review of a customer's environment I discovered that their most recent catalog backup failed with status code 2 (none of the requested files were backed up). A quick check revealed the horrifying condition that the catalog backups had been failing for a week and the last catalog backup had just expired.

I checked the most recent bpdbm logs (in /usr/openv/netbackup/logs/bpdbm) and noticed the following curious entries:

<16> backup_dbspaces: ODBC connection failed.
<16> backup_dbspaces: ErrMsg [Sybase][ODBC Driver][SQL Anywhere]Database server not found, ErrCode -1, Sqlstate 08001

Google searches revealed that the /usr/openv/db/data/vxdbms.conf file had been a culprit for some people. Apparently this file contains parameters for backing up the NetBackup catalog database. In my case, I immediately noticed the file pointed to a server other than the master server.

It seemed reasonable that I had found the root cause of the failed catalog backups, though I had no idea how or why the file might have changed. Also, never having the occasion to edit the vxdbms.conf file I wanted to be sure I was doing the right thing, so I opened a support case.

Support confirmed for me that the file should point to the master server. The the following procedure resolved the status code 2 on the catalog backups. (We had to gracefully terminate some running jobs first.)

/usr/openv/db/bin/create_nbdb -config_only -force
/etc/init.d/netbackup stop
/etc/init.d/netbackup start

NetBackup 7 Restore Blues, Catalog Woes

2010-11-08T08:00:00.000-08:00

I recently ran into an issue where a customer couldn't restore files from a backup. In fact the NetBackup GUI immediately reported "There are no files matching the specified criteria." Normally this would implicate my search criteria; perhaps I specified an incorrect policy type or media server. However, I repeatedly got the same error even when all I changed was the ending search date (specifying an earlier backup).

In the NetBackup "Backup, Archive, and Restore" applet I could generate a backup listing for backups prior to 9/15/2010, so I knew they were there. Merely moving the end date earlier than 9/15 generated the above error.

In fact, after a little testing I found I could generate the error for all backup images for all clients prior to a certain date. Command line attempts to find the file (via bplist with the appropriate start/end dates) immediately resulted in status code 227 (no entity was found), so I was pretty sure it was a catalog problem.

The customer had recently upgraded their master server from 6.5.6 to 7.0, so it was tempting to assume the upgrade had somehow corrupted part of the catalog. However, the timing didn't fit the circumstances.

Analysis of extant backup images via bpimagelist (using the -L flag) showed an interesting coincidence. All backups prior to 9/15 (and some taken early the morning of) were marked "DB Compressed: Yes" in the bpimagelist -L output.

Apparently the customer had enabled catalog compression, which dutifully compressed catalog entries as directed, but prior to the NBU7 upgrade. The NetBackup administrator guide for UNIX, volume 1 states enabling catalog compression reduces disk consumption at the expense of performance when a user lists or restores files.

Now things started settling into place. The customer transitioned from SPARC Solaris to RedHat Linux 5.4 as part of the NetBackup upgrade. By default, RedHat Linux installations do not include ncompress. Installing the ncompress RPM onto the master server immediately resolved the issue.