"Unfortunately this has not been documented very well." Fun with VERITAS Cluster Server

So Justin and I are wrapping up a large refresh project for a client where we're moving them from an existing configuration running Oracle on Sun 6800s with VCS over to new M5000s.  As you'd expect this includes a migration to Solaris 10 as well as upgrades to VERITAS Volume Manager, File System, and NetBackup (that's Symantec Storage Foundation and NetBackup to some of you).


The application team went through their testing over the last month or so and we completed our VCS test matrix in preparation for cutover.  During the cutover, though, we noticed the following message in the alert log:

WARNING:Oracle instance running on a system with low open file descriptor
        limit. Tune your system to increase this limit to avoid
        severe performance degradation.

Thinking that we'd missed a resource control setting somewhere we went through the process of validating those settings.  Then, seeing that they looked correct, we asked the DBA to stop and restart the database manually only to find that the error message above didn't appear.  Using VCS to stop and start the database would generate this error every time, though.


We opened a case with Symantec and started to troubleshoot.  Thankfully we found that in VCS 5.1 SP1 Symantec added a file called vcsenv that hardcodes limits for CPU time, core file size, data segment size, file size, and the number of open file descriptors before we ran out of window for the cutover.


The location and contents of the file are shown below, including where we set the number of file descriptors to 8192.

bash-3.00# cd /opt/VRTSvcs/bin

bash-3.00# more vcsenv

# $Id: vcsenv,v 2.8 2010/09/30 05:45:29 ptyagi Exp $ #

# $Copyrights: Copyright (c) 2010 Symantec Corporation.

# All rights reserved.

#

# THIS SOFTWARE CONTAINS CONFIDENTIAL INFORMATION AND TRADE SECRETS OF

# SYMANTEC CORPORATION.  USE, DISCLOSURE OR REPRODUCTION IS PROHIBITED

# WITHOUT THE PRIOR EXPRESS WRITTEN PERMISSION OF SYMANTEC CORPORATION.

#

# The Licensed Software and Documentation are deemed to be commercial

# computer software as defined in FAR 12.212 and subject to restricted

# rights as defined in FAR Section 52.227-19 "Commercial Computer

# Software - Restricted Rights" and DFARS 227.7202, "Rights in

# Commercial Computer Software or Commercial Computer Software

# Documentation", as applicable, and any successor regulations. Any use,

# modification, reproduction release, performance, display or disclosure

# of the Licensed Software and Documentation by the U.S. Government

# shall be solely in accordance with the terms of this Agreement.  $

# This is just a sample as to how you can specify various environment

# variables you need to set.  Uncomment/add/modify the values as per

# your requirement.

#

# Specify the default language in which you want to bring up the

# VCS agents.

# LANG=C; export LANG

VCSHOME="${VCS_HOME:-/opt/VRTSvcs}"

#This is required for agents which use dynamic VCSAPI libraries

LD_LIBRARY_PATH="$VCSHOME/lib:$LD_LIBRARY_PATH"

export LD_LIBRARY_PATH

# Setting ulimit.

# Common For Linux, HP-UX, SunOS & AIX

ulimit -t unlimited     # CPU Time

ulimit -c unlimited     # Core File Size

ulimit -d unlimited     # Data Seg Size

ulimit -f unlimited     # File Size

ulimit -n 8192          # File Descriptor

if [ `uname` = "AIX" ];then

        RT_GRQ=ON

        export RT_GRQ

fi

"Unfortunately this has not been documented very well." Fun with VERITAS Cluster Server