PowerHA fix information

Readme for PowerHA 7.1.3 Service Pack 1

Service Pack 1 important content

This service pack includes important updates for the following features:

SAP Smart Assist
Hyperswap Support

You can view and install individual fixes ased on your specific configuration.

APAR IV56377: allow syslog to manage cluster.log instead of clcycle

The PowerHA utility clcycle manages most log files generated by the product. clcycle will automatically manage cluster log files such that individual logs are limited to a maximum size and are removed after they reach a certain age or are overwritten by newer version. In general, the cluster logs are written to directly by the cluster components, and the maintenance performed y clcycle is coordinated with those components. One exception is the cluster.log file which is written to using the AIX syslog facility.

In order to manage the cluster.log, clcycle needs to refresh the syslogd subsystem, which can cause syslogd to lose track of other refresh functions that were scheduled by other system components.

With APAR IV56377, clcycle will no longer manage cluster.log by default, and the syslog facility will e configured to manage cluster.log instead. Installing this update will automatically update the /etc/syslog.conf configuration file to configure syslog management of cluster.log. If you want clcycle to continue to manage cluster.log, you can override this behavior as described in the following section.

With this update, the default behavior of clcycle can also be customized as follows:

In general, PowerHA will default to these rules for all log files:

If you wish to customize these values, you can override them by specifying different values in the /etc/environment file on each cluster node. Add the following entries to override the default(s):

CLCYCLE_MAX_SIZE = <size in bytes>
add this entry to limit the maximum size of any saved log file
CLCYCLE_MAX_LOGS = <number of old files to save>
add this entry to change the number of old log files preserved by clycle
CLCYCLE_MAX_DAYS = <cycle log files older than this number of days>
add this entry to change the age at which log files will be cycled
CLCYCLE_CLUSTER_LOG = <FALSE|TRUE>
if you want clcycle to manage the cluster.log file instead of syslog, you can set this value to TRUE

Critical Volume Group handling

PowerHA SystemMirror includes special volume group monitoring which can be customized with a specifice response when the group fails. You can now define a volume group as a "critical volume group" and select a response when a failure occurs:

This facility is useful for monitoring and protecting Oracle RAC voting disks.

Disabling automatic restart of ERS

In order for the SAP smart assist to correctly manage failovers, you must disable automatic restart of ERS.

To disable automatic restart of ERS, you will need to edit the profile for the ERS instance located in /usr/sap/<SID>/ERS<nn>/profile/

Other related APARs

It is important to keep your entire installation up to date with the most recent and important fixes for SysteMirror, AIX, CAA, RSCT, and any other components on which you rely. The following list is not all inclusive and may not be applicable to your installation, but you should consider applying these fixes as needed.

APARs included with this service pack
IV54588 CLMGR: Add support for reconstructing repos disk
IV57311 CLMGR REPLACE REPOSITORY FAILS AFTER MANUAL CHREPOS
IV57312 CLMGR query might interfere with RG moves
IV57901 CL_PVO FAILS TO DETECT IF ECM VG IS ONLINE IN NON CONC MODE
IV57902 CL_PVO FAILS TO ACTIVATE MISSING/REMOVED DISKS
IV57903 CL_PVO FAILS TO HANDLE LACK OF QUORUM (712=IV57735)
Important fixes for other components (available from Fix Central)
IV53656 CAA: RECOVER CLUSTER FROM CACHE FILE
IV57462 CAA: DEADMAN TIMER TRIGGERED WITH SANCOMM AND SHUTDOWN OF NODE
IV58961 CAA: LOSS OF REPOS AND REBOOT IN UNICAST CLUSTER CAUSES DMS TIMEOUT
IV53376 SVC SCSI interactions with AIX is causing long delays
IV52962 Error configuring CAA with this disk as repos disk. AIX DD fix http://www.ibm.com/support/docview.wss?uid=isg1IV52962
IV57361 JFS2 FS MARKED CORRUPT WHEN USING FIND COMMAND NEEDING FSCK RUN
IV53587 XMGC NOT TRAVERSING ALL KERNEL HEAPS (HIPER)
IV57435 VARYONVG -CA OF CONC VG WITH FIRST PV REMOVED CAN FAIL OR HANG
IV52351 Reboot of Site A node causes Site B node to reboot
IV44243 HAGSD CORE ASSERT FAILED ASSERTION: NULL != ATTRIBUTES
IV52904 "BUFFER WRAP" HANDLING MAY CAUSE CTHAGS EXIT AND NODE FAILURE
IV56614 CORE DUMP WHEN MULTIPLE HAGSD INSTANCES RUN UNDER POWERHA 7
IV56613 HAGSD CORE DUMP ON BAD PACKET

This Service Pack includes the following PTFs with associated APARs for PowerHA 7.1.3 and is now available for download as of May, 2014.

New APARs included in this service pack
IV53751clinfo may terminate in config with sites
IV54588clmgr improves caa repository disk recovery
IV55094rg left in releasing after shutdown during stop cluster services
IV55095ha smit cluster security screens shown but not active
IV55096verification error parent directory is already in export list
IV55097change password utility fails to exit correctly
IV55098site field is missing in change serviceip smit panel
IV55099clverify warning about super strict mirroring
IV55101clrgmove -r fails with error message
IV55102cl_set_vg_fence_height errors in smit change/show volume group
IV55103disk can be local, so we should not discover vg
IV55104asyncrhonous cache recovery failure due to stale pvid table
IV55105nfs crossmounts not remounted at network up event
IV55106dlpar cod cpu activated for hard coded 30 days
IV55107site type smit input has f4 list when it should not
IV55108spprc cspoc.log full of 'lsvg -l' lvm errors at midnight clver
IV55109vg is not parsed properly
IV55110smit menu "add a network" missing field for public/private
IV55111correct ver_mping code to read better mping command output
IV55112cl_verify_svcpprc_config performance degradation
IV55113tsm client smart assist will not list proper fs for backup
IV55114cspoc create new vg and rg shows 'putlvodm' error message
IV55115clver core dump when applying snapshot
IV55116cldare fails with multicast communication error.
IV55117cannot change "notify_method" to empty value
IV55118add f1 help to group services log length menu option
IV55120some data can be missing from verbose clmgr queries
IV55121websphere mq rg in error state as start/stop scripts fail.
IV55122clmgr add snapshot "save_logs=true" fails
IV55123clmgr start cluster with automatic correct errors failure
IV55124snapshot method labels are not saved in the ".info" file
IV55125codeving only 1 node using "clmgr delete cluster" gives bad
IV55126if adding mq sa fails, cleanup is not done properly.
IV55128swap wont happen automatically for system mgs
IV55129f4 picklist for add a node may not list all available choices
IV55130clconfig and cldare don't use customized clevents log directory
IV55131ers monitor scheduling prevents startscript to be called
IV55132networks defined as private may still be heartbeat
IV55133clmgr allows 'cluster name' change on configured cluster
IV55134smit menu 'remove a repository disk' shows active rep.disk
IV55135svc verification incorrectly fails from ping dos errors
IV55136rg move can disable firstalias unintentionally
IV55138sap smart assist fails in smitty after discovery.
IV55139cluster verification shows "warning" on clhosts.client
IV55140sap admin functions will not be reflected in powerha
IV55141error adding ipv6 pers ip to ipv4 nw with specific prefix len
IV55142"clmgr view report cluster type=html" can emit errors
IV55143timeout attribute not working with clmgr stop
IV55144multicast verification fails with nodes sharing same name
IV55145ha712 incorr fallover during ha start of node at site with id 1
IV55146tivoli monitoring agent builder throws errrors with hacmp.my
IV55147recovery action should be checked in suspended state
IV55148ha clrginfo -m shows online app on remote node as offline
IV55149clver can core dump when ipv6 addresses configured
IV55150express sync ios can starve in a async glvm configuration
IV55151setgr add repository might fail if external type variable is
IV55152some clmgr commands might give a "parameter not set" error
IV55153lvm may mark asynchronous pv as missing due to io timeout
IV55154clmodnetwork core dumps on network name length
IV55155unexpected rg_move by errnotify with lvm_io_fail
IV55156no information on 'critical volume group' functionality
IV55157clmgr fails when monitor method script has arguments
IV55158ha svcpprc cgs are processed for rg move when not in the rg
IV55159after migration from 6.1 to 7.1.x, clstart may fail
IV55160clmgr might have missing information for repositories
IV55161clmgr does not display resource group secondary nodes
IV55162systemmirror for aix does not allow caa services management
IV55163ha smit cluster heartbeat settings wrong ranges in help text
IV55164the clmgr node and host queries do not indicate the local host
IV55165lvm + hyperswap unable to bring rdg online
IV55166actriveaddrnode mib var incorrect for dynamically added boot ips
IV55167clmgr may not display node priority policy values correctly
IV55168cannot change "notify_method" to empty value
IV55185"app monitor script not exist/executable" error at verification
IV55241cluster verification hangs in 1 cpu node
IV55564dlpar "not found" error during cod cpu processing
IV55937clstart may not work and node crash at assert
IV55938could not able to add existing ldap server to powerha
IV55939glvm ras enhancements
IV55940process monitor will permit more processes than instance count
IV55941powerha 'lazy update' not working for jfs2 with 'inline' log
IV55942memory leak in clevmgrdes
IV55943application monitor methods do not allow arguments / space
IV55944cannot use clmgr to set hyperswap recovery attribute.
IV55945swap fails for mgs
IV55953lslpp -lor gives wrong return code
IV55955timestamp out of sync issue still happens even iv41182 applied
IV56377change cluster.log to avoid refresh of syslogd
IV56576srdf emc 'options' file incorrectly modified by powerha
IV56798powerha may sometimes assign a wrong hostname to a nodename
IV57309clcycle of clinfo.log fails
IV57310systemmirror will not create linked cluster
IV57311clmgr replace repository fails after manual chrepos
IV57312a resource group containing a volume group might not fallover
IV57334extra warning reported for site-specific service ip label.
IV57395sap assist 'cleanipc' failure due to incorrect libpath
IV57728powerha sa for sap may fail to configure (a)scs instance
IV57742cldare fails fqdn caa nodenames vs. short ha communicat
IV57901cl_pvo fails to detect if ecm vg is online in non conc mode
IV57902cl_pvo fails to activate missing/removed disks
IV57903cl_pvo fails to handle lack of quorum
IV58550sap assist the start of the scs instance may fail randomly
IV58982physical partition size in megabytes limited to 1024 in cspoc
IV58983rework custom configuration path for cluster creation
IV58984configure split merge policies does not recognize cluster type
IV58985smit internal error for cm_configure_split_merge_lnk with ja_jp
IV58986powerha/svc metro mirror hacmp.out shows svc error: cmmvc5977e
IV58987clver does not detect when svc ip is defined as alias in aix odm
IV58989nfsd may not be restarted by cl_export_fs due to bad timing
IV58990clmgr cannot modify same-node/site resource group dependencies
IV58991clver does not detect netmask mismatch between aix and ha odm
IV58992incorrect build information from "clmgr query version"
IV58993adding a new node to an existing cluster and site may fail.
IV58994messages about address family when adding persistent labels
IV58996invalid character in expression when taking cluster snapshot
IV58997clstrmgr may core on startup of first node
IV58998clstrmgr exits if hacmp.out greater than 2gb
IV58999clmgr cluster reports may not display a company logo properly
IV59000discovery of nfs fails in case of multiple sids
IV59001rdgs move to halt, when primary storage is down
IV59002setting split merge poicies fails with parsing error
IV59005wrong values added for nfs_ip and nfs_mount for sap instance
IV59018smcaactrl emits error typeste: not found
IV59020spprc vg fence and varyonvg errors in hacmp.out at clstart
IV59519shutdown -f may reboot instead of halt
IV60097spelling mistake in "recommended user actions" for manual
IV60101misleading primary node for ers instance
IV60102discovery of tsm admin center v6.3.4 fails.
IV60103xiv clverify may fail for cg related errors
IV60104clmgr modify cluster nodes=... does not work
IV60105"clmgr add vg" will emit an error if a notify method is given
IV60106swap may fail
IV60107clrgmove may fail
IV60108lsrpvserver -a take loads of time to display result
IV60110start of second ers instance fails in case of resource sharing
IV60111rg move to err state due to syntax error
IV60112syncvg parallel with async io can cause io timeout
IV60113bring rg online fails for active active env
IV60114verify and sync failed with syntax error
IV60115initial cluster setup smit panel throws errors
IV60116single node swap default value is disable
IV60117clcycle uses hard coded path for cluster.log
IV60119clmgr replace repos fails for remote site in a linked cluster
IV60120non-english locales can cause clmgr query commands to hang
IV60451v&s fails for multiple vgs in a rg of a xiv cluster
IV60453io with b_option flag type rpv overlap starve for syncvg writes
Filesets with associated APARs
PTF#FilesetAssociated APARs
U864225cluster.adt.es.client.include7.1.3.1 IV60104
U864199cluster.es.assist.common7.1.3.1 IV55138
U864197cluster.es.assist.sap7.1.3.1 IV55103 IV55109 IV55140 IV57395 IV57728 IV58550 IV59000 IV59005 IV60101 IV60110
U864227cluster.es.assist.tsmadmin7.1.3.1 IV60102
U864181cluster.es.assist.tsmclient7.1.3.1 IV55113
U864182cluster.es.assist.wmq7.1.3.1 IV55121 IV55126
U864186cluster.es.client.clcomd7.1.3.1 IV55146 IV56798
U864187cluster.es.client.lib7.1.3.1 IV54588 IV55110 IV55120 IV55122 IV55123 IV55124 IV55125 IV55130 IV55133 IV55141 IV55142 IV55143 IV55151 IV55152 IV55154 IV55157 IV55160 IV55161 IV55162 IV55164 IV55167 IV55943 IV55944 IV57310 IV57311 IV57312 IV58993 IV58994 IV58999 IV60104 IV60105 IV60119
U864192cluster.es.client.rte7.1.3.1 IV53751 IV55159
U864188cluster.es.cspoc.cmds7.1.3.1 IV55097 IV55102 IV55162
U864195cluster.es.cspoc.rte7.1.3.1 IV55114 IV55938 IV58996
U864189cluster.es.genxd.cmds7.1.3.1 IV55128 IV55937 IV55945 IV59001 IV60103 IV60106 IV60113 IV60116 IV60451
U864223cluster.es.genxd.rte7.1.3.1 IV55937 IV60097
U864220cluster.es.server.diag7.1.3.1 IV55096 IV55099 IV55111 IV55115 IV55116 IV55130 IV55139 IV55144 IV55149 IV55168 IV55185 IV55241 IV57334 IV58987 IV58991
U864190cluster.es.server.events7.1.3.1 IV55105 IV55106 IV55110 IV55114 IV55120 IV55122 IV55136 IV55155 IV55162 IV55164 IV55165 IV55167 IV55564 IV55953 IV55955 IV57901 IV57902 IV57903 IV58989 IV58990 IV59020 IV60104 IV60107 IV60120
U864185cluster.es.server.rte7.1.3.1 IV55094 IV55095 IV55098 IV55102 IV55107 IV55110 IV55117 IV55118 IV55129 IV55134 IV55145 IV55148 IV55156 IV55163 IV55166 IV55940 IV55942 IV55943 IV56377 IV58982 IV58983 IV58984 IV58985 IV58997 IV58998 IV59002 IV60115
U864183cluster.es.server.utils7.1.3.1 IV55098 IV55101 IV55108 IV55124 IV55130 IV55131 IV55132 IV55161 IV55162 IV55941 IV55953 IV56377 IV57309 IV57310 IV57742 IV58992 IV58999 IV59018 IV59519 IV60104 IV60107 IV60117 IV60120
U864224cluster.es.sr.cmds7.1.3.1 IV56576
U864193cluster.es.sr.rte7.1.3.1 IV55147 IV56576 IV60111 IV60114
U864191cluster.es.svcpprc.rte7.1.3.1 IV55112 IV55135 IV55158 IV58986
U864184cluster.man.en_US.es.data7.1.3.1 IV55110 IV55122 IV55162
U864198cluster.msg.en_US.assist7.1.3.1 IV55109 IV60101
U864196cluster.msg.en_US.es.server7.1.3.1 IV55124 IV55133 IV55160 IV55161 IV55162 IV56377 IV57310 IV57311 IV58987 IV58991 IV58994 IV60104
U864226cluster.msg.en_US.sr7.1.3.1 IV56576
U864194cluster.msg.en_US.svcpprc7.1.3.1 IV55112 IV55135
U864228cluster.xd.glvm7.1.3.1 IV60107
U864221glvm.rpv.client7.1.3.1 IV55104 IV55150 IV55153 IV55939 IV60112 IV60453
U864222glvm.rpv.server7.1.3.1 IV55939 IV60108