During internal testing, a rare but potentially serious problem has been discovered in GPFS. Under certain conditions, a read from a cached block in the GPFS pagepool may return incorrect data which is not detected by GPFS. The issue is corrected in GPFS 3.3.0.5 (APAR IZ70396). All prior versions of GPFS are affected.

The issue has been discovered during internal testing, where an MPI-IO application was employed to generate a synthetic workload. IBM is not aware of any occurrences of this issue in customer environments or under any other circumstances. Since the issue is specific to accessing cached data, it does not affect applications using DirectIO (the IO mechanism that bypasses file system cache, used primarily by databases, such as DB2� or Oracle).

This issue is limited to the following conditions:

The workload consists of a mixture of writes and reads, to file offsets that do not fall on the GPFS file system block boundaries;
The IO pattern is a mixture of sequential and random accesses to the same set of blocks, with the random accesses occurring on offsets not aligned on the file system block boundaries; and
The active set of data blocks is small enough to fit entirely in the GPFS pagepool.

The issue is caused by a race between an application IO thread doing a read from a partially filled block (such a block may be created by an earlier write to an odd offset within the block), and a GPFS prefetch thread trying to convert the same block into a fully filled one, by reading in the missing data, in anticipation of a future full-block read. Due to insufficient synchronization between the two threads, the application reader thread may read data that had been partially overwritten with the content found at a different offset within the same block. The issue is transient in nature: the next read from the same location will return correct data. The issue is limited to a single node; other nodes reading from the same file would be unaffected.

Complete these steps to install the fix package:

Unzip and extract the BFF image(s) from the *.tar.gz file:

gzip -d -c < filename >.tar.gz | tar -xvf -

Verify the update's BFF image(s) in the directory.

Normally, the BFF images in the directory would be similar to the following:
Unnnnnn.gpfs.base.bff Unnnnnn.gpfs.msg.en_US.bff Unnnnnn.gpfs.docs.data.bff where nnnnnn represents the six (6) digits of the PTF number for the BFF image.

For specific filenames, check the Readme for the GPFS update by clicking the "View" link for the update on the Download tab.

Follow the installation and migration instructions in your GPFS Concepts, Planning and Installation Guide.

In the below instructions, node-by-node upgrade cannot be used to migrate from GPFS 3.2 or 3.3 to later releases. For example, upgrading from 3.3.x.x to 3.5.y.y requires complete cluster shutdown, upgrade install on all nodes and then cluster startup.

Upgrading GPFS may be accomplished by either upgrading one node in the cluster at a time or by upgrading all nodes in the cluster at once. When upgrading GPFS one node at a time, the below steps are performed on each node in the cluster in a sequential manner. When upgrading the entire cluster at once, GPFS must be shutdown on all nodes in the cluster prior to upgrading.

When upgrading nodes one at a time, you may need to plan the order of nodes to upgrade. Verify that stopping each particular machine does not cause quorum to be lost or that an NSD server might be the last server for some disks. Upgrade the quorum and manager nodes first. When upgrading the quorum nodes, upgrade the cluster manager last to avoid unnecessary cluster failover and election of new cluster managers.

Prior to upgrading GPFS on a node, all applications that depend on GPFS (e.g. Oracle) must be stopped. Any GPFS file systems that are NFS exported must be unexported prior to unmounting GPFS file systems. If tracing was turned on, then tracing must be turned off before shutting down GPFS as well.
Stop GPFS on the node. Verify that the GPFS daemon has terminated and that the kernel extensions have been unloaded (mmfsenv -u). If the command mmfsenv -u reports that it cannot unload the kernel extensions because they are "busy", then the install can proceed, but the node must be rebooted after the install. By "busy" this means that some process has a "current directory" in some GPFS filesystem directory or has an open file descriptor. The freeware program lsof can identify the process and the process can then be killed. Retry mmfsenv -u and if that succeeds then a reboot of the node can be avoided.
Upgrade GPFS using the installp command or via SMIT on the node.

[July 12, 2012]

GPFS file systems created with versions earlier than 3.4 should not be migrated using the mmmigratefs command with the --fastea option until a fix is provided from IBM. IBM plans to make the fix available in GPFS versions 3.5.0.3 (APAR IV24151) and 3.4.0.15 (APAR IV24150). An ifix will also be available from IBM Service.

[January 24, 2011]

A fix introduced in GPFS 3.3.0-11 and in GPFS 3.4.0-3 changed the returned buffer size for file attributes to include additional available information, affecting the TSM incremental backup process due to the selection criteria used by TSM. As a result of this buffer size change, TSM incremental backup will treat all previously backed up files as modified, causing the dsmc incremental backup process to initiate new backups of all previously backed up files. If the file system being backed up is HSM managed, this new backup can result in recall of all files which have been previously backed up. This effect is limited to files backed up using TSM incremental backup; there are no known effects on files backed up using either GPFS mmbackup or the TSM selective backup process.

This issue is resolved in GPFS 3.3.0-12 (APAR IZ92779) and GPFS 3.4.0-4 (APAR IZ90535). Customers using the TSM Backup/Archive client to do incremental backup (via dsmc incremental command) should not apply GPFS 3.3.0-11 or GPFS 3.4.0-3, but should wait to apply GPFS 3.3.0-12 or GPFS 3.4.0-4. Any customer using TSM incremental backup and needing fixes in GPFS 3.3.0-11 or 3.4.0-3 should apply an ifix containing the corresponding APAR before executing dsmc incremental backup using these PTF levels, to avoid the additional file backup overhead, and (in the case of HSM-managed file systems) the potential for large scale recalls caused by the backup. Please contact IBM service to obtain the ifix, or to discuss your individual situation.

[October 26, 2010]

The restriction below is no longer in effect. GPFS file systems with file system format version less than 9.00 as reported by mmlsfs (V2.3 and older) can now be mounted on a GPFS V3.4 cluster safely.

[July 30, 2010]

Restrictions: File systems that currently have file system format version less than 9.00, as reported by mmlsfs (this format version corresponds to GPFS V2.3 and older) cannot be mounted on a GPFS V3.4 cluster. This restriction will be lifted in a future GPFS update.

[April 1, 2010]

Click here for details.

[November 9, 2009]

GPFS 3.3.0-1 does not correctly operate with file systems created with GPFS V2.2 (or older). Such file systems can be identified by running "mmlsfs all -u": if "no" is shown for any file system, this file system uses the old format, and the use of GPFS 3.3.0-1 is not possible. GPFS 3.3.0-2 corrects this issue.

The update images listed below and contained in the tar image with this README are maintenance packages for GPFS. The update images can be directly applied to your system.

The update images require a prior level of GPFS. Thus, the usefulness of this update is limited to installations that already have the GPFS product. Contact your IBM representative if you desire to purchase a fully installable product that does not require a prior level of GPFS.

After all BFFs are installed, you have successfully updated your GPFS product.

Update to Version:

3.5.0.2

Update from Version:

3.5.0.0 through 3.5.0.1

Update (tar file) contents:

README

changelog
U851760.gpfs.docs.data.bff
U852154.gpfs.base.bff
U852155.gpfs.msg.en_US.bff

Unless specifically noted otherwise, this history of problems fixed for GPFS 3.5.x applies for all supported platforms.

Problems fixed in GPFS 3.5.0.2 [May 30, 2012]

mmbackup will exit 1 when auditlog file is not available for result analysis after backup transaction is done.
Fix a problem stealing buffers in a large pagepool after installing 3.4.0.11.
When backup partially fail, mmbackup continues to compensate shadow file even thoughthere are multiple failed reported for the same file in auditlog file.
Fixed a bug in log recovery which could result in a "CmpMismatch" file system corruption problem.
Fix for the iP->i_count == 0 kernel assert in super.c. This problem onlyaffects Linux 2.6.36 and later.
Fix a rare deadlock where a kernel process gets blocked waiting for a free mailbox to send to the GPFS daemon.
mmbackup will exit 1 when incremental backup partially fail and shadow file compensation succeed.
Correct mmlsfileset output for junctions of deleted filesets in some cases.
Fix a memory allocation problem when online mmfsck runs on a node with a heavy mmap workload.
mmbackup will not stop processing even though there's no auditlog file if only expiration processing is done.
mmbackup will display progress msg "Expiring files..." correctly if expiration transaction takes longer than 30 mins.
Prevent the cluster manager from being expelled as a consequence of some communication outage with another node.
mmbackup with multiple TSM clients will catch all error messages from dsmc command output.
Fixes problem where the 'expelnode' callback indicates that the chosen node had joined the cluster first.
Fix a problem with nBytesNonStealable accounting.
Fixed message handler for filesystem quiesce which caused a GPFS assert when filesystem manager failed while filesystem is being quiesced.
Fix printing of long fileset names in mmrepquota and mmlsquota commands.
Fix mmap operations to go through nsd server when direct access to disks are no longer possible.
Fix mmsetquota to handle numerical fileset names.
mmbackup can backup files/directories with long pathname as long as GPFS and TSM support.
Fix an error message in mmchattr command with -M/R/m/r option.
Fix a problem that restripe failed in to an inifinite loop when sg panicked on the busy node.
mmbackup will display backup/expiration progress message in every interval specified by MMBACKUP_PROGRESS_INTERVAL environment variable if specified. Otherwise, mmbackup will display backup/expiration progress message every 30 mins.
Fixed rare assert when deleting files in a fileset.
Fixed rare hang problem during sg or token recovery.
Fix deadlock when doing inode scan (mmapplypolicy/mmbackup) in small pagepool.
getxattr for ACLs may ovewrite the kernel buffer if small buffer sizes (less than 8 bytes) are specified.
When mmbackup shadow file is rebuilt by --rebuild or -q option, mmbackup will get CTIME information from TSM server, hence files modified after previous backup but before shadow is rebuilt will be backed up by consequent incremental backup.
GNR: fix a problem where certain errors from a pdisk, like media errors, caused RecoveryGroup open to fail. Change code to continue attempting to open the RecoveryGroup and simply discount the pdisk(s) returning media errors(and unexpected error codes).
Prevent disks from being marked as 'down' when a node with the configuration option unmountOnDiskFail=yes receives an I/O error or loses connectivity to a disk.
When mmbackup can't backup files, the message is more informational.
Fixed mailbox calls which can lead to deadlock during filesystem quiesce. The deadlock is most likely to happen on a extremely overloaded system.
When backup fail (partially or entirely) due to error from TSM client, mmbackup will display error msg from TSM cleint for easy problem detection. But mmbackup will display the error msg only once for the same error even though multiple times occur.
Make GPFS more resilient to sporadic errors during disk access. Upon an unexpected error during disk open, such as ERROR_INSUFFICIENT_BUFFER, GPFS now retries the call after a brief pause.
When compesating shadow file takes long time because backup partially fail, mmbackup will show progress message.
Fixed the backward compatibility error in reading data across node on different versions. This is needed if you are upgrading from 3.4.0.6 or lower version number to 3.4.0.12 or higher GPFS version.
mmbackup Device -t incremental and --rebuild is valid syntax and will work properly.
Fix the problem that deldisk returned success even though if failed.
handleRevokeM loops when a lease callback is not responded to.
This update addresses the following APARs: IV19037 IV19165 IV20350 IV20610 IV20613 IV20615 IV20618 IV20619 IV20625 IV20627 IV20630 IV20634.

Product/Component Name:	Platform:	Fix:
General Parallel File System	AIX 6.1 AIX 7.1	GPFS-3.5.0.2-power-AIX

Readme and Release notes for release 3.5.0.2 General Parallel File System 3.5.0.2 GPFS-3.5.0.2-power-AIX Readme