Unless specifically noted otherwise, this history of problems fixed for IBM Spectrum Scale 4.2.x applies for all supported platforms.

Changes to this release of the IBM Spectrum Scale licensed program and the IBM Spectrum Scale library include the following:

Enhanced operating system support with spectrumscale installation toolkit
The spectrumscale installation toolkit now also supports the following operating systems: Red Hat Enterprise Linux 6.8 and SLES 12 on the Intel x86_64 architecture Red Hat Enterprise Linux 6.8 on the PPC64 architecture Red Hat Enterprise Linux 7.1 and 7.2 on the PPC64LE architecture

Enhanced Support for accessing data over SMB without setting the READ_ACL ("Read Permissions") bit on every file and directory
Files and folders that have not been accessible previously due to missing READ_ACL (gpfs) / "Read Permissions" (win) rights can now be accessed as long as the ACL is not accessed (given that the other access rights like READ grant access).

Extended status information
The subroutines gpfs_fstat_x() and gpfs_stat_x() provide extended status information. The subroutines directly return a gpfs_iattr64_t structure that contains many additional fields that are available only through this call, such as file creation time and file generation time.

GPFS log time stamp with time zone information
The default time stamp format for the GPFS log now includes time zone information and is similar to the ISO 8601 time stamp format. With the new format, you can convert times unambiguously to absolute times and you can sort and merge entries more easily. You can switch between the new log time stamp format and the previous format with the mmfsLogTimeStampISO8601 attribute of the mmchconfig command. You can also specify the log time stamp format for the entire cluster or for individual nodes. If you are migrating to v4.2.2, you can avoid automatically switching to the new time stamp format by specifying the mmfsLogTimeStampISO8601 parameter when you run the command mmchconfig release=LATEST.

Transparent cloud tiering
The following changes are made: Support for cloud data sharing. It allows data to be moved across disparate geographical locations and/or heterogeneous application platforms. Support for multiple node classes. You can enable and manage independent groups of Transparent cloud tiering nodes in different node classes for use with different network configurations per node class. Support for multiple file systems where one node class points to one file system. Support for the mmcloudmanifest tool, which you can use to parse the manifest file that contains the list of files exported to the cloud. Support for displaying the cloud service version installed on each node in a node class. Support for IBM Cloud Object Storage on IBM SoftLayer. Support for locally displaying the thumbnail of files without recalling them from the cloud storage tier. Support for configuring the number of days for which the deleted files are to be retained on the cloud.

IBM Spectrum Scale GUI changes
The following main changes are made in the IBM Spectrum Scale management GUI: Added new Home page. The Home provides an overall summary of the IBM Spectrum Scale system configuration and health status of its components and services that are hosted on it. Added new Files > Transparent Cloud Tiering page. The Transparent Cloud Tiering page provides both summarized and attribute-wise details of the Transparent Cloud Tiering service, which is integrated with the IBM Spectrum Scale system. Added new Storage > NSDs page. The NSDs page provides an easy way to monitor the performance, health status, and configuration aspects of the all network shared disks (NSD) that are available in the IBM Spectrum Scale cluster. File system detailed view is added in the Files > File Systems page. The detailed performance view helps to drill-down to various performance aspects. To access the detailed view, select the file system for which you need to view the performance details in the File Systems page and select View Details. Support for external keystone server object user authentication is added. You can now configure either an internal or external keystone server for authenticating the object users. The IBM Spectrum Scale system health component, mmhealth, replaces internal GUI health monitoring system. All the GUI panels and components that display the system health status are modified to display the health status reported by mmhealth. Introduced component-wise email notifications. The system administrator can now configure the email notifications in the Settings > Event Notifications page to send email notifications to the recipients, if events are reported in the following functional areas: Authentication Block and iSCSI services CES network Transparent Cloud Tiering NSD File system GPFS GUI Hadoop connector Keystone Network NFS Object Performance monitoring SMB Object authentication Node CES The Simple Network Management Protocol (SNMP) Management Information Base (MIB) is modified.

Improved performance monitoring using Grafana:
You can use the Grafana tool to analyze and display performance data. Grafana is an open source tool that uses a performance monitoring bridge to set up and populate graphs, which can be easily viewed and analyzed.

Logging file system activity using Varonis:
File system activities can now be logged using the Varonis DatAdvantage software.

Mixed operating systems support with spectrumscale installation toolkit
You can now use the spectrumscale installation toolkit to install GPFS and deploy protocols in a cluster that contains nodes that are running on different operating systems.

Network checking: mmnetverify command
With the mmnetverify command, you can verify the network configuration and operation of a group of nodes before you organize them into an IBM Spectrum Scale cluster. You can also run the command after you create a cluster to analyze network problems. Tests include address checks, ping tests, remote shell and file copy tests, time-date checks, TCP connection checks, message size tests, bandwidth tests, and flooding tests. The command prints full information and error logs about all the nodes tested.

New sensors added to the list of performance metrics:
The following three new sensors have been added to the list of performance metrics GPFSFileset GPFSPool GPFSWaiter

Quality of Service for I/O operations (QoS)
The mmchqos command has added capabilities. You can assign IOPS to individual nodes, to a node class, to a list of nodes in a text file, or to a remote cluster. You can also preserve and reuse your IOPS assignments by configuring them in a stanza file.

RDMA CM (Connection Manager) and RoCE improvements
You can now create multiple IP network interfaces per RDMA device and per VLAN. With RDMA over Converged Ethernet (RoCE), you can now create virtual IP network interfaces (such as eth0:1) and IP network interfaces with multiple IP addresses.

REST-style API for performing IBM Spectrum Scale tasks
The IBM Spectrum Scale REST API is an HTTP programming API for performing command-level IBM Spectrum Scale tasks. With the REST API, you can automate storage management operations and integrate IBM Spectrum Scale capabilities into your applications. The API can be installed on a single cluster node and requires an Apache server to be installed on the same node. It runs on HTTPS and uses JSON syntax to frame data inside HTTP requests and responses. In this release the API supports only high-prioriy operations, including operations on filesets, snapshots, and quotas and retrieving information about CES objects, file systems, and nodes.

Documented commands, structures, and subroutines
New commands The following commands are new: mmnetverify mmrest New structures There are no new structures. New subroutines The following subroutines are new: gpfs_fstat_x gpfs_stat_x Changed commands The following commands were changed: mmchconfig mmchqos Changed structures The following structures were changed: gpfs_iattr64_t Changed subroutines There are no changed subroutines. Deleted commands There are no deleted commands. Deleted structures There are no deleted structures. Deleted subroutines There are no deleted subroutines.

Object storage improvements
Added support for the object heatmap data tiering policy. Added support for secure communication between the proxy server and the other backend servers. Added support for enabling object access on existing filesets.

Problems fixed in IBM Spectrum Scale 4.2.1.2 [November 3, 2016]

Fix problem with choosing only fully cached files for Cache Eviction.

Fix a problem in which mmfsck ends prematurely with exit status 666:9:2.

Fix a mmcrsnapshot command hang. This would only happen if a snapshot exists before and after the inode space expanded on a problem file system or fileset.

Fix a deadlock that can occur because of a race condition between dmapi calls with file handles and a file deletion.

Fix a problem in which the node freezes and it is required to be restarted. This can occur when vfs hits it's open file limit found in /proc/sys/fs/file-max.

Fix an ioctl error that can occur when running the mmgetstate -a command.

Improve consistency check for CES-only related IP movements.

Fix a problem in which different up-calls are dropped.

Fix a faulty Events Exporter assertion. The faulty assertion could cause the mmfsd daemon to fail in rare instances while generating an internal dump or responding to certain "mmfsadm saferdump" commands.

Provide a new config option afmSyncAllXattrs to synchronize all xattrs.

Fix a "cryptographic library could not be initialized" error that can occur when changing a cipherList in a P8LE environment.

This fix improves GNR's error handling of pdisk faults that can occur after the fault-tolerance of the system is exceeded.

Fix the 'pStepReceiveP->phase == phase' assert that can occur when fsck is run in a tight loop.

When there are tons of users and groups intertwined with a lot of filesets with --perfileset-quota enabled this fix can improve the speed of mmcheckquota greatly.

Fix a problem in which an offline fsck overflowed a duplicate fragment buffer.

Fix a 'opP != __null' assert hit by offline fsck when handling a corrupt compressed files.

Fix a mmap read issue that can occur when getting stale data from the kernel page cache.

Fix a problem in which a NFS client gets an IO error. This can occur in a Ganesha environment.

Fix a problem in which fcntl lock waiters are not resumed when a linux holder downgrades.

Fix a problem in which 3524 storage enclosures are excluded when syncing storage enclosures. This fix applies to GSS/ESS customers.

Fix a problem in which mmrepquota shows the fileset id but does not show the name of the added fileset quota.

Fix a problem which is causing a disconnect between the AFM DR pairs. This can occur using the mmcesdr command after initial configuration has been completed on both clusters.

Fix a "[Errno 2] No such file or directory" error that shows up in the /var/adm/ras/mmsysmonitor.log when the sqlite3 and the tar executables do not exist in /usr/bin/tar and /usr/bin/sqlite3. This can cause the FILESYSTEM and DISK monitors of Sysmonitor to get into an UNKNOWN state.

Fix the exp(!ebP->replyDone) assert that can occur when DMAPI is being used.

Fix a problem in the AFM environment where replication would stop because of an error that might occur while replaying Rename operations. This can cause AFM filesets to go into a recovery and/or resync state.

Fix a problem in which IBM Spectrum Scale Object Protocols functionality is affected by security vulnerabilities in the TLS and SSL protocols (CVE-2015-2808 and CVE-2014-3566). This fix is applicable only if Object is enabled and SSL for keystone is configured.

Fix a slow down in offline fsck repair. This occurs when inodes in a fileset are not deleted when an afm fileset is converted to normal.

Fix a file system hang that occurs during mounting when migrating from an old version of GSS/ESS. This can occur when the logtip vdisk was created with a blocksize of 256k.

Fix a problem in a cluster configured with CCR, where a node which is rebooted may be unable to rejoin the cluster. After the reboot, files /var/mmfs/ccr/ccr.paxos.1 and /var/mmfs/ccr/ccr.paxos.2 may have a length of 0. The problem tends to happen in VMs. Error message "Not enough CCR quorum nodes available" (err 809) may be produced for most commands.

Fix offline fsck reporting false positive directory entry problems.

If some physical disks in GPFS Native RAID (GNR) suffer a large number timeouts (very slow IOs), they should be declared "slow", drained and scheduled for replacement. This defect may have prevented them from being declared slow, meaning they will continue to lower performance and risk outages.

Fix a deadlock that can occur when a pagepool is too small in a RDMA enabled environment.

Fix a problem in which a write fails on a file which does not have any remote Pcache EA's set. This will cause the fileset to need to be recovered later.

Fix a problem on LROC devices greater that 4TB in which the display of mmdiag lroc shows an incorrect Max Capacity value.

Fix a "FSErrBadDittoAddr" file system struct error that can occur on a clone child file.

Fix a problem in which extra snapshots are created. This can occur if an error 5 occurs while creating a snapshot at home.

Fix a gateway node crash that can occur if it accesses garbage content being passed in from gpfsReadAfmDRLastRPOSnapName.

Fix an error 124 which can occur if the control interface is not set up properly during DR setup.

Fix a setTime assert. This can occur if mmap is being heavily used.

Fix a problem in which trying to set the remote attributes fails. This can occur during a prefetch on an already fetched immutable/appendOnly file.

This update addresses the following APARs: IV89733 IV89807 IV89899 IV89900.

Problems fixed in IBM Spectrum Scale 4.2.1.1 [September 9, 2016]

Fix code for handling mmfsctl suspend-write operation to prevent blocking of close operation after reading file/directory. This issue only affect nodes running AIX.

Fix a GNR logging issue that may cause RG recovery failure and data lost.

Fix a SIGV problem when running mmdefrag on FPO file system with trace enabled.

Make GSKit the default encryption engine on s390.

Fix a bug in which start protocol tracing (with mmprotocoltrace) would fail on nodes where daemon node name and admin node name were not the same.

Fix a problem in which updating quota files will fail after running mmfsck.

Fix a problem in which GPFS may have incorrect contact nodes information in memory even though the correct contact nodes information was provided by the mmremotecluster add command. This could happen if the same cluster was previously added with incorrect contact nodes.

Fix a race condition that could cause incorrect file size being reported to an application. This only affect Linux and only when the application checks the file size right after increasing file size via writing past end of file.

This fix will make sure the NFSv3 files are closed when using Ganesha.

Resolve interdependencies of Mkdirs and Renames so as to avoid queue flush getting stuck.

Fix a deadlock that can occur during look up of a .snapshots/ directory of a fileset.

Fix a problem that causes mmsdrrestore on all CCR nodes to fail if the command is issued from a nonquorum node.

Make GPFS work on Intel Broadwell CPU with kernel 3.7 or later.

Workloads with heavy use of medium-sized writes (100K-500K) may see improved performance.

Fix Assert exp(threadP != __null) in file rbmd11628b/src/avs/fs/mmfs/ts/classes/tscomm/llcomm.C which can occur under heavy stress.

Fix a high contention in read code path which improves the performance when large number of theads/processes read the same block of a file concurrently.

Fix a GPFS daemon failure during QosIoMon::startManager - logAssertFailed - acquireMgrMutex. Apply this fix if you use QOS features.

Fix an issue in the AFM environment where fileset recovery causes gateway node to crash if recovery detects that the symlink have attribute(Example. chown -h user:name) changes.

Fix snapshot restore when building restore operation lists. This issue only happens when there are files with inode numbers bigger than the maximum value of 32 bits integer in the file system.

Fix a problem in which gpfs.snap prematurely exits if the GUI process is not running.

Fix the "Assert on Structure Error on FileMetadata::getSnapDataBlockDiskAddr" when reading a clone child file from a snapshot file system. This issue would only occur on a clone child file and only when the data of the clone child file is modified in the root file system after creating the snapshot.

Fix a signal 11 error that was caused by a race condition of mmadddisk and block allocation.

Fix an assert "fsdaP->getNValidAddrs() == nAllocated" which happens on a FPO file system while doing fragment block allocation.

Fix signal 11 caused by a very rare race condition between recovery group resign/relinquish and pdisk I/O activities.

Fix a bug in which mmgetstate reports the wrong status during heavy stress.

Fix a problem in which the mmsmb exportacl getid command failed to return the correct SID of a user or group.

Fix a GNR server crash that can occur during pdisk discovery.

Fix a deadlock that can occur during a heavy IO-load while using LROC devices while requesting pagepool buffer allocations.

This fix allows GPFS on Windows to fetch IMU/RFC2307 mappings from a single alternate trusted domain.

Fix a deadlock that can occur when accessing snapshot file data.

Fix a 760(E_CDITTO_LOCK) error which can occur on a punch hole operation on a clone child file.

Fix a not implemented error that can occur calling GPFS APIs on a system in which we have reached the maximum number of open files.

Fix a potential deadlock during a PIT repair/restripe run if a file is truncated when the new end of file is being repaired/restriped by the PIT process.

Fix an assert which happens when logtipback vdisk is created before loghome vdisk.

Fix a NT_STATUS_NONE_MAPPED error which can occur using the 'mmsmb export change' command when attempting to set the 'admin users' SMBOption.

The logic for preserving the state of world for expel data collectioniis adjusted to avoid causing mutex contention and andditional node expels in an unstable cluster.

Fix an assertion in GPFS send message when shutting down an invalid socket number.

Fix a deadlock that can occur when prefetch recovery is happening and applications are trying to access the files in the fileset.

mmchattr --delete-attr can remove gpfs.BGF, gpfs.WAD or gpfs.WADFG now.

Fix a problem when both maxblocsize and scatterBufferSize are equal. The daemon asserts when the buffer memory usage is high.

Fix an assert that is caused by a race between SGExceptionLocalPanicThread and repairFile.

Fix a problem in which SGDesc migrate code handles multiple disk failures incorrectly.

Fix information messages from mmapplypolicy which contain "hit" counts for EXCLUDE rules, which were incorrectly reported as 0 when -N and -g options are used.

Fix a segmentation fault that can occur running the mmsetquota command. This issue would only happen when GPFS overwrite tracing is enabled on Linux.

Fix a clone parent file restore failure. This issue would only happen when the clone files need to be restored and the clone parent files have "immutable" attribute in restoring snapshot.

Fix rename issues with Recovery for Diretories and files when the special block files inside a directory gets renamed.

Before linux-2.6.38, the implementation of fallocate(2) on gpfs needs an open instance which has delay close semantics. This may cause a timing window after the open instance is closed. read(2) operation cannot see up to date mmap modification until the timing window is gone (Assume no msync(2) is called). This fix closes the time window.

Fix output for mmgetstate with -Y flag so that it matches GPFS documentation.

GPFS commands may fail when public key expired. This fix re-generated the key.

Fix a possible segmentation fault in offline fsck during mutlipass dir scan. Multipass directory scan happens when there is insufficient memory to hold the fsck data structures required for the directory scan phase.

Fix resync thread not getting killed by deadlock detection code if it takes a long time to complete.

Fix a rare assert "isSGPanicked" in the Asynchronous DirectI/O code path.

Fixed an assert on when DBGASSERT is enabled and dm_get_dmattr is called on files in .snapshot directory.

Fix a ENOMEM failure in tslsenclslot. The error condition occurs under heavy loads only. This fix is required for all platforms.

Fix an accert that can occur on a GW node if the dr primary conversion is stopped, and the fileset is unlinked and AFM is disabled.

Fix errors that occur after a file set manager is moved from nodeA to nodeB and new filesets are created. The errors occur because nodeA can not see the new filesets.

Fix a problem that causes ACL inconsistency on the sgmgr node. This may cause an FSSTRUCT or an assert.

Fix code to prevent a potential GPFS daemon assert during file system restripe.

Fix an alloc segment steal problem that can lead to a "More than 22 minutes searching for a free buffer" assert.

This fix reduces the ThreadStateMutex hold time in deadlock detection and waiter related functions to make GPFS run more smoothly.

mmexportfs: Fix an issue where the file system exported but the output data file is missing.

Fix a minor problem by not sending a delete snapshot request to secondary.

Fix a problem with PCACHE_REMOVES on renamed NFS files that contain special characters in their name returning E_BUSY from home.

Fix the long waiter "InodeDeleteThread, 'waiting for XW lock'" caused by a self-deadlock issue, by correcting the release orders of GPFS file lock and Linux inode. This issue would see under Linux system only.

Fix an issue in AFM environment where prefetch recovery could deadlock with allready running management programs (ex. create snapshot).

Add an undocumented config parameter dataCollectionPendingDelay which controls how long we try to preserve the state of the world while collecting expel debug data. The code path is also optimized in general.

Fix allocation code to prevent an infinite loop during restripe that could cause long waiters. This could occur when disks are full in multiple failure groups.

Add a "flushOnClose" config parameter to force dirty data to be flushed to disk whenever a file is closed.

Fix revalidation issue with resync over gpfs backend.

Fix a problem when creating iSCSI volumes with more than 16 bytes name.

Fix a problem in which Ganesha fails to start when netgroup name starts with numeric character.

Fix a problem in which mmbackup --rebuild fails. This occurs on AIX only.

Fix an assert that can occur after a nfs error getting sparse info of a file using control interface.

Fix the "Node name is not valid" failure for mmrestorefs command.

Fix the way errorcodes are being decoded and converted to AFM range from GPFS or system range.

Fix an issue in AFM DR environment where DR failback fails to restore from the RPO snapshot. This happens when fileset restore tries to punch hole on a file and file was sparse in RPO snapshot.

Fix a bug where mmchcluster may fail when run on a non-configure server to disable CCR.

This fix will not show old open file or directory names that are renamed while they are open.

Fix a problem in which AFM does not honor async delay if softQMem thresold is set as 0. It plays ops to home immediately.

This update addresses the following APARs: IV85089 IV87625 IV87626 IV87627 IV87628 IV87629 IV87630 IV87631 IV87632 IV87633 IV87634 IV87635 IV87636 IV87637 IV87638 IV87639 IV87645 IV87809 IV87813 IV88298 IV88855.

Product/Component Name:	Platform:	Fix:
IBM Spectrum Scale	Linux 64-bit,x86_64 RHEL Linux 64-bit,x86_64 SLES	Spectrum_Scale_Protocols_Advanced-4.2.2.0-x86_64-Linux

Readme and Release notes for release 4.2.2.0 IBM Spectrum Scale 4.2.2.0 Spectrum_Scale_Protocols_Advanced-4.2.2.0-x86_64-Linux Readme