Unless specifically noted otherwise, this history of problems fixed for IBM Spectrum Scale 4.2.x applies for all supported platforms.

Changes to this release of the IBM Spectrum Scale licensed program and the IBM Spectrum Scale library include the following:

Addition of a new topic for AFM DR regarding failback of multiple filesets
A new topic was added that details a method for minimizing application downtime during the failback procedure.

Authentication considerations changes
The following changes are done: * Authentication support matrix has been divided to separate out the File and object protocols and accordingly, the corresponding explanation is modified. * The matrix is further divided based on the authentication service that is used. * A diagram is added to explain the high level flow of authentication for File protocols. * "Authentication for file access" topic is renamed to "Authentication and ID mapping for file access".

Directory preallocation
In environments in which many files are added to and removed from a directory in a short time, you can improve performance by setting the minimum compaction size of the directory. The minimum compaction size is the number of directory slots, including both full and empty slots, that a directory is allowed to retain when it is compacted.

Express Edition no longer available
IBM Spectrum Scale Express Edition is no longer available.

FPO enhancements
* Uses the mmrestripefile command to check whether the replicas of data blocks are matched for one file * Provides QoS support for autorecovery * Supports locality-aware data copy

Installation toolkit support for gpfs.adv and gpfs.crypto packages
The installation toolkit now supports installation, deployment, and upgrade of gpfs.adv and gpfs.crypto packages.

Installation toolkit support for populating cluster definition file
The installation toolkit now supports populating the cluster definition file with the current cluster state.

IBM Spectrum Scale GUI changes
The following main changes are made in the IBM Spectrum Scale GUI: * Supports mounting and unmounting of file systems on selected nodes or group of nodes using GUI. * Added new Storage > Pools page. The Pools page provides details about configuration, health, capacity, and performance aspects of storage pools. * Added new Files > Active File Management page. This new GUI page helps to view the configuration, health status, and performance of AFM, AFM DR, and gateway nodes. * Added new Monitoring > Tips page. The tip events give recommendations to the user to avoid certain issues that might occur in the future. A tip disappears from the GUI when the problem behind the tip event is resolved. * Added option to select events of type "tip" in the Settings > Event Notifications > Email Recipients page. You can configure whether to send email to the recipients if a tip event is reported in the system. * Added detailed view in the Files > Filesets page. You can access the detailed view of individual filesets either by double-clicking the individual filesets or by selecting View Details option. * Modified the Storage > NSDs page to list the rack, position, and node of the NSD in an FPO-enabled environment. This helps to sort the NSDs based on these parameters. The failure group definition is also modified to accommodate these new parameters. * Added the Customize the number of replicas option in the Files > Information Lifecycle page to specify the number of replicas in a file placement rule. * Modified the Settings > Event Notifications page to accept both IP address and host name for the email server. * Added Nodes and File Systems tabs in the detailed view that is available in the Files > Transparent Cloud Tiering page. * Added a separate Properties tab in the detailed view that is available in the Monitoring > Nodes, Files > File Systems, and Storage > NSDs pages.

Introduction of IBM Spectrum Scale management API Version 2
The architecture and syntax of IBM Spectrum Scale management API is changed. The new implementation is based on the GUI stack. The GUI server is managing and processing the API requests and commands. Version 2 has the following features: * Reuses the GUI deployment's backend infrastructure, which makes introduction of new API commands easier. * No separate configuration is required as the GUI installation takes care of the basic deployment. * Fixes scalability issues and introduces new features such as filter parameter, field parameter, and paging. * Supports large clusters with thousands of nodes. * All POST, PUT, and DELETE requests are completed asynchronously. A "jobs" object is created immediately when such a request is submitted. * The APIs are driven by the same WebSphere® server and object cache that is used by the IBM Spectrum Scale GUI. * The mmrest command is no longer required for configuring the management API. The IBM Spectrum Scale GUI installation and configuration takes care of the API infrastructure configuration. As the syntax and architecture are changed for API, modified the entire set of commands, which were available in the Version 1. New API commands are also added for improved flexibility.

Linux on z Systems™ enhancements
The following changes are made: * IBM Spectrum Scale for Linux on z Systems now supports Remote Cluster Mount (Multi-cluster). * SLES 12.2 and RHEL 7.3 are now supported by IBM Spectrum Scale for Linux on z Systems.

mmcallhome command: Addition of --long option to mmcallhome group list command
The --long option displays the long admin node names.

mmchconfig command: Setting an InfiniBand partition key
The --verbsRdmaPkey attribute specifies an InfiniBand partition key for a connection between a node and an InfiniBand server that is included in an InfiniBand partition.

mmdiag command: Status and queue statistics for NSD queues
The --nsd parameter displays the status and queue statistics for NSD queues.

mmfsck command: Severity of errors
The command displays a summary of the errors that were found that includes the severity of each error: CRITICAL, NONCRITICAL, or HARMLESS. You must specify the verbose or semi-verbose parameter to get this output.

mmhealth command: Addition of new options to command
Addition of AFM and THRESHOLD options to the mmhealth node show and mmhealth cluster show commands. The AFM option displays the health status of a gateway node or cluster. The THRESHOLD option monitors whether the node-related thresholds rules evaluation is running as expected, and if the health status has changed as a result of the threshold limits being crossed. Addition --clear option to the mmhealth node eventlog command. This option clears the event log's database. Addition of threshold add and threshold delete option to the mmhealth command. This option allows users to create and delete threshold rule. Addition of event hide, event unhide, and list hidden options to the mmhealth command. The event hide option hides the specified TIP events, while the event unhide option reveals all TIP events that were previously hidden. The list hidden option shows all the TIP events that are added to the list of hidden events. Addition of config interval option to the mmhealth command. The config interval option allows you to set the interval for monitoring the whole cluster.

mmkeyserv command: Updating a certificate or a connection
You can now get a fresh certificate from an Remote Key Management (RKM) server without rebuilding the connection. You can also temporarily update a connection by adding backup servers, reordering the list of backup servers, or changing the timeout, number of retries, or retry interval.

mmlslicense command: Displaying disk and cluster size information
You can now get information about disk and cluster size with the mmlslicense command.

mmnetverify command: Enhancements
Several enhancements increase the capabilities of the mmnetverify command. Network checks are added to measure the total bandwidth to check connectivity with the CTDB port, and to check connectivity with servers that are used with the Object protocol. If there are multiple local nodes, the command is run on all the local nodes in parallel. The lists of local nodes and target nodes accept node classes. The --ces-override parameter causes the command to consider all the nodes in the configuration to be CES-enabled.

mmrestripefile command: Fix inconsistencies between file data and replicas
The -c option compares the data of individual files with their replicas and attempts to fix any inconsistencies.

Monitoring of AFM and AFM DR
Using commands: * Functionality added to mmhealth,mmdiag, and mmperfmon * New parameters added to mmpmon. Using IBM Spectrum Scale GUI: * Added new Files > Active File Management page. This new GUI page helps to view the configuration, health status, and performance of AFM, AFM DR, and gateway nodes.

Mount options specific to IBM Spectrum Scale: syncnfs is now the default on Linux nodes
In the mount options specific to IBM Spectrum Scale, syncnfs is now the default on Linux nodes. On AIX® nodes, nosyncnfs is the default.

Protocol support on remotely mounted file systems
You can create an NFS/SMB export on a file system that is mounted from a remote cluster.

Questionnaire for an AFM DR deployment

Tip added to event status to inform users when a configuration is not optimal
A new event type TIP is added to system health monitoring. A Tip is similar to a state-changing event, but can be hidden by the user. Like state-changing events, a tip is removed automatically if the problem is resolved.

Quality of Service for I/O operations (QoS): Detailed statistics
You can now display more detailed statistics about IOPS rates for the QoS programs that are running on each node. The statistics are intended to be used as input for programs that analyze and display data.

Support for Samba 4.5

Transparent cloud tiering enhancements.
The following changes are done: * Support for configuring and deploying WORM solutions. Your files will remain WORM-compliant, both in the file system and on the cloud object storage. * Support for configuring Transparent cloud tiering with a proxy server. * Support for configuring cloud retention time, which overrides the default value. * Support for restoring only the file stubs from the cloud storage tier in situations where files are deleted from the local file system. * Substantial improvement in the performance when files are transparently recalled from the storage tier. * Support for manually deleting orphaned cloud objects before retention time expires. * Support for migrating files in the co-resident state, by which applications can directly access data without performing any recall operation.

-Y option
Added the -Y option to the following commands: * mmblock mmhealth mmlsfileset * mmcloudgateway mmkeyserv mmlsfs * mmdf mmlscluster mmlslicense * mmdiag mmlsconfig mmlsmgr mmlsquota * mmgetstate mmlsdisk mmlsmount mmlssnapshot * mmlsnodeclass mmnetverify * mmlsnsd mmnfs mmlspolicy * mmrepquota mmsmb mmuserauth

New commands
mmclidecode

Changed commands
* mmblock mmhealth mmlsfileset mmlsnodeclass * mmcloudgateway mmkeyserv mmlsfs mmlsnsd * mmdf mmlscluster mmlslicense mmlspolicy * mmdiag mmlsconfig mmlsmgr mmlsquota mmsmb * mmgetstate mmlsdisk mmlsmount mmlssnapshot * mmbackup mmcallhome mmcesdr mmchattr * mmchqos mmcrnsd mmfsck mmgetstate * mmimgbackup mmimgrestore mmkeyserv * spectrumscale mmnetverify mmrepquota * mmuserauth mmchconfig mmhadoopctl * mmprotocoltrace mmnfs

Changed structures
gpfs_iattr64_t

Changed subroutines
gpfs_prealloc

Deleted commands
mmrest

New messages
6027-1525, 6027-1756, 6027-2392, 6027-2393, 6027-2503, 6027-2504, and 6027-3258

Changed messages
6027-1023, 6027-1725

Problems fixed in IBM Spectrum Scale 4.2.2.3 [March 16, 2017]

Fix an issue where mmlsdisk may get segfault when it receives a SIGTERM.

Fix a gpfs cluster hang that can occur after a mmdiag --threads hangs.

Fix the no such file or directory error in gpfs_get_fssnaphandle on Linux.

Fix a problem in deleting AFM recovery snapshots that are leftover after using the mmdelsnapshot command.

Fix a rare case assert "Assert exp(e == E_OK)" which can happen while running the mmcrfs command.

Fix a 'incompleteOk' assertion that can occur during CCR enable.

Fix hang problems with reads over nfs target for AFM filesets.

Fix a problem in which FlushPending succeeds on a fileset which is in an unlinked state.

Fixed the slow system monitoring startup due to a deadlock caused by clearing the FAILED CES flag as a part of a failover upon the startup of system monitoring.

Fix a deadlock that can occur when changeSecondary of a primary fileset is in progress.

When a token manager fails or token manager list changes, GPFS will do token domain recovery. DOMAIN_RECOVERING status may cause token reset request being mishandled and leave a token in COPYSET status forever which makes subsequent requests on that token hang.

Fix a gpfs daemon crash that can occur while replaying operations to home with node failures at home.

Fix a GNR recovery group failure to recover due to too many unstable drives.

Fix a quorum loss that can occur when tiebreaker disks are not available during GPFS startup (CCR).

Fix a deadlock that can occur when trying to queue transfers to the old gateway that was serving the fileset, this can occur if the new gateway node for that fileset is running a recovery for that fileset.

Fix a deadlock in the AFM environment where adding a new gateway node using mmchnode could cause a deadlock when IO is happening.

Fix a problem where the fileset is left in an intermediate state after a changeSecondary failure.

Fix a rare race condition which can lead to assert (blockToExpand == ofP->metadata.getLastDataBlock()) in expandLastBlock.

Fix a no space error in FPO pool if the specified WADFG is invalid.

Fix a problem in which mmchqos -N node1 pool=system,other=200iops is applied to all nodes and not just node1.

Fix unexpected NFS errors that can occur durring snapshot deletion.

Fix a problem in which Ganesha operation numbers do not match GPFS operation numbers.

Fix a problem in the convertToPrimary command where if the command is run with the --secondary-snapname option, it reports that the fileset is left in primInitFail state.

Fix a problem in which mmrepquota reports unused quota entries when -v is not specified.

Fix a mmprotocoltrace crash that can occur on a cluster with an enabled sudo wrapper.

Fix a problem where GPFS did not interpret the return value from IOCINFO ioctl call on AIX correctly. That caused the wrong disk size to be reported during mmadddisk.

Fix a problem in which mmnfs change config did not accept negative values.

Fix a potential dirty read during small sequential read of a compressed file if the compressed file is overwritten from a different node while the small sequential read continues to retrieve obsolete data in a small side cache for decompressed data.

This bug only affects ESS and GSS systems. It shows up as a GPFS daemon failure due to a signal 11 in NotGlobalMutexClass::acquire, called from AUExtentGroup::lcCallback.

Fix a problem in which users were not able to use 'DEFAULT' as the value for a Ganesha configuration attribute.

Fix a problem in which "mmlsquota -g" fails to get gid. This can occur on linux or AIX. This can occur if there is a very long line in /etc/passwd or /etc/group.

Fix a kernel crash while gpfs handling OPENHANDLE_GET_VERIFIER op for Ganesha.

Fix a problem in which internal recovery snapshots could not be deleted.

Address a problem where fileset doesn't move out of disconnected state when the home has an Active NFS running.

Improve message and user action information when kernel NFS service is preventing CES NFS service startup.

Fix memory leak on client file read when checksums are enabled.

Fix err 124 while resync is run when fs version is lower than 4.1.1.

Fix code to avoid segmentation fault during purging CCR request queue.

Fix an issue where the CWD may appear in the prefix of device names in /proc/mounts and /etc/mtab.

Fix a problem in which AFM prefetch does not fetch empty directories.

Fix a problem in which CES can not assign public IP addresses to CES designated nodes.

Fix a problem in which a user could not use the '-' character in a netgroup name when attempting to add an export 'mmnfs export add'.

Fix a problem in which mmchcarrier --replace fails trying to update firmware. This fix applies to GSS/ESS customers.

Fix code to prevent potential kernel crash when performing read/write on very large file. This could occur when number of prefetched buffers goes over 32767.

Fix a problem in which GPFS is left in arbitrating state. This can occur if you configure a tiebreaker disk that not all quorum nodes have access to.

Fix kernel panic when GPFS Dead Man Switch timer expired in GSS and ESS configurations.

This update addresses the following APARs: IV93134 IV93163 IV93596 IV93597 IV93710 IV93711 IV93712 IV93713 IV93714 IV93715 IV93716 IV93813 IV93952 IV93953 IV93956 IV93958.

Problems fixed in IBM Spectrum Scale 4.2.2.2 [January 27, 2017]

Fix a mmfsd daemon crash for assert rmsgP != __null.

Fix a problem in which mmcrrecoverygroup fails when the admin and daemon networks are different. This fix applies to GSS/ESS customers.

Fix a signal 11 that can occur with active AFM filesets and heavy IO.

This fix provides AIX encryption performance improvements.

Fix a problem where the system monitoring framework was not reacting to configuration changes.

Fix a problem in which the admin network is not listed in the output of "mmhealth node show network".

Fix a problem in which gpfs_getacl returns ENOSPC. This can occur when the acl length exceeds the size of the buffer provided.

Fix a problem in which gpfs.service gets disabled and ccrmonitor gets killed after upgraded.

Fix a problem in which RemoteCmd gets stuck waiting for the pcacheListMutex.

Fix a problem in which "mmhealth cluster show" shows the wrong information for the components which were unhealthy and then became HEALTHY, while the corresponding services were STOPPED, DISABLED or SUSPENDED.

Fix a multipath device failure that reads "blk_cloned_rq_check_limits: over max size limit" which can occur when kernel function bio_get_nr_vecs() returns a value which is larger than the value of max sectors of the block device.

Fix a problem with CES node fail-over when the failing CES node stays powered off.

Fix a problem where error messages regarding cesiplist were shown in /var/adm/ras/mmsysmonitor.log on non CES-enabled clusters.

Fix a problem in which mmbackup falsely claims skipped files if multiple tasks combine outputs.

Fix an issue in the AFM environment where reads and evictions of a file can cause deadlocks.

mmperfmon query option -N now supports the following inputs: node number, node short name, node full name, node ip address.

Fix a 'isValidSocket(sock)' assertion that can occur when the incoming CCR request rate is much higher than the CCR server can handle.

This update addresses the following APARs: IV92103 IV92104 IV92105 IV92397 IV92398.

Problems fixed in IBM Spectrum Scale 4.2.2.1 [December 16, 2016]

CNFS: fix recursive calls during shutdown which may cause LOGASSERT.

Fix data corruption that can occur writing large files using parallel IO and multiple gateway nodes.

Fix a kernel crash during mmshutdown in kxAttachSharedMemory by tsctl process.

Fix "mutexMagic != 0x59C0DEAD" assert by the SGExceptionLocalPanicThread that can occur during quorum loss.

mmlsfileset: allow remote file system. Option -i and -d are now allow.

mmstartup: verify kernel/module configuration before start runmmfs.

Fix a very rare race condition that can lead to kernel panic. This issue could only occur on Linux when a mix of AIO and buffered IO are being used to read and write to the same file from multiple nodes.

Fix data mismatch problems and snapshot mismatches between old primary and acting primary during an applyUpdates in the failbackToPrimary procedure.

Fix a problem in which mmbackup returns the wrong number of objects handled. This can occur if NUMBERFORMAT is set incorrectly.

Fix Kernel BUG: illegal operation locks_wake_up_blocks+0x6c.

mmlsconfig: may not return correct value due to stale cache.

Fix code to prevent a possible kernel panic during GPFS daemon shutdown on Linux. In order for problem to occur, filesystem must be mounted with active workload when GPFS daemon is been shutdown.* Fix assert in the Gpfs daemon by disallowing GetAttr for DR Primary fileset.

Fix assert in the daemon that can occur during a Gateway queue transfer and new gateway node joins at the same time.

Fix slow offline fsck repair. Allow doDeferredDeletions() to cleanup afm pre-destroyed inodes in ex-afm filesets.

Fix a deadlock that can occur if a node of the cluster is down while GPFS attempts to establish a RDMA connection using RDMA CM.

Fix an issue in AFM environment where queue memory usage threshold (afmHardMemThreshold) is not honored.

Fix a remote error 17 creating and deleting hardlinks to files during log recovery.

Fix a mmsdrrestore command failure that can occur when a tiebreaker disk is in use (CCR setup).

This fix will make sure data is synced to disk only when required.

Fix mmfsd crash in an AFM environment that can occur when a lookup on the same file is triggered from multiple nodes.

Fix a debug assert in performPcacheMsg that can occur when the hardQMemLimit is hit for a fileset.

During IFS System upgrade, occasionally node crash is observed if AFM is configured and in use.

Fix Renames clogging the AFM queue during recovery for IW filesets.

Apply this fix if you are troubled by bogus messages in the mmfs log files during file system creation.

This modification does not change the functionality of GPFS, neither affects the appearance of the software, however it improves the effectiveness (speed) of the code in certain disk operations and introduces a mechanism (by explicitly distinguishing user- and kernel-space objects) that can be used for implementing other critical parts on s390x platform.

Fix a deadlock that can occur during high volume file creations and deletions in a multi cluster environment.

Fix mmlsquota -j returning a wrong answer that can occur if there is a special character in the name of the stripe group.

Handle upgrade of Windows node from 3.5 and 4.1 to 4.2 correctly.

Fix a problem in which mmces fails to change the address of object protocol attributes.

Honor parent folder's DELETE_CHILD right during rename operation from Windows node.

Fix a problem in which mmbackup with -B value > 32768 causes missed files.

Applications using dmapi for HSM may benefit by knowing if a file that is being migrated is currently open and being accessed on some node in the cluster. Extend the dmapi dm_get_fileattr command to take a new flag "DM_AT_FOPEN" to query if the file is open anywhere in the cluster.

Prevent the execution of mmhealth on unsupported nodes.

Change in dmapi dm_stat_t structure returned by dm_get_fileattr used by TCT. Update must be coordinated with TCT update.

This update addresses the following APARs: IV90831 IV90833 IV90864 IV90865 IV90869 IV90874 IV90878 IV90880 IV90891 IV90892 IV91054 IV91144 IV91145 IV91327 IV91443.

Product/Component Name:	Platform:	Fix:
IBM Spectrum Scale	Linux PPC64LE RHEL Linux PPC64LE SLES	Spectrum_Scale_Protocols_Advanced-4.2.3.0-ppc64LE-Linux

Readme and Release notes for release 4.2.3.0 IBM Spectrum Scale 4.2.3.0 Spectrum_Scale_Protocols_Advanced-4.2.3.0-ppc64LE-Linux Readme