Readme file for IBM® Spectrum Conductor 2.4.1 Interim Fix 559004
Readme
file for: IBM Spectrum Conductor
Product/Component
release:
2.4.1
Update name: Interim Fix 559004
Fix ID: sc-2.4.1-build559004
Publication date: September 28, 2020
This interim fix
resolves four issues:
· Too many removed executors causes Spark master out of memory issues.
· Submitting the spark-submit command (Spark batch application) using cluster mode with the REST API fails if the instance group is configured with the Enable SSL for Spark UIs only encryption option within the cluster management console.
· Executor fails to clean up the work directory, if there are subdirectories in the unpacked tar file.
· The cluster administrator user cannot access Spark service logs due to permission 600.
Note: The default number of retained finished executors is 1000. To change this number, specify --conf spark.ego.master.ui.retained.executor.num=number when submitting Spark applications.
2. Download location
3. Products or components affected
4. Installation and configuration
5. List of files
6. Product notifications
7. Copyright and trademark information
APAR: P103871
Download interim fix 559004 from the following location: http://www.ibm.com/eserver/support/fixes/.
Component name, Platform, Fix ID:
Spark 2.4.3, linux-x86_64 or linux-ppc64le, sc-2.4.1-build559004
1. Download the sc-2.4.1.0_build559004.tgz package to a
local directory on your computer. Decompress the file. Once decompressed, you
will have the following Spark package:
Spark2.4.3-Conductor2.4.1.tgz
2. Launch a browser and log in to the cluster management console as a cluster administrator.
3. Remove the existing Spark 2.4.3 package from the cluster:
a. Click Resources > Frameworks > Spark Management.
b. Select version 2.4.3.
c. Click Remove.
d. In the confirmation dialog, click Remove.
4. Add the new Spark 2.4.3 package to the cluster:
a. Click Resources > Frameworks > Spark Management.
b. Click Add.
c. Click Browse and select the Spark2.4.3-Conductor2.4.1.tgz package downloaded previously.
d. Click Add.
Configuring
Optionally, if you want to add group read permission to Spark service logs of an instance group, in the process of creating or modifying an instance group within the cluster management console:
1. In the Basic Settings tab under Spark version click Configuration.
2. Scroll down to the Additional Environment Variables section, click Add an Environment Variable, and add an environment variable with the name SPARK_EGO_LOG_GROUP_READABLE and with a value true.
3. Click Save.
Any user that should have read access to the Spark service logs should be either a member of the primary user group of the execution user of the instance group, or be a member of the user group that is provided as input in the parameter “Administrator user group” of the instance group.
Spark2.4.3-Conductor2.4.1.tgz
To receive information about product solution and patch updates automatically, subscribe to product notifications on the My Notifications page http://www.ibm.com/support/mynotifications/ on the IBM Support website (http://support.ibm.com). You can edit your subscription settings to choose the types of information you want to get notification about, for example, security bulletins, fixes, troubleshooting, and product enhancements or documentation changes.
© Copyright IBM Corporation 2020
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM®, the IBM logo and ibm.com® are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml