Readme file for IBM® Spectrum Conductor 2.4.1 RFE 148714
Readme file for: IBM Spectrum Conductor
Product/Component release: 2.4.1
Fix ID: sc-2.4.1-build600535-boa
Publication date: July 5, 2021
Enhancement so that if the Spark shuffle service is blocked on a certain host, then the system should also block the executor and driver on that host, so they are not scheduled on host and fail.
1. Download location
2. Product and components affected
3. Installation and configuration
4. List of files
5. Product notifications
6. Copyright and trademark information
Download interim fix 600535 from the following location: http://www.ibm.com/eserver/support/fixes/
Component name, Platform, Fix ID:
Spark 2.4.3, linux-x86_64 or linux-ppc64le, sc-2.4.1-build600535-boa
Installation
Before you begin, IBM Spectrum Conductor 2.4.1 must be installed on an IBM Spectrum Conductor 2.4.1 supported operating system.
1. Download the sc-2.4.1.0_build600535.tgz package to a local directory on your computer and decompress the file. Once decompressed, you will have the following Spark packages:
Spark2.4.3-Conductor2.4.1.tgz
Spark3.0.1-Conductor2.4.1.tgz
2. Launch a browser and log in to the cluster management console as a cluster administrator.
3. Remove the existing Spark package from the cluster. For example, to remove the existing Spark 2.4.3 package:
a. Click Resources > Frameworks > Spark Management.
b. Select version 2.4.3.
c. Click Remove.
d. In the confirmation dialog, click Remove.
4. Add the new Spark package to the cluster. For example, to add the new Spark 2.4.3 package:
a. Click Resources > Frameworks > Spark Management.
b. Click Add.
c. Click Browse and select the Spark2.4.3-Conductor2.4.1.tgz package downloaded previously.
d. Click Add.
5. Create a new instance group that uses the new Spark package. For details, see https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/developing_instances/instance_create_about.html.
6. If required, upgrade your existing instance groups to use the new Spark package. For details, see https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/managing_instances/instance_update_spark_version.html.
Configuration
After updating to the new Spark package, enable the Spark shuffle service enhancement by configuring the Spark parameter spark.ego.driver.blockhost.supervise in the cluster management console:
1. Click Workload > Instance Groups. Select your instance group and click Configure.
2. In the Basic Settings tab, go to the Spark version section, and click Configuration.
3. Scroll down to the Additional Parameters section, click Add a Parameter, to add a new parameter with these settings:
· Name: spark.ego.driver.blockhost.supervise
· Value: true
4. Click Save.
Spark2.4.3-Conductor2.4.1.tgz
Spark3.0.1-Conductor2.4.1.tgz
To receive information about product solution and patch updates automatically, subscribe to product notifications on the My Notifications page http://www.ibm.com/support/mynotifications/ on the IBM Support website (http://support.ibm.com). You can edit your subscription settings to choose the types of information you want to get notification about, for example, security bulletins, fixes, troubleshooting, and product enhancements or documentation changes.
© Copyright IBM Corporation 2021
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM®, the IBM logo and ibm.com® are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml