Readme File for IBM® Spectrum Conductor 2.4.1 Interim Fix 544525

Readme file for: IBM Spectrum Conductor

Product release: 2.4.1

Update name: Interim Fix 544525

Fix ID: sc-2.4.1-build544525-welfg
Publication date: April 13, 2020

 

This interim fix enhances the existing data loader to collect Spark application resource usage metrics, and index into a new Elasticsearch index called ibm-cws-reporting-spark-application-metric-<yyyy.mm.dd>.

 

1.      Scope

2.      Installation

3.      Uninstallation

4.      Product notifications

5.      Copyright and trademark information

1.   Scope

Before you install this enhancement to your cluster, note the following requirements:

Architecture

x86_64, ppc64le

Product version

IBM Spectrum Conductor 2.4.1

2.   .Installation

Follow these instructions to download and install this enhancement to your cluster.

Prerequisites

o   IBM Spectrum Conductor 2.4.1 must be installed in your cluster.

Package

Name

Description

conductorsparkmgmt-2.4.1.0_noarch_build544525.tar.gz

GUI package for this enhancement.

conductorsparkcore-2.4.1.0_architecture_build544525.tar.gz

Core package for this enhancement.

Installation

a.      Log on to the management host as the cluster administrator and source the environment:

(BASH) . $EGO_TOP/profile.platform

(CSH) source $EGO_TOP/cshrc.platform

b.      Stop the plc service:

$ egosh service stop plc

c.      Stop Elasticsearch services:

$ egosh service stop elk-elasticsearch-master elk-elasticsearch elk-elasticsearch-data

d.      On each of your management hosts (or just one management host if you are using a shared file system):

i.        Download the conductorsparkmgmt-2.4.1.0_noarch_build544525.tar.gz and conductorsparkcore-2.4.1.0_architecture_build544525.tar.gz packages (for example, to a /conductorfixes directory).

ii.      Install the conductorsparkmgmt-2.4.1.0_noarch_build544525.tar.gz package:

$ egoinstallfixes /conductorfixes/conductorsparkmgmt-2.4.1.0_noarch_build544525.tar.gz

Note: Running the egoinstallfixes command automatically backs up the current binary files to a fix backup directory for recovery purposes. Do not delete this backup directory; you will need it if you want to recover the original files.

iii.     Install the conductorsparkcore-2.4.1.0_architecture_build544525.tar.gz package:

$ egoinstallfixes /conductorfixes/conductorsparkcore-2.4.1.0_architecture_build544525.tar.gz

Note: Running the egoinstallfixes command automatically backs up the current binary files to a fix backup directory for recovery purposes. Do not delete this backup directory; you will need it if you want to recover the original files.

iv.     Run the pversions command to verify the installation:

$ pversions -b 544525

e.      If you configured mghost (shared location), copy the template to the $EGO_CONFDIR directory:

$ cp $EGO_TOP/integration/elk/init/template/ibm-cws-reporting-spark-application-metric.json $EGO_CONFDIR/../../integration/elk/init/template/ibm-cws-reporting-spark-application-metric.json

f.       Configure the duration, in days, to purge ibm-cws-reporting-spark-application-metric-* indices in the $EGO_CONFDIR/../../integration/elk/conf/indexcleanup/cws.conf file.

For example, to purge an index older than 30 days, append the following line to the $EGO_CONFDIR/../../integration/elk/conf/indexcleanup/cws.conf file:

ibm-cws-reporting-spark-application-metric-* 30

g.      Start Elasticsearch services:

$ egosh service start elk-elasticsearch-master elk-elasticsearch elk-elasticsearch-data

Note: Ensure that all Elasticsearch services are started before proceeding to the next step.

h.      Verify that the ibm-cws-reporting-spark-application-metric-* mapping created successfully:

                             i.          Run the following command:

$ curl -k -u <id>:<pw> -H'Content-Type: application/json' -XGET <ES_master_URL>/_template/ibm-cws-reporting-spark-application-metric --tlsv1.2

where <id> and <pw> are the cluster ID and password, and <ES_master_URL> is the Elasticsearch master URL

                            ii.          Verify that the command returns the following information:

{"ibm-cws-reporting-spark-application-metric":{"order":1,"index_patterns":["ibm-cws-reporting-spark-application-metric-*"],"settings":{"index":{"number_of_shards":"1","number_of_replicas":"1"}},"mappings":{"dynamic":"strict","_source":{"enabled":true},"properties":{"InstanceGroupConsumerShareRatio_Driver":{"type":"long"},"InstanceGroupConsumerParentShareRatio_Executor":{"type":"long"},"InstanceGroupConsumerParentName":{"type":"keyword"},"ApplicationName":{"type":"keyword"},"UserName":{"type":"keyword"},"EndTime":{"format":"strict_date_time","type":"date"},"InstanceGroupUUID":{"type":"keyword"},"InstanceGroupResourceGroup_Driver":{"type":"keyword"},"InstanceGroupConsumerShareRatio_Executor":{"type":"long"},"Priority":{"type":"long"},"ApplicationState":{"type":"keyword"},"InstanceGroupName":{"type":"keyword"},"StartTime":{"format":"strict_date_time","type":"date"},"SlotsUsed_CPU":{"type":"double"},"InstanceGroupConsumerParentShareRatio_Driver":{"type":"long"},"ApplicationType":{"type":"keyword"},"ApplicationUrl":{"type":"keyword"},"@timestamp":{"format":"strict_date_time","type":"date"},"InstanceGroupResourceGroup_Executor":{"type":"keyword"},"CoresUsed":{"type":"double"},"ApplicationID":{"type":"keyword"},"MemoryUsed_CPU":{"type":"double"}}},"aliases":{}}}

It is important to verify "order":1 and "_source":{"enabled":true.

If the command does not return the exact output above, run the following the commands   to delete the mapping and indices (if any were created):

$ curl -k -u <id>:<pw> -H'Content-Type: application/json' -XDELETE <ES_master_URL>/_template/ibm-cws-reporting-spark-application-metric --tlsv1.2

$ curl -k -u <id>:<pw> -H'Content-Type: application/json' -XDELETE <ES_master_URL>/ibm-cws-reporting-spark-application-metric-* --tlsv1.2

Then, restart the Elasticsearch services, and repeat step h to verify that the ibm-cws-reporting-spark-application-metric-* mapping created successfully.

i.        Start the plc service:

$ egosh service start plc

3.   Uninstallation

If required, follow these instructions to uninstall this enhancement from your cluster:

a.       Log on to each management host as the cluster administrator and source the environment:

(BASH) . $EGO_TOP/profile.platform

(CSH) source $EGO_TOP/cshrc.platform

b.      Log on to each host in the cluster and roll back this fix:

$ egoinstallfixes -r 544525

c.      If you configured mghost (shared location), remove the template from $EGO_CONFDIR:

$ rm $EGO_CONFDIR/../../integration/elk/init/template/ibm-cws-reporting-spark-application-metric.json

4.   Product notifications

To receive information about product solution and patch updates automatically, subscribe to product notifications on the My notifications page http://www.ibm.com/support/mynotifications/ on the IBM Support website (http://support.ibm.com). You can edit your subscription settings to choose the types of information you want to get notification about, for example, security bulletins, fixes, troubleshooting, and product enhancements or documentation changes.

5.   Copyright and trademark information

© Copyright IBM Corporation 2020

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM®, the IBM logo and ibm.com® are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.