Readme for IBM® Spectrum Symphony 7.1.2 Interim Fix 441444 

Readme file for: IBM Spectrum Symphony

Product/Component Release: 7.1.2

Update Name: Interim Fix 441444

Fix ID: sym-7.1.2-build441444

Publication date: February 28, 2017

This interim fix resolving multiple issues relating to Apache Ambari integration with IBM Spectrum Symphony 7.1.2 and provides the following fixes: 

·         SSL support for the cluster management console: When IBM Spectrum Symphony 7.1.2 is integrated with Apache Ambari, the cluster management console uses HTTP, instead of the default HTTPS. This interim fix provides a resolution for this issue.

·         Disables Spark on EGO for IOP 4.2 integration: The Spark on EGO framework in IOP 4.2 only supports the "ego-client" mode and is not compatible with other user applications (such as "yarn" or in "cluster" mode). This interim fix enables you to disable Spark on EGO integration with IBM Spectrum Symphony 7.1.2.

·         Enables logging in to the cluster management console from IOP 4.2 using LDAP accounts: With the default security plug-in, enabling the ENABLE_BI_AUTH parameter would list LDAP users but did not allow logging in to the cluster management console as a LDAP user. This interim fix provides authentication through the PAM+default plug-in, enabling logons to the cluster management console as a LDAP user. To submit MapReduce jobs without explicitly entering the username and password, this fix extends the ENABLE_BI_AUTH parameter to be used with the PAM+default plug-in.

·         Fixes submission user issues for Symphony MapReduce jobs: When IBM Spectrum Symphony 7.1.2 is integrated with Apache Ambari, the submission user for all Symphony MapReduce jobs was the built-in cluster administrator “Admin” account. This interim fix provides a resolution for this issue, so that the submission user is the correct OS user.

·         Fixes NullPointerException exceptions for Hive jobs: In some cases, Hive jobs failed with the following exception: MRServiceJava.MRService: Exception caught: null java.lang.NullPointerException. This interim fix provides a resolution for this issue. 

 

Contents

1.     List of fixes 

2.     Download location 

3.     Products or components affected

4.     Installation and configuration

5.     List of files

6.     Copyright and trademark information 

1.  List of fixes

APAR: P102094

2.  Download location

Download Fix 441444 from the following location: http://www.ibm.com/eserver/support/fixes/ 

3.  Components affected 

Component names, Platform, Fix ID:

Install / EGO / MapReduce, Linux 64-bit, sym-7.1.2-build441444

4.  Installation and configuration

System requirements

Linux x86_64

 

Prerequisite

The required integration scripts must be run for IBM Spectrum Symphony 7.1.2. 

 

Before installation 

1.     From the Ambari dashboard, stop all services.

2.     Back up the following files on all Symphony management hosts in your cluster:

$EGO_CONFDIR/../../gui/conf/server_gui.xml

$EGO_LIBDIR/sec_ego_pam_default.so

3.     Back up the following files on all Symphony management and compute hosts:

$SOAM_HOME/mapreduce/integration/7.1.2/IBM_BigInsights_4.2/ jar_integration.sh

$SOAM_HOME/mapreduce/integration/7.1.2/IBM_BigInsights_4.2/hadoop-mapreduce-client-core-plus-BI_4.2_IOP.jar

$SOAM_HOME/mapreduce/7.1.2/linux-x86_64/lib/hadoop-2.7.x/ hadoop-mapreduce-client-core-2.7.2.jar

$SOAM_HOME/mapreduce/conf/pmr-site.xml

$EGO_CONFDIR/ego.conf

$SOAM_HOME/mapreduce/integration/7.1.2/IBM_BigInsights_4.2/JavaSoamApi.jar

4.     Back up the following files on the Ambari server host:

/usr/lib/ambari-server/web/javascripts/app.js.gz

/var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/gui-integration.sh

/var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/integration_all.sh

/var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/kerberos_handling.sh

 

Installation

1.     Log on to the Ambari server host and decompress the tar.gz package:
> tar zxfo sym-7.1.2.0-iop4.2_x86_64_build441444.tar.gz -C /

2.     Log on to all Symphony hosts and decompress the tar.gz package:

> tar zxfo sym-7.1.2.0_x86_64_build441444.tar.gz -C $EGO_TOP

3.     Restart the Ambari server and check to confirm that the following Ambari cache files are synchronized on all Ambari hosts:

/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/gui-integration.sh

/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/integration_all.sh

/var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/kerberos_handling.sh

4.     To disable Spark on EGO from the IOP 4.2 integration, run the disable_spark_integration.sh script only on the Ambari server and select “y” to restart Ambari server host. For example:

              > cd /var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/tools

       > ./disable_spark_integration.sh -u admin -p admin -x 8080

Cluster: sym02 BigInsights 4.2

Cluster name: sym02, stack name: BigInsights, stack version: 4.2

***********************************************************************

Is Ambari User Name admin correct? (y/n)y

***********************************************************************

Is Ambari Cluster Name sym02 correct? (y/n)y

***********************************************************************

Is Ambari Web GUI Port 8080 correct? (y/n)y

Disabling Spark integration...

Disabling Spark successfully on host <testsym02.example.com>…

***********************************************************************

Restart Ambari now? (y/n)y

……

Spark integration disabled successfully.

5.     To log on to the cluster management console from IOP 4.2 using a LDAP account, complete these steps:

a.     On each management host, configure the following parameters for authentication through the PAM+default plug-in:

o    Modify the following parameters in $EGO_CONFDIR/ego.conf to use the PAM+default plug-in:

EGO_SEC_PLUGIN=sec_ego_pam_default

EGO_SEC_CONF=/opt/EGOShare/kernel/conf

NOTE: Set EGO_SEC_CONF to the value of $EGO_CONFDIR. If ENABLE_BI_AUTH is configured in EGO_SEC_CONF (for example, EGO_SEC_CONF=/opt/ego/kernel/conf,, ENABLE_BI_AUTH), you must manually remove “ENABLE_BI_AUTH”. 

o    Enable the ENABLE_BI_AUTH parameter in $EGO_CONFDIR/pamauth.conf to enable user validation for the PAM+default plug-in:

ENABLE_BI_AUTH=Y

b.    To log on as LDAP users from compute hosts, you must configure the EGO_SEC_PLUGIN parameter in $EGO_CONFDIR/ego.conf to enable the PAM+default client plugin for those compute hosts:

EGO_SEC_PLUGIN=sec_ego_ext_co

NOTE: If ENABLE_BI_AUTH is configured in EGO_SEC_CONF (for example, EGO_SEC_CONF=/opt/ego/kernel/conf,, ENABLE_BI_AUTH), you must manually remove “ENABLE_BI_AUTH”. 

6.     Complete the following steps to enable the cluster management console over HTTP or HTTPS:

o    To enable SSL, copy the SSL settings from server_gui.xml.ssl to server_gui.xml on Symphony management hosts under the $EGO_CONFDIR/../../gui/conf/ directory.

o    To disable SSL, copy the non-SSL settings from server_gui.xml.nossl to server_gui.xml on Symphony management hosts under $EGO_CONFDIR/../../gui/conf/ directory.

7.     Start the IBM Spectrum Symphony service stack from the Ambari dashboard.

8.     Select the IBM Spectrum Symphony Services page. In the Service Actions list, click Integration. 

All Symphony MapReduce jobs now use the OS user as the submission user. You can also successfully run Hive jobs.

9.     Run the GUI integration script as "root" if a Symphony node is not installed on the Ambari server host.

> cd /var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/

> ./gui-integration.sh -r <sym_gui_host_name> [-u <ambari_agent_user>]

 

After installation

1.     Open your browser and clear the browser cache. 

2.     Log on to the Ambari server as the Ambari server user account admin.

3.     Start the IBM Spectrum Symphony service stack from the Ambari dashboard.

4.     Check that the cluster management console:

a.       On the Ambari console. select the Services tab on the navigation bar. In the list of services, select Symphony. From the Quick Links list, select Symphony UI. It should locate to the Symphony Web Console.

b.       On the Ambari console. select the Services tab on the navigation bar. In the list of services, select MapReduce2. From the Quick Links list, select JobHistory UI. It should locate to the MapReduce Job List Page. 

5.     Check that Spark on EGO integration is disabled:

a.       Start the Open Source Spark service from the Ambari dashboard.

b.       Check to confirm that the following note does not show in the Spark service summary: 

Spark is running on Symphony Resource Manager.

 

Uninstallation

1.     Shut down the all services stack from the Ambari dashboard.

2.     Restore the following files on all Symphony management hosts with the backed-up file:

$EGO_CONFDIR/../../gui/conf/server_gui.xml

$EGO_LIBDIR/sec_ego_pam_default.so

3.     Restore the backup of the following files on all Symphony management and compute hosts:

$SOAM_HOME/mapreduce/integration/7.1.2/IBM_BigInsights_4.2/ jar_integration.sh

$SOAM_HOME/mapreduce/integration/7.1.2/IBM_BigInsights_4.2/hadoop-mapreduce-client-core-plus-BI_4.2_IOP.jar

$SOAM_HOME/mapreduce/7.1.2/linux-x86_64/lib/hadoop-2.7.x/ hadoop-mapreduce-client-core-2.7.2.jar

$SOAM_HOME/mapreduce/conf/pmr-site.xml

$EGO_CONFDIR/ego.conf

$SOAM_HOME/mapreduce/integration/7.1.2/IBM_BigInsights_4.2/JavaSoamApi.jar

4.     Restore the backup of the following files on the Ambari server host:

/usr/lib/ambari-server/web/javascripts/app.js.gz

/var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/gui-integration.sh

/var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/integration_all.sh

/var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/kerberos_handling.sh

5.     Open your browser and clear the browser cache.

6.     Start the Ambari server and check to confirm that the following files are synchronized with the Ambari server files on all Ambari hosts:

/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/gui-integration.sh

/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/integration_all.sh

/var/lib/ambari-server/resources/stacks/BigInsights/4.2/services/SYMPHONY/package/scripts/integration/kerberos_handling.sh

7.     Complete the following steps to enable Spark on EGO integration:

a.       Start IBM Spectrum Symphony service stack from the Ambari dashboard.

b.       Select the IBM Spectrum Symphony Services page; in the Service Actions list, click Integration.

c.       Check to confirm that the following note shows in the Spark service summary:

Spark is running on Symphony Resource Manager.

 

5.  List of files 

disable_spark_integration.sh

gui-integration.sh

integration_all.sh

sec_ego_pam_default.so

kerberos_handling.sh

jar_integration.sh

hadoop-mapreduce-client-core-plus-BI_4.2_IOP.jar

hadoop-mapreduce-client-core-2.7.2.jar

JavaSoamApi.jar

 

6.  Copyright and trademark information

© Copyright IBM Corporation 2017

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM®, the IBM logo, and ibm.com® are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.