Readme File for IBM Spectrum Symphony 7.3.2 Interim Fix 601026

Readme File for IBM^® Spectrum Symphony 7.3.2 Interim Fix 601026

Readme File for: IBM Spectrum Symphony

Product Release: 7.3.2

Update Name: Interim Fix 601026

Fix ID: sym-7.3.2-build601026-cs

Publication Date: May 18, 2022

This interim fix is a cumulation of several fixes for IBM Spectrum Symphony 7.3.2 on Windows:

1. Support after installing interim fix sym-7.3-build600835 in an IBM Spectrum Symphony cluster, where an end user who does not have EGO_ACTIVITY_VIEW permission on the root consumer is unable to query the activities for which the user has permission. (Original Fix ID: sym-7.3-build600868)

2. Allowing a service instance (SI) from remaining in suspended state, after enabling memory limitations in an IBM Spectrum Symphony cluster. (Original Fix ID: sym-7.3-build600824-cs)

3. Allowing the CPU limit to be specified in hyper-threads, instead of cores, for IBM Spectrum Symphony. (Original Fix ID: sym-7.3-build600707-cs, sym-7.3-build600818-cs)

4. Eliminating an SSM memory leak issue in IBM Spectrum Symphony. (Original Fix ID: sym-7.3-build600795)

5. Support for a VEMKD performance log mismatch issue when using a VEMKD binary built later than July 27, 2021, in an IBM Spectrum Symphony. (Original Fix ID: sym-7.3-build600723)

6. Improvements to VEMKD’s performance when processing the VEM_HOST_INFO request, to avoid VEMKD from blocking the handling of the request during the VEMKD recovery stage in an IBM Spectrum Symphony cluster. This fix includes a new parameter called EGO_OPTIMIZE_HOST_INFO, to further improve the VEMKD performance during the recovery stage. (Original Fix ID: sym-7.3-build600184-cs)

7. Support to prevent VEMKD from crashing when adding new dimensions into an existing multidimensional scheduling resource plan. (Original Fix ID: sym-7.3-build600730)

8. Allowing the IBM Spectrum Symphony SSL client to load certificate authority (CA) certificates from the Windows certificate store (rather than from disk). (Original Fix ID: sym-7.3-build600667-cs)

9. Upgrading from IBM Spectrum Symphony 7.1.2 without encountering VEMKD ExceptionCode: c0000005. (Original Fix ID: sym-7.3-build546291-cs)

10. An increase to the maximum length for a client name, used by SD or SSM when either of these registers a client, from 256 to 1024 characters in an IBM Spectrum Symphony cluster. (Original Fix ID: sym-7.3-build546154)

11. Support to set limits on resource usage (CPU or virtual memory) for multidimensional scheduling for all service instances (SIs) in an application, or for a single SI on a host, based on multidimensional slot definitions. (Original Fix ID: sym-7.3.0-build544742-cs)

Contents

1. List of fixes

2. Download location

3. Product and components affected

4. Installation and configuration

5. Uninstallation

6. List of files

7. Product notifications

8. Copyright and trademark information

1. List of fixes

APAR: P104580

2. Download location

Download interim fix 601026 from the following location: https://www.ibm.com/eserver/support/fixes/

3. Product and components affected

Component name, Platform, Fix ID:

EGO/SOAM, Windows 64, sym-7.3.2-build601026-cs

4. Installation and configuration

Follow the instructions in this section to download and install this interim fix on your cluster.

System requirements

Windows 64-bit

Installing on IBM Spectrum Symphony management and compute hosts

a. Log on to the primary host as the cluster administrator, disable all applications, and shut down all primary and primary candidate hosts:

> egosh user logon –u Admin –x Admin

> soamcontrol app disable all

> egosh service stop all

> egosh ego shutdown hostname

b. On each primary candidate host (or just one host if you are using a shared file system), copy the sym-7.3.2.0-egocore-4.0.0.0_build601026.msp and sym-7.3.2.0-soamcore-7.3.2.0_build601026.msp package, and complete one of the following steps:

o To perform an interactive installation, double-click the .msp package and follow the prompts.

o To perform a silent installation from the IBM Spectrum Symphony command line, enter the following command:

C:\>msiexec /update C:\sym-7.3.2.0-egocore-4.0.0.0_build601026.msp /l*v updateSym.log /norestart /quiet REINSTALLMODE=omus

C:\>msiexec /update C:\ sym-7.3.2.0-soamcore-7.3.2.0_build601026.msp /l*v updateSym.log /norestart /quiet REINSTALLMODE=omus

The command syntax is as follows:

C:\>msiexec /update sym_package_name_path /l*v sym_install_log /norestart /quiet REINSTALLMODE=omus

where:

o sym_package_name_path is the fully qualified path to the .msp package.

o sym_install_log is the log file for the upgrade.

c. Log on to the primary host as the cluster administrator, start up all primary and primary candidate hosts, and enable your applications:

> egosh ego start hostname

> soamcontrol app enable application_name

Installing on IBM Spectrum Symphony client hosts

a. Log on to the IBM Spectrum Symphony client host and stop the running client.

b. On each IBM Spectrum Symphony client host, download the symclnt-7.3.2.0_build601026.msp package to a temporary location on the host and install the package:

o For an interactive installation, double-click the symclnt-7.3.0.0_build600667.msp package and follow the prompts.

o For a silent installation, enter the following command from the command prompt:

C:\>msiexec /update C:\symclnt-7.3.2.0_build601026.msp /l*v install.log /norestart /quiet REINSTALLMODE=omus

The command syntax is as follows:

C:\>msiexec /update sym_package_name_path /l*v install.log /norestart /quiet REINSTALLMODE=omus

Where:

o sym_package_name_path is the fully qualified path to the .msp package; in this case, C:\symclnt-7.3.2.0_build601026.msp.

o install.log is the log file for the fix.

5. Configuration

a. After applying this fix, to enable the CPU limit in hyper-threads instead of cores:

1) Set the SOAM_ENABLE_HYPERTHREAD_CPU_LIMIT environment variable for the session manager in the application profile:

<env name="SOAM_ENABLE_HYPERTHREAD_CPU_LIMIT">true</env>

</osType>

</osTypes>

2) Configure the CPU limit in hyper-threads, under the service/osTypes/osType section:

</resourceUsage>

3) Register the updated application profile:

> soamreg application_profile.xml

Once enabled, the following ssm.log message will be logged at the INFO level:
SOAM_ENABLE_HYPERTHREAD_CPU_LIMIT is enabled. The CPU fencing limit must be defined in CPU threads.

b. Set limits on resource usage (CPU or virtual memory) in multidimensional scheduling for all service instances (SIs) in an application, or for a single service instance on a host, based on multidimensional slot definitions.

Types of resource usage limits for multidimensional scheduling

With this enhancement, you can set two types of multidimensional scheduling resource usage limits: limit CPU usage and limit virtual memory usage.

· Limit CPU usage:

An IBM Spectrum Symphony service consumes CPU time on a host. You can limit the number of cores on which an IBM Spectrum Symphony service is expected to run by configuring job objects for Windows (no configuration is required for job objects). You can then limit CPU usage, which is a hard limit. For example, if four cores are available on the host and the SI is configured for one, it will use only one core as a maximum.

· Limit virtual memory usage:

You can limit virtual memory usage for Windows for multidimensional scheduling. IBM Spectrum Symphony converts the maxmem multidimensional metric a virtual memory limit, in the IBM Spectrum Symphony application profile.

You can configure the maximum amount of memory that an IBM Spectrum Symphony service and its descendent processes can consume using job objects for Windows (no prerequisite configuration is needed for job objects, as the SIM creates a job object and once the SI process is created, the SIM attaches the SI to the job object and applies the limits).

Configuring multidimensional resource usage limits

Configuring multidimensional resource usage limits on Windows is similar to configuring on Linux. Refer to Flow to configure multidimensional scheduling and Configuring resource limits for multidimensional scheduling on Linux as reference.

To enable application-level resource usage limitations in the application profile, either edit the profile manually, or use the cluster management console. To configure resource usage limits on Windows by editing the profile manually:

1) Specify the metrics ncpus (number of cores) and maxmem (memory in megabytes (MBs)) settings in your application profile:

<Metric name="ncpus">1</Metric>

<Metric name="maxmem">2000</Metric>

</SlotDefinition>

</SlotDefinitions>

</MDS>

With this configuration, IBM Spectrum Symphony converts the ncpus metric value of 1 to a cpuLimit parameter value of 1 in the IBM Spectrum Symphony application profile. Likewise, it converts the maxmem metric value of 2000 to a virtualMemoryLimit parameter value, of 2000 in the application profile. (Although IBM Spectrum Symphony also supports the maxswpand maxtmp metrics, only the ncpus and maxmem metrics are used to convert CPU and service instance resource usage limit configuration.)

2) Re-register the application profile:

soamreg profile.xml

3) Verify your configured CPU and memory usage limits by submitting tasks from IBM Spectrum Symphony, and then checking the SIM log (Installation_top\soam\logs\sim.${hostname}.${application_name}.1.log). Verify whether IBM Spectrum Symphony workload is killed by the IBM Spectrum Symphony multidimensional scheduling CPU and memory usage settings when the limits are reached.

For example, this message verifies that the virtual memory limit set at 2000 megabtyes was reached, and the workload was killed because it required 2084 megabytes, which is greater than your memory limit of 2000 megabytes:

2020-04-06 23:14:06.212 Eastern Standard Time ERROR [7596:14344] sim.apihandler.APIHandlerBase - APIHandlerBase::onServiceExit(): The process <9048> exited with abnormal status and may have been killed by the IBM Spectrum Symphony multidimensional scheduling application level memory limit feature. The total virtual memory limit is <2000M>, and the total virtual memory usage is <2084M>.

Troubleshooting multidimensional resource usage limits

Take note of the following scenarios when using resource usage limits for multidimensional scheduling:

· When you enable resource usage limits for multidimensional scheduling and if the memory used is more than the limitation set, the SIM receives a message from the Windows system to kill the SI. However, the Windows API does not always relay this message, and therefore, the SIM may not know to kill the SI.

To ensure that the SI will be restarted and the task will be retried as expected, as a best practice, in your service code, use a try-catch pair statement (try and catch) when applying memory to catch the out of memory message, and add an exit success statement (exit(0)) to indicate that the SIM has successfully killed the SI. For example:

try

    ... ...

    p[i]=new int[nSize];

    ... ...

catch(std::bad_alloc& ex)

    m_logger->log(MyLogger::INFO_LOG, "My alloc in onInvoke(), catch the bad_alloc and get memory failed");

    exit(0);

catch(...)

    m_logger->log(MyLogger::INFO_LOG, "My alloc in onInvoke(), catch the ... and get memory failed");

    exit(0);

· When you enable resource usage limits for multidimensional scheduling and if the memory used is more than the limitation set, the SIM kills the SI. This kill operation occurs during any SI phase, such as during the Register or CreateService methods. Next, the controlCode actions (such as blockHost), defined in the application profile, are triggered.

Therefore, if you find that there are sessions in open state and cannot be closed, or if you see blockHost messages in the SIM log, you may have blocked hosts. Use the Hosts Blocked for Application page within the cluster management console to see a list of blocked hosts, and then unblock them by clicking Unblock (see Removing a host from the blocked hosts list for details). You can also unblock hosts by re-registering the application profile using the soamreg command.

c. Enable the client loading CA certificates from the Windows certificate store:

CA_THUMBPRINT_LIST is used to configure the thumbprint of the CA. If there is only one CA, configure it with the format thumbprint@store. If there are multiple CAs in the certificate chain, separate each thumbprint@store string using a colon (:), for example, thumbprint1@store1:thumbprint2@store2. The system will use the thumbprint configured in CA_THUMBPRINT_LIST to acquire a CA certificate from the Windows certificate store. If CA_THUMNPRINT_LIST is configured, CAFILE and CAPATH will be ignored.

IBM Spectrum Symphony supports the following SSL communications:

· Between VEMKD and the VEMKD client

· Between SD and SDK

· Between SSM and SDK

· Between SSM and SIM

Configuring other types of SSL, will fail, as they are not supported.

Configuring SSL between VEMKD and the VEMKD client in the ego.conf file

Assuming that EGO_TRANSPORT_SECURITY, EGO_KD_TS_PORT, and EGO_DEFAULT_TS_PARAMS are properly configured, add the following configuration the ego.conf file:

EGO_CLIENT_TS_PARAMS="SSL[CIPHER=AES256-GCM-SHA384,CA_THUMBPRINT_LIST=c08599998f2e79900c4d7fefb72e3ebb383723bf@ca:9baf4fc8fd8d70d7148c9ffd9d3af24bc19fdfc6@root]"

Configuring SSL between SD and SDK, or SSM and SDK, in the sd.xml file

Assuming that SD_SDK_TRANSPORT, SD_SDK_TRANSPORT_ARG, SSM_SDK_TRANSPORT and SSM_SDK_TRANSPORT_ARG are properly configured, add the following configuration to your sd.xml file:

<ego:EnvironmentVariable name="SDK_TRANSPORT">TCPIPv4SSL</ego:EnvironmentVariable>

<ego:EnvironmentVariable name="SDK_TRANSPORT_ARG">SSL[CIPHER=AES256-GCM-SHA384,CA_THUMBPRINT_LIST=c08599998f2e79900c4d7fefb72e3ebb383723bf@ca:9baf4fc8fd8d70d7148c9ffd9d3af24bc19fdfc6@root]</ego:EnvironmentVariable>

Configuring SSL between SSM and SIM in the application profile

Assuming that SSM_SIM_TRANSPORT and SSM_SDK_TRANSPORT_ARG are properly configured, add the following configuration to your application profile:

6. Uninstallation

If required, follow the instructions in this section to uninstall this interim fix from your cluster:

Uninstalling from IBM Spectrum Symphony management and compute hosts

a. Log on to the primary host as the cluster administrator, disable all applications, and shut down all primary and primary candidate hosts:

> soamcontrol app disable all

> egosh ego shutdown hostname

b. Roll back to the previous version of IBM Spectrum Symphony either from the Windows Control Panel or from the IBM Spectrum Symphony command line:

o To roll back from the Windows Control Panel, go to Control Panel > Programs and Features > View installed updates, click Update for Symphony 7.3.2 (build “601026”), and click Uninstall.

o To roll back from the IBM Spectrum Symphony command line, enter the following command:

C:\>msiexec /uninstall {FAAC0FD7-8B0A-4EB4-80DA-A41E9AE51F42} /package {E56C0FC4-F6C0-4FDC-8037-E576A6650CAA} /norestart /quiet /l*v sym_rollback.log

C:\>msiexec /uninstall {E9317DE0-AC39-4172-BB90-6BFD47BF4E4E} /package {E0D84198-BD82-44E9-BE9E-56BC19B4AE75} /norestart /quiet /l*v sym_rollback.log

The command syntax is as follows:

C:\>msiexec /uninstall interim_fix_code /package product_code /norestart /quiet /l*v rollback_log

where:

o interim_fix_code is the identifier of the .msp package for the interim fix; in this case, {FAAC0FD7-8B0A-4EB4-80DA-A41E9AE51F42} and {E9317DE0-AC39-4172-BB90-6BFD47BF4E4E}.

o product_code is the identifier of the .msi file in the product installation package; in this case, {E56C0FC4-F6C0-4FDC-8037-E576A6650CAA} and {E0D84198-BD82-44E9-BE9E-56BC19B4AE75}.

o rollback_log is the name of the log file to capture details of the interim fix rollback; in this case, sym_rollback.log.

c. Log on to the primary host as the cluster administrator, start all the primary and primary candidate hosts, and enable your applications:

> egosh ego start hostname

> soamcontrol app enable appName

Uninstalling from IBM Spectrum Symphony client hosts

a. Log on to the IBM Spectrum Symphony client host and stop the running client.

b. On each IBM Spectrum Symphony client host, uninstall the fix:

o To roll back from the Windows Control Panel, go to Control Panel > Programs and Features > View installed updates, click Update for Symphony 7.3.2 (build “601026”) and click Uninstall.

o To roll back from the IBM Spectrum Symphony command prompt, enter the following command:

C:\> msiexec /uninstall {C0AEEAA0-F964-467E-9D06-3704024E39F1} /package {D0374E2E-2D3C-4240-9642-D99621AD5463} /norestart /quiet /l*v rollback.log

The command syntax is as follows:

C:\> msiexec /uninstall interim_fix_code /package product_code /norestart /quiet /l*v rollback.log

where:

o interim_fix_code is the identifier of the .msp package for this interim fix, in this case, {DF854676-7335-4A71-ABF0-D060230327C9}.

o product_code is the identifier of the .msi file for the original product installation package, in this case, {D0374E2E-2D3C-4240-9642-D99621AD5463}.

o rollback.log is the name of the log file to capture details of the rollback.

7. List of files

sym-7.3.2.0-egocore-4.0.0.0_build601026.msp

4.0\etc\vemkd.exe

4.0\etc\vemkd.pdb

4.0\etc\egosc.exe

4.0\etc\egosc.pdb

4.0\etc\wsm.exe

4.0\bin\egosh.exe

4.0\bin\egosh.pdb

4.0\lib\libvem.dll

4.0\lib\libvem.pdb

4.0\lib\libvem400.dll

4.0\lib\libvem400.pdb

4.0\lib\libsec.dll

4.0\lib\policy\libMDPlan.dll

4.0\lib\policy\libMDPlan.pdb

sym-7.3.2.0-soamcore-7.3.2.0_build601026.msp

       soam\7.3.2\w2k3_x64-vc7-psdk\etc\sd.exe
    soam\7.3.2\w2k3_x64-vc7-psdk\etc\sd.pdb
    soam\7.3.2\w2k3_x64-vc7-psdk\etc\ssm.exe
    soam\7.3.2\w2k3_x64-vc7-psdk\etc\ssm.pdb
    soam\7.3.2\w2k3_x64-vc7-psdk\etc\sim.exe
    soam\7.3.2\w2k3_x64-vc7-psdk\etc\sim.pdb
    soam\7.3.2\w2k3_x64-vc7-psdk\lib64\soambase.dll
    soam\7.3.2\w2k3_x64-vc7-psdk\lib64\soambase.pdb
    soam\7.3.2\w2k3_x64-vc7-psdk\lib\soambase.dll
    soam\7.3.2\w2k3_x64-vc7-psdk\lib\soambase.pdb
    soam\7.3.2\w2k3_x64-vc7-psdk\lib64\libvem.dll
    soam\7.3.2\w2k3_x64-vc7-psdk\lib64\libvem400.dll
    soam\7.3.2\w2k3_x64-vc7-psdk\lib\libvem.dll
    soam\7.3.2\w2k3_x64-vc7-psdk\lib\libvem400.dll
    soam\7.3.2\w2k3_x64-vc7-psdk\lib\libsec.dll
    soam\7.3.2\w2k3_x64-vc7-psdk\lib64\libsec.dll

symclnt-7.3.2.0_build601026.msp

       lib64\soambase.dll
    lib64\soambase.pdb
    lib64\libvem.dll
    lib64\libvem400.dll
    lib64\libvem400.pdb
    lib64\libsec.dll

       lib\soambase.dll
    lib\soambase.pdb
    lib\libvem.dll
    lib\libvem400.dll
    lib\libvem400.pdb
    lib\libsec.dll

8. Product notifications

To receive information about product solution and patch updates automatically, subscribe to product notifications on the My Notifications page http://www.ibm.com/support/mynotifications/ on the IBM Support website (http://support.ibm.com). You can edit your subscription settings to choose the types of information you want to get notification about, for example, security bulletins, fixes, troubleshooting, and product enhancements or documentation changes.

9. Copyright and trademark information

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM®, the IBM logo, and ibm.com® are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.