Use these instructions to install the LoadLeveler update RPMs on IBM Systems running supported versions of Red Hat Enterprise Linux and SUSE Linux Enterprise Server.

Read-me-first installation notes

This installation procedure assumes that you are updating a machine running the full version of LoadLeveler.

License RPM

Do not proceed if the LoadL-full-license- < OS-ARCH > - < installed_license_version > license package is not installed. For example, if you are currently running LoadLeveler 4.1.x for Linux, you cannot install an update for LoadLeveler 5.1.x. You must first upgrade to LoadLeveler 5.1.0.3-0 before you can install an update package for a 5.1.0.X-0 release. Please contact your IBM marketing representative for information on how to obtain the appropriate LoadLeveler CD-ROM that has the license package.

When uninstalling the "LoadL- < fileset > -full" RPM currently installed on your machine, do not uninstall the "LoadL-full-license" RPM for your currently installed LoadLeveler release. The "LoadL- < fileset > -full" update RPM has a dependency on the currently installed license RPM. Also, the LoadLeveler daemons will not start unless the license package is installed and you have accepted the terms and conditions of the license agreement.

Submit-only machines

The update steps for a LoadLeveler submit-only machine are similar to those associated with a machine running the full product. Simply replace the relevant "full" file names by their "so" counterparts. Also, on a submit-only machine there is no need to execute the llctl drain, llctl stop, or llctl start commands because LoadLeveler daemons do not run on a submit-only machine.

File name conventions

Download packages and RPMs for LoadLeveler updates use consistent file naming conventions. By looking at the components of the file name, you can determine which update package you will need to download based on the machine's architecture, Linux operating system, and the installed version of LoadLeveler.

Example

If your system is running scheduler and resource manager RHEL 6 with LoadLeveler 5.1.0.3-0 installed, and you want to download and install the LoadLeveler 5.1.0.4-0 update, then the file name components would be as follows:

Download package:

LL- < fileset > - < update_version > . < arch > . < OS-ARCH > .tar.gz
Specifies the LL_scheduler-5.1.0.4-0.x86_64-Linux-RHEL6.tar.gz file
Specifies the LL_resmgr-5.1.0.4-0.x86_64-Linux-RHEL6.tar.gz file

Package RPMs:

LoadL- < fileset > -full- < OS-ARCH > - < update_version > . < arch > .rpm
Specifies the LoadL-scheduler-full-RH6-X86_64-5.1.0.4-0.x86_64.rpm file
Specifies the LoadL-resmgr-full-RH6-X86_64-5.1.0.4-0.x86_64.rpm file

LoadL-scheduler-so- < OS-ARCH > - < update_version > . < arch > .rpm
Specifies the LoadL-scheduler-so-RH6-X86_64-5.1.0.4-0.x86_64.rpm file

LoadL-utils- < OS-ARCH > - < update_version > . < arch > .rpm
Specifies the LoadL-utils-RH6-X86_64-5.1.0.4-0.x86_64.rpm file

LoadL-resmgr-kbdd- < OS-ARCH > - < update_version > . < arch > .rpm
Specifies the LoadL-resmgr-kbdd-RH6-X86_64-5.1.0.4-0.x86_64.rpm file

BLUE GENE SYSTEM ONLY:
LoadL-scheduler-bluegene- < OS-ARCH > - < update_version > . < arch > .rpm
Specifies the LoadL-scheduler-bluegene-RH6-PPC64-5.1.0.6-0.ppc64.rpm file

Currently installed LoadLeveler and License RPMs:

LoadL- < fileset > - < OS-ARCH > - < currently_installed_version >
Specifies that LoadL-scheduler-full-RH6-X86_64-5.1.0.3-0 is installed
Specifies that LoadL-resmgr-full-RH6-X86_64-5.1.0.3-0 is installed

LoadL-full-license- < OS-ARCH > - < installed_license_version >
Specifies the LoadL-full-license-RH6-X86_64-5.1.0.3-0 license

where

< fileset >

The LoadLeveler scheduler or resource manager fileset.

< OS-ARCH >

The Linux Operating System and platform architecture of the machine where you are installing a LoadLeveler update package. For example, if you are upgrading an installation of LoadLeveler on a 64-bit AMD Opteron and Intel EM64T processors machine running RedHat 6, then < OS-ARCH > would be RH6-X86_64.

< update_version >

Specifies the version number of the LoadLeveler update package that you want to install on your system. For example, if you are updating to LoadLeveler 5.1.0.4-0, then the < update_version > is the number 5.1.0.4-0. The < update_version > appears in the file name of the package download (*.tar.gz file) and in the update RPMs in the package.

< arch >

Used in RPM file names. Specifies the platform architecture, where < arch > =
x86_64 (64-bit IBM System x) or

< base_version >

Used only in the file name of the downloadable update package (gzipped tar file). Specifies the LoadLeveler Version/Release to which the update can be applied, for example, 5.1.X.

< currently_installed_version >

Specifies the version number of the current LoadLeveler release installed on your system.

< installed_license_version >

Specifices the license RPM for the base release (or a lower release) of the LoadLeveler version you are updating. For example, to install the LoadLeveler 5.1.0.4 update, you must have an installed license RPM for LoadLeveler on Linux.

Installation procedure
1. Change to the < UpdateDirectory > directory, that is, the directory where the *.tar.gz file for the LoadLeveler update resides and where you have write access:
  
  cd < UpdateDirectory >
2. Extract the RPM files from the tar file:
  
  tar -xzvf LL- < fileset > - < update_version > . < arch > . < OS-ARCH > .tar.gz
  
  At the end of this step the files extracted from the archive should match the files listed in the Readme ("View" link) for the LoadLeveler update you downloaded.
3. Verify that the LoadLeveler "license" RPM for the LoadLeveler version that you are updating is currently installed on this system:
  
  rpm -qa | grep LoadL
  
  The output of this command should be similar to the following:
  
  LoadL- < fileset > -full- < OS-ARCH > - < currently_installed_version >
  LoadL-full-license- < OS-ARCH > - < installed_license_version >
4. If LoadLeveler resource manager fileset is running on this machine, enter the following command to "drain" the LoadL_schedd and LoadL_startd daemons on this machine and all other machines in the LoadLeveler cluster:
  
  llctl -g drain
  
  Note: To avoid potential incompatibility problems, all machines in the cluster should be upgraded to the same LoadLeveler update release before restarting the LoadLeveler daemons.
5. Use the llstatus command to verify that the LoadL_schedd and LoadL_startd daemons are in "Drned" (drained) state, and then enter the following command to stop the LoadLeveler daemons:
  
  llctl -g stop
6. To apply the update, use one of the following options:
  
  OPTION 1
  
  Uninstall the currently installed "LoadL- < fileset > -full" RPM and then install the "LoadL- < fileset > -full" update package by running the following commands:
  
  rpm -e LoadL- < fileset > -full- < OS-ARCH > - < currently_installed_version >
  rpm -ivh LoadL- < fileset > -full- < OS-ARCH > - < update_version > . < arch > .rpm
  
  For Blue Gene/Q only:
  Run these additional commands:
  rpm -e LoadL-scheduler-bluegene- < OS-ARCH > - < currently_installed_version >
  rpm -ivh LoadL-scheduler-bluegene- < OS-ARCH > - < update_version > . < arch > .rpm
  
  OPTION 2
  
  Use the -U option of the rpm command to apply the update directly:
  
  rpm -Uvh LoadL- < fileset > -full- < OS-ARCH > - < update_version > . < arch > .rpm
  For Blue Gene/Q only:
  Run this additional command:
  rpm -Uvh LoadL-scheduler-bluegene- < OS-ARCH > - < update_version > . < arch > .rpm
7. Repeat step 6 for all the machines in the LoadLeveler cluster. On completion of this task, restart the LoadLeveler daemons with the following command:
  
  llctl -g start
For further information, consult the LoadLeveler Library for the appropriate version of the LoadLeveler Linux Installation Guide.

Setting up control groups on a diskless (or stateless) cluster for preemption, process tracking, workload management (WLM), and checkpoint/restart
Control groups are used for preemption, process tracking, workload management (WLM), and checkpoint/restart. A control group file system must be configured and mounted in order to enable these functions. The /etc/cgconfig.conf file is used to define control groups, their attributes, and their mount points. On diskless systems the /etc/cgconfig.conf file should be part of the diskless image.
Complete the following steps to setup control groups on a diskless cluster.
1. Create the cgconfig.conf file in either an existing directory or a new directory in the /install/postscripts directory on the EMS management server.
  
  For example:
  mkdir -p /install/postscripts/admin_files
  vi /install/postscripts/admin_files/cgconfig.conf
  
  Note: Files and subdirectories under /install/postscripts on the EMS are copied to the /xcatpost directory on the compute nodes before the postscripts are run.
2. If you plan to use checkpoint/restart or LoadLeveler process tracking and preemption, the first entry in the cgconfig.conf file must be:
  
  mount {ns = /cgroup/freezer;freezer = /cgroup/freezer;}
3. If you plan to use WLM, include the following in the cgconfig.conf file:
  
  mount {
    cpu = /cgroup/cpu;
    cpuacct = /cgroup/cpuacct;
    memory = /cgroup/memory;
  }
  
  group LOADL {
    memory{
      memory.limit_in_bytes = ;
      memory.soft_limit_in_bytes = ;
      memory.memsw.limit_in_bytes = ;
  }
  cpu{}
  cpuacct{}
  }
4. In a script that is configured in the xCAT database as a postscript for the compute nodes, add the following:
  cp /xcatpost/admin_files/cgconfig.conf /etc/cgconfig.conf /sbin/cgconfigparser -l /etc/cgconfig.conf
  
  where "admin_files" is the directory name you used in step 1.
  
  If you plan to use scheduling affinity, add the following lines to this script after invoking /sbin/cgconfigparser
  
  mkdir /dev/cpuset
  mount -t cpuset none /dev/cpuset

Notes

Unless specifically noted otherwise, this history of problems fixed for LoadLeveler 5.1.0.x applies to:

LoadLeveler 5.1.0.x for Red Hat Enterprise Linux 6 (RHEL6) on servers with 64-bit Opteron or EM64T processors
LoadLeveler 5.1.0.x for SUSE LINUX Enterprise Server 11 (SLES11) on servers with 64-bit Opteron or EM64T processors
LoadLeveler 5.1.0.x for Red Hat Enterprise Linux 6 (RHEL6) on POWER servers

Restriction section

For LL 5.1.0:

If the scheduler and resource manager components on the same machine are not at the same level, the daemons will not start up.
Please refer to the "Known Limitations" section under the fix pack README for more limitation information for this release.

Additional Information section

For LL 5.1.0:

Please refer to the "Setting up control groups on a diskless (or stateless) cluster for preemption, process tracking, workload management (WLM), and checkpoint/restart" under the "Installation Information" section for more information on how to setup control groups.

General LoadLeveler problems

Problems fixed in LoadLeveler 5.1.0.11 [August 27, 2012]

The coredump problem when fetching step adapter usage information has been fixed.
It has been fixed that the command "llstatus -l -L" shows submit-only node down.
The negotiator daemon now correctly free the memory so that the core dump will not occur.
Resource Manager only:
- The Region Manager has been modified to ignore all adapters on the same subnet as the adapter that was filtered out with adapter_list. Instead of the Region Manager marking those adapters down, those adapters will remain in an HB_UNKNOWN state.
Blue Gene:
- If a Blue Gene job terminates due to a kill timeout, the node used by the job is availble for future jobs after the block in use has been freed.
Scheduler only:
- Only the messages from the last iteration of topdog scheduling is printed out in the command "llq -s".The intermediate message is not printed.
- The accounting record which has a negative wall clock value is now skipped by the llsummary command.

Problems fixed in LoadLeveler 5.1.0.10 [July 20, 2012]

The region manager failover and recovery code is changed to ensure that the resource manager is notified when a region manager becomes active which makes all active nodes and adapters available for scheduling.
Resource Manager only:
- The resource manager daemon will not crash once startup LL if set D_FULLDEBUG for RESOURCE_MGR_DEBUG in LoadL_config file.
Blue Gene:
- LoadLeveler Changes to use the new checkIO() call for V1R1M1 BlueGene software.
- The dependency check for the libbgsched shared object is removed from the LoadLeveler Blue Gene rpm so that the rpm nodeps option is no longer required.
- LoadLeveler llqres command will display the information for the Blue Gene reservation which specifying bg_block.
- A check that was preventing Blue Gene reservations from being modified has been fixed so the change request can be processed.
- When some nodeborard is down in one midplane, the Blue Gene small block job can run in the midplane if the resource can meet the job requirement.
- The nodeboard list that is returned from the BGQ scheduler API may not always be in order. LoadLeveler will sort this list to ensure it is in order before indexing on it.

Problems fixed in LoadLeveler 5.1.0.9 [June 19, 2012]

Update for LINUX on 64-bit Opteron or EM64T processors ONLY
Implemented internal LoadLeveler data contention improvements.
Jobs were rejected when the schedd daemon was unable to determine the protocol versions for the nodes allocated to a job step it was trying to dispatch. The corre ct protocol version is being called now so that the jobs will be started correctly.
Fixed Negotiator daemon memory leaks.
Incorrect error messages seen for user prolog/epilog during llctl ckconfig command which is fixed by correcting the internal user variables names.
Corrected inefficiency when reading configuration data from the database and protect against these kinds of performance issues that had caused LoadLeveler from st arting when large systems are configured.
Corrected the lldbupdate to be able to update from 5.1.0.6 to 5.1.0.9.

Problems fixed in LoadLeveler 5.1.0.8 [June 15, 2012]

Update for LINUX on POWER ONLY
Implemented internal LoadLeveler data contention improvements.
Jobs were rejected when the schedd daemon was unable to determine the protocol versions for the nodes allocated to a job step it was trying to dispatch. The correct protocol version is being called now so that the jobs will be started correctly.
Fixed Negotiator daemon memory leaks.
Incorrect error messages seen for user prolog/epilog during llctl ckconfig command which is fixed by correcting the internal user variables names.
Corrected inefficiency when reading configuration data from the database and protect against these kinds of performance issues that had caused LoadLeveler from starting when large systems are configured.
Corrected the lldbupdate to be able to update from 5.1.0.6 to 5.1.0.8.

Problems fixed in LoadLeveler 5.1.0.7 [June 8, 2012]

Do not install LL 5.1.0.7 service update if you are using or planning to use a database for the LoadLeveler configuration.
The llstatus command shows the startds to be up even though the llrstatus command shows the startd and the region manager they report to is actually down. The central manager will now be notified by the resource manager when the startd is marked as down by the resource manager so the llstatus command will now show the correct output state as the llrstatus command.
Fixed llconfig from core dumping if trying to add a new machine_group or region to a cluster that has more than 128 machines.
Fixed llconfig to correctly set the island in the maching_group.
Blue Gene:
- LoadLeveler will correctly calculate the I/O ratio per midplane based on hardware state to support a mixed I/O environment on Blue Gene/Q.

Problems fixed in LoadLeveler 5.1.0.6 [April 27, 2012]

Mandatory service pack for Red Hat Enterprise Linux 6 (RHEL6) on POWER servers.
The CAU value is now allocated correctly on all the nodes on which the job is run.
Resource Manager only:
- Fixed dead lock in Region manager daemon when determining heartbeat status and llstatus information will now show the correct status after reconfig.
- Fixed startd daemon from core dump when preempting a running job via suspend method.
- Fixed checking of process tracking during job termination so jobs will be able to terminate correctly in an environment that does that hve process tracking set.
Blue Gene:
- Enhanced the support for Blue Gene block booting failures by draining problem hardware from the LoadLeveler cluster.
- Fixed problems with LoadLeveler scheduling blocks using pass through.
- Updated llq -h command output to reflect changes in Blue Gene terminology (Partitions are now referred to as Blocks)
- Corrected display of connectivity for large blocks in llsummary output.
- Fixed a problem calculating the minimum block size for LoadLeveler jobs when midplanes contain error with iolinks.

Problems fixed in LoadLeveler 5.1.0.5 [April 4, 2012]

Fixed some memory leaks in Startd and Schedd daemons.
If there is no network statement in the job command file, then the default network is used, which assumes ethernet. If the cluster does not have ethernet configured, then the job will stay in the "ST" state and not run. The default network support will now use the adapter associated with the hostname with which the machine is configured in the administration file.
Fixed LoadL_master from core dumping during llctl stop in database environment due to timing locks.
Fixed LoadL_negotiator from core dumping by not sending corrupted job step data to the central manager.
Fixed lldbupdate command from getting the 2544-019 error message by parsing the database information correctly so LoadLever will be able to start up.
Resource Manager only:
- A problem in pe_rm_connect() that caused read() to be called on a socket that was not ready to be read has been corrected, allowing pe_rm_connect() to continue to retry to the connection for the specified rm_timeout amount of time.
Scheduler only:
- The list of reserved resources was not being updated properly when the reservation requesting a 0 count ended, leading to the core dump. That reservation list is now being being updated correctly in all cases.

Problems fixed in LoadLeveler 5.1.0.4 [March 16, 2012]

Mandatory service pack for Red Hat Enterprise Linux 6 (RHEL6) and SUSE LINUX Enterprise Server 11 (SLES11) on servers with 64-bit Opteron or EM64T processors.
LoadLeveler can now display the host name correctly based on the name_server configuration. The previous limitation of the name_server keyword being ignore is now lifted.
On SLES11, the lldbupdate fails to connect to the database due to incorrect odbc.ini location is now corrected.
Fixed Linux schedd daemon core dump in a mixed AIX and Linux cluster when submitting a job from the AIX cluster.
Fixed potential central manager deadlock.

Product/Component Name:	Platform:	Fix:
LoadLeveler	Linux 64-bit,x86_64 SLES 11	LL_resmgr-5.1.0.11-x86_64-Linux-SLES11

Readme and Release notes for release 5.1.0.11 LoadLeveler 5.1.0.11 LL_resmgr-5.1.0.11-x86_64-Linux-SLES11 Readme