Solution Readme

===============================================================================
Readme file for: IBM® Platform LSF
Product/Component Release: 9.1.3
Update name: Fix 404991
Publication date: 14 July 2016
Last modified: 14 July 2016

Abstract:

Enhance the LSF external scheduler plugin API. You can use the LSF external scheduler plugin API to customize existing scheduling policies or implement new ones that can operate with existing LSF scheduler plugin modules.

===============================================================================

=========================
CONTENTS
=========================
1. Abbreviations
2. About the LSF eternal scheduler plugin API
3. Supported operating systems
4. Products or components affected
5. Installation and configuration
6. Copyright

=========================
1. Abbreviations
=========================

API application program interface

=========================
2. About the LSF eternal scheduler plugin API
=========================

The LSF external scheduler API now allows an external plugin to register checkAlloc and notifyAlloc callback functions. Essentially the LSF scheduling framework calls all registered checkAlloc functions while an allocation is being created for a job. Within a checkAlloc callback function a plugin can examine the allocation and optionally limit the allocation while specifying a reason. The LSF scheduling framework calls all registered notifyAlloc functions once an allocation has been created (or destroyed) for a job. Within a notifyAlloc callback function a plugin can examine the allocation, and update its own state accordingly.

Besides the enhanced external scheduler plugin API, LSF will now support jobs submitted with resource requirement strings up to 4095 characters in length. This behaviour must be configured by setting LSB_EXTENDED_RSRCREQ_STR=Y in the lsf.conf configuration file, otherwise the limit of 511 characters is used. Regardless of the setting for LSB_EXTENDED_RSRCREQ_STR the value of the environment variable LSB_SUB_RES_REQ set within a job's environment will be truncated to 511 characters.

lsf.conf

LSB_EXTENDED_RSRCREQ_STR
Normally within LSF the job-level resource requirement string is limited to 511 characters. With LSB_EXTENDED_RSRCREQ_STR=Y|y this limit is increased 4095 characters.
Syntax:

LSB_EXTENDED_RSRCREQ_STR=Y|y|N|n

Default:

A brief description of the enhanced external scheduler plugin API follows. See the lssched.h header file for additional details.

Signature for functions that check allocations for peer jobs.
typedef int (*RsrcReqHandler_CheckAllocFn)(void *info, INT_Job *job, INT_Alloc *alloc, INT_AllocLimitList *allocLimitList);

Signature for notify allocation functions.
typedef int (*RsrcReqHandler_NotifyAllocFn)(void *info, INT_Job *job, INT_Alloc *alloc, INT_AllocLimitList *allocLimitList, int flag);

Iterate to the next group of candidate hosts for a job.
struct candHostGroup *extsched_cand_getnextgroup(INT_CandGroupList *candGroupList);

Returns the resources on the host.
struct hostResources *extsched_host_resources(INT_Host *host);

Helper routine returning the string from the enum restype.
char *extsched_getResType(const struct resources *r);

Provide the reason why a host is not considered usable for a job.
int extsched_reason_set(INT_Reason *reasonPtr, struct candHost *candHost, int reasonId);

Remove a candidate host from the group.
void extsched_cand_removehost(struct candHostGroup *group, int index);

Get number of available slots on a candidate host.
struct hostSlot *extsched_cand_getavailslot(struct candHost *candHost);

Register a new handler for job resource requirements.
int extsched_resreq_registerhandler(int handlerId, RsrcReqHandlerType *handler);

Attach handler-specific data to a set of peer jobs.
void extsched_resreq_setobject(INT_RsrcReq *resreq, int handlerId, char *key, void *handlerData);

Register a functions that adjusts allocation decisions for jobs.
int extsched_alloc_registerallocator(AllocatorFn allocator);

Get user-specified resource requirement message for the peer jobs.
extern char **extsched_resreq_getextresreq(INT_RsrcReq *resreq, int *msgCnt);

Get queue level MANDATORY_EXTSCHED and DEFAULT_EXTSCHED for job.
int extsched_resreq_getqueextsched(INT_RsrcReq *resreq, char **mand_extsched, char **default_extsched);

Get the user-specified resource requirements messages for the job.
char **extsched_job_getextresreq(INT_Job *jobPtr, int *msgCnt);

Get the number of slots requested by a job.
int extsched_job_getaskedslot(INT_JobBlock *jobPtr);

Get the distribution of slots on various candidate hosts in job's allocation.
struct hostSlot* extsched_alloc_gethostslot(INT_Alloc *alloc, INT_CandGroupList *candGroupList, int *hostCnt, struct candHostGroup **group);

Return the type of allocation given to the job (reservation or dispatch).
int extsched_alloc_type(INT_Alloc *alloc);

Adjust the number of slots allocated to a job on a candidate host, if possible.
int extsched_alloc_modify(INT_JobBlock *jobptr, INT_Alloc **alloc, struct hostSlot *host, INT_Reason *reason, struct candHostGroup *group);

Register a function that decides job dispatch order for all queues.
int extsched_order_registerOrderFn4AllQueues(JobOrderFn4Que orderFn);

Check if the jobList is empty.
int extsched_order_isJobListEmpty(INT_JobList *jobList);

Get the first job in the job list.
INT_JobBlock *extsched_order_getFirstJobOfList(INT_JobList *jobList);

Get the next job in the job list.
INT_JobBlock *extsched_order_getNextJobOfList(INT_JobBlock *currJob, INT_JobList *jobList);

Get the previous job in the job list.
INT_JobBlock *extsched_order_getPreJobOfList(INT_JobBlock *currJob, INT_JobList *jobList);

Get the number of jobs in the job list.
int extsched_order_getJobNumOfList(INT_JobList *jobList);

Get the priority of a job, which could be defined by bsub -sp.
int extsched_order_getJobPriority(INT_JobBlock *job);

Get the sequence number of a job.
unsigned int extsched_order_getJobSeqNum(INT_JobBlock *job);

Get the submission time of a job.
time_t extsched_order_getJobSubmitTime(INT_JobBlock *job);

Get the user name of a job.
char *extsched_order_getJobUser(INT_JobBlock *job);

Function to be called from within checkAlloc function to determine the context in which this function is called.
extsched_SchedContext extsched_getCheckAllocContext(void);

To be called from within a registered checkAllocFn, in order to block a job from using the allocation.
int extsched_addAllocLimit(INT_AllocLimitList *allocLimitList, INT_Alloc *alloc, char *host, int maxTasks, int reasonId, char *reasonDetail, int handlerId, int bucketReason);

Function to get the list of jobs that are preemptable by the given job on the given host.
int extsched_getPreemptableJobs(INT_Job *job, char *host, int *nPreemptableJob, INT_Job ***preemptableJob);

Function to extract information from the internal representation of a resource allocation.
int extsched_getAlloc(INT_Alloc *alloc, extsched_Alloc **infoPtr);

Frees the data structure created by extsched_getAlloc().
void extsched_freeAlloc(extsched_Alloc **info);

Get information on the internal allocation limit list data structure. It is intended to be called from within a checkAlloc function.
int extsched_getAllocLimitList(INT_AllocLimitList *allocLimitList, extsched_AllocLimitList **infoPtr);

Frees the data structure generated by extsched_getAllocLimitList().
void extsched_freeAllocLimitList(extsched_AllocLimitList **info);

This function returns the current allocation that is made to a job, if it exists.
INT_Alloc *extsched_getJobAlloc(INT_Job *job);

Routine frees the allocation structure and sets *allocPtr NULL. This should only be called by an allocator function.
void extsched_freeIntAlloc(INT_Alloc **allocPtr);

Routine to get the address of the (internal) resource requirement structure associated with the given job.
INT_RsrcReq *extsched_getRsrcReqForJob(INT_Job *job);

Routine that exposes part of the job's internal resrource requirement.
extsched_rsrcReqInfo *extsched_getRsrcReqInfo(INT_RsrcReq *rsrcreq);

The counterpart for extsched_getRsrcReqInfo() to free the returned object.
void extsched_freeRsrcReqInfo(extsched_rsrcReqInfo **info);

Routine to extract information from the internal representation of a job.
int extsched_getJobInfo(INT_Job *job, struct jobInfo *info);

Frees data in the info structure, allocated by extsched_getJobInfo(). Does not call free() on info itself.
void extsched_freeJobInfo(struct jobInfo *info);

Routine allowing an external scheduling plugin to request a modification of a job's resource requirement expression, i.e. the bsub -R string (job level rsscreq).
int extsched_modifyJob(INT_Job *job, char *rsrcReq);

Routine allowing an external scheduling plugin to request a modification of a job's resource requirement expression to it's original value at submission, or the latest bmod value.
int extsched_resetJob(INT_Job *job);

This call reports success/failure about the last call to extsched_modifyJob or extsched_resetJob for jobs. The associated job array is broken into elements and the LSBE_* error is determined.
int extsched_getSchedJobModError(INT_Job *job, int *nElem, extsched_JobElement **jobList, int **errList);

Set the main pending reason (not host based). PEND_EXTSCHED_REASON
int extsched_setMainReason(INT_Reason *reason, char *msg, int bucketLevel);

Get a pointer to a host's internal representation.
INT_Host *extsched_getHost(char *hostname, char *clustername);

Get the local cluster name.
char *extsched_clustername(void);

This function will provide the host ID (hostname and clusternam) provided with the internal host pointer.
int extsched_getHostID(INT_Host *hostPtr, char **hostname, char **clustername);

This is meant to be used from inside an allocator function. Pass in the jobBlock pointer to obtain the corresponding INT_Job pointer.
INT_Job *extsched_get_INT_Job(INT_JobBlock *jobBlock);

This is a helper function to be used from within a notifyAlloc callback function to determine what is happening with the job.
int extsched_determineEvent(INT_Job *job, INT_Alloc *alloc, INT_AllocLimitList *allocLimitList, int flag);

This function is used to register a callback function that will be invoked during preemption to evaluate whether a job can be preempted. The callback function should return zero to block preemption, or a non-zero value to allow preemption.
typedef int (*CanBePreemptedFn)(INT_Job *job);
int extsched_register_canBePreempted( CanBePreemptedFn canBePreemptedFunc );

=========================
3. Supported operating systems
=========================

linux2.6-glibc2.3-x86_64

=========================
4. Products or components affected
=========================

lssched.h

bsub
bmod
brestart
mbatchd
mbschd
mesub

mbschd plugins:
schmod_advrsv.so
schmod_affinity.so
schmod_aps.so
schmod_bluegene.so
schmod_cpuset.so
schmod_craylinux.so
schmod_crayx1.so
schmod_dc.so
schmod_default.so
schmod_demand.so
schmod_dist.so
schmod_fairshare.so
schmod_fcfs.so
cal_jobweight.so
schmod_jobweight.so
schmod_limit.so
schmod_mc.so
schmod_parallel.so
schmod_preemption.so
schmod_pset.so
schmod_ps.so
schmod_reserve.so
schmod_rms.so
schmod_xl.so

=========================
5. Installation and Configuration
=========================

5.1          Before installation

     (LSF_TOP=Full path to the top-level installation directory of LSF.)
    1)    Log on to the LSF master host as root
    2)    Set your environment:
            -      For csh or tcsh: % source LSF_TOP/conf/cshrc.lsf
            -      For sh, ksh, or bash: $ . LSF_TOP/conf/profile.lsf

5.2          Installation steps

    1)    Go to the patch install directory: cd $LSF_ENVDIR/../9.1/install/
    2)    Copy the patch file to the install directory $LSF_ENVDIR/../9.1/install/
    3)    Run patchinstall: ./patchinstall <patch>

5.3          After installation

    1)    Log on to the LSF master host as root
    2)    Optionally update the lsf.conf configuration to enable extended resource requirement string feature.
           Edit the $LSF_ENVDIR/lsf.conf file. In the Parameters section that the following are set.
                LSB_EXTENDED_RSRCREQ_STR=Y
    3)    Run
                lsadmin limrestart -f all
badmin mbdrestart

5.4          Uninstallation

    To roll back a patch:

    1)    Log on to the LSF master host as root
    2)    Run ./patchinstall -r <patch>
    3)    Run
                lsadmin limrestart -f all
badmin mbdrestart

5.5          Using the new external scheduler plugin API. (Optional)

    Replace the lssched.h header file. This file is located in <LSF_TOP>/9.1/include/lsf. Extract the new
lssched.h file from the patch and copy it into <LSF_TOP>/9.1/include/lsf. For information on creating
    your own scheduler plugin refer to the contents of <LSF_TOP>/9.1/misc/examples/external_plugin/.

=========================
6. Copyright
=========================

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM®, the IBM logo and ibm.com® are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide.

Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the

Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml