===============================================================================
Readme file for: IBM® Spectrum LSF Resource Connector for SoftLayer
Product/Component Release: 10.1.0.2
Publication date: 1 June 2017
Last modified: 23 May 2017
Abstract: This feature enables LSF to request virtual compute hosts from an IBM
SoftLayer (Bluemix) service.
===============================================================================
=========================
CONTENTS
=========================
1. Abbreviations
2. About Spectrum LSF Resource Connector for SoftLayer
3. Supported operating systems
4. Products or components affected
5. Installation and configuration
6. Operation
7. Logging and troubleshooting
8. Configuration reference
9. Copyright
=========================
1. Abbreviations
=========================
DNS - Domain Name Service
JRE - Java Runtime Environment
RC - Resource Connector
VLAN - Virtual Local Area Network
=========================
2. About Spectrum LSF Resource Connector for SoftLayer
=========================
This feature enables LSF clusters to borrow virtual compute hosts from the IBM SoftLayer (Bluemix) service if the workload demand exceeds cluster capacity. The Resource Connector generates requests for additional hosts from SoftLayer and dispatches jobs to dynamic hosts that join the LSF cluster. When the demand reduces, the Resource Connector shuts down slave LSF daemons and cancels allocated SoftLayer virtual servers.
You can find more details about the LSF Resource Connector in the Knowledge Center at https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_welcome/lsf_kc_resource_connector.html
=========================
3. Supported operating systems
=========================
The LSF Resource Connector for SoftLayer is tested and works with the following systems:
- IBM Spectrum LSF 10.1.0.2 Standard Edition.
- Linux x86 Kernel 3.10, glibc 2.17 CentOS 7.x
- Java Runtime Environment 1.8
It is expected to work on Linux x86 distributions using kernel 2.6 and higher.
=========================
4. Products or components affected
=========================
Spectrum LSF Resource Connector for SoftLayer works with Spectrum LSF 10.1.0.2 or higher.
=========================
5. Installation and configuration
=========================
-------------------------
5.1. Prerequisites
-------------------------
The following steps assume that you already have set up an LSF cluster and meet the requirements
listed below. They also assume that you have an existing SoftLayer account.
* You must have an existing LSF cluster set up and have root access to the LSF master host.
* You must be able to restart the LSF cluster master daemon.
* You must be familiar with operating the SoftLayer Console.
* You must have sufficient permissions to create and configure a custom virtual server image and, optionally, postprovision scripts, and order virtual servers on IBM SoftLayer.
* The VLAN to be used by SoftLayer virtual servers must be configured so that they can communicate with the LSF master host.
* This component requires Java Runtime Environment (JRE) version 1.8 or newer and a
Linux version compatible with that JRE
-------------------------
5.2. Create a SoftLayer custom virtual image template
-------------------------
5.2.1. Manually order a SoftLayer computing instance with a standard OS image
- Log in to your account from the SoftLayer Customer Portal (https://control.softlayer.com/).
- Click Devices in the Order section and order a virtual server. Select the desired configuration for this virtual server.
- In Advanced System Configuration, choose a Host and Domain Name.
- Submit the order. You will receive an email notification indicating that the Virtual Server is currently in the provisioning process. Standard delivery times are 20 minutes or less.
- Check Devices > Device List in the SoftLayer portal to see whether the cloud server is provisioned. After the server is available, you can customize this compute instance.
5.2.2. Customize the compute instance by installing IBM Spectrum LSF on it
To customize the Softlayer instance you will use SSH (with the public IP as root user). Obtain the root password for this device from the SoftLayer portal by clicking the device name and selecting the Passwords tab in the device details.
1) Use `scp` to copy the IBM Spectrum LSF installation packages to this Softlayer instance, for example:
- lsf10.1_lnx310-lib217-x86_64.tar.Z
- lsf10.1_lsfinstall_linux_x86_64.tar.Z
- lsf_std_entitlement.dat
Use the binary package appropriate for your virtual server OS version.
2) Log on as root.
For example:
ssh 169.55.162.62 -l root
3) Configure DNS.
Configure the DNS settings to ensure that the virtual server can look up the host name and IP address of the LSF master host.
4) Edit slave.config and install LSF.
Unpack lsf10.1_lsfinstall_linux_x86_64.tar.Z into a working directory. Edit `slave.config` as required. Typical contents of that configuration file are shown below (values given are examples; please use values appropriate for your environment):
LSF_TOP="/home/sluser/lsf"
LSF_ADMINS="sluser"
LSF_TARDIR="/home/sluser/lsf/"
LSF_ENTITLEMENT_FILE="/home/sluser/lsf/lsf_std_entitlement.dat"
LSF_SERVER_HOSTS="master.myserver.com"
LSF_LOCAL_RESOURCES="[resource softlayercomp] [resource define_ncpus_threads]"
LSF_LIM_PORT="7869"
Install a slave host:
./lsfinstall -s -f slave.config
For more details, refer to the following:
http://www-01.ibm.com/support/knowledgecenter/SSETD4_9.1.3/lsf_unix_install/lsf_installnewunix_deploycluster_tsk.dita?lang=en
Start the LSF daemons on the virtual server manually and make sure that it can join the LSF cluster of the configured master host as a dynamic host.
5.2.3. Create the custom image template
By saving the previously customized virtual server image as a template you will be able to provision a preconfigured LSF slave host at any time.
Go to Device Details for this host at Devices > Device List > Device Details > Actions, choose an Image Name (for example, LSFComputeImage), and click Create Image Template.
When image creation is successful, go to Devices > Manage > Images to view the image you created. Remember the Image Name for host factory configuration.
5.2.4. Create a provisioning script for the LSF compute node virtual server
SoftLayer can download the provisioning script from an HTTPS server and execute it automatically when the virtual server is started and all its virtual devices are connected. The provisioning script is required to configure the LSF compute node to join the correct cluster and present correct shared resources. You can host your provisioning script on another virtual server in SoftLayer and configure it to use only the SoftLayer internal network for increased security. You can also host your script on any HTTPS server available on the public network.
An example provisioning script is shown below. Note how it reads from the getUserMetadata API to obtain the configuration variables set by the Resource Connector during provisioning.
~~~
#!/bin/bash
logfile=/var/log/postprovisionscripts.log
echo START `date '+%Y-%m-%d %H:%M:%S'` >> $logfile
#Do not remove this part of the script to support passing LSF user data to VM run time environment
STARTTIME=`date +%s`
TIMEOUT=60
URL="https://api.service.softlayer.com/rest/v3/SoftLayer_Resource_Metadata/getUserMetadata.txt"
USERDATA=`curl -s $URL` 2>>$logfile
#
while [[ "$USERDATA" == [Nn]"o user data"* ]] && [[ `expr $NOWTIME - $STARTTIME` -lt $TIMEOUT ]]; do
sleep 5
NOWTIME=`date +%s`
USERDATA=`curl -s $URL` 2>>$logfile
done
# check if we got user data eventually
if [[ "$USERDATA" != [Nn]"o user data"* ]]; then
# user data is expected to be a semicolon-separated key=value list
# like environment variables; split them into an array
IFS=\; read -ra ARR <<<"$USERDATA"
for VAR in ${ARR[@]}; do
eval "export $VAR"
done
else
echo "USERDATA: $USERDATA" >>$logfile
echo EXIT AT `date '+%Y-%m-%d %H:%M:%S'` >>$logfile
exit -1
fi
echo "CURRENT ENVIRONMENT:" >>$logfile
env >> $logfile
#Set the correct path for LSF_TOP, where LSF is installed on the VM host
LSF_TOP=/opt/lsf
LSF_CONF_FILE=$LSF_TOP/conf/lsf.conf
source $LSF_TOP/conf/profile.lsf
#Do not remove this part of the script to support rc_account resource for SoftLayer
#You can similarly set additional local resources if needed
if [ -n "${rc_account}" ]; then
sed -i "s/\(LSF_LOCAL_RESOURCES=.*\)\"/\1 [resourcemap ${rc_account}*rc_account]\"/" $LSF_CONF_FILE
echo "update LSF_LOCAL_RESOURCES lsf.conf successfully, add [resourcemap ${rc_account}*rc_account]" >> $logfile
fi
#If there is no DNS server to resolve host names and IPs between master host and VMs,
#then uncomment the following part and set the correct master LSF host name and IP address
#master_host='hostm.example.com'
#master_host_ip='10.115.206.151'
#echo ${master_host_ip} ${master_host} >> /etc/hosts
#echo $master_host > $LSF_ENVDIR/hostregsetup
#lsreghost -s $LSF_ENVDIR/hostregsetup
#Start LSF Daemons in dynamic VM host.
lsadmin limstartup
sleep 5
lsadmin resstartup
badmin hstartup
echo END AT `date '+%Y-%m-%d %H:%M:%S'` >> $logfile
~~~
-------------------------
5.3. Configure Spectrum LSF
-------------------------
5.3.1. Install IBM Spectrum LSF Resource Connector for SoftLayer patch
-------------------------
Steps to install the Resource Connector patch:
1) Log on as root to the LSF master.
2) Install Python 2.6.6 or higher if not already installed.
3) Load shell profile before installation.
- For csh or tcsh: % source $LSF_TOP/conf/cshrc.lsf
- For sh, ksh, or bash: $ . $LSF_TOP/conf/profile.lsf
4) Install the Resource Connector patch
a) Download the LSF Resource Connector for SoftLayer patch from IBM FixCentral
(https://www-945.ibm.com/support/fixcentral) and copy the patch file to the LSF master host.
b) Run patchinstall: $LSF_TOP/10.1/install/patchinstall <patchname>
5) Copy Resource Connector for SoftLayer configuration files
The patch installer copies the following new configuration files for SoftLayer under $LSF_TOP/10.1/resource_connector/softlayer/conf/:
softlayerprov_templates.json
softlayerprov_config.json
credentials
You must move these files to the $LSF_TOP/conf/resource_connector/softlayer/conf directory and change the ownership of the files to the cluster administrator.
6) Update Resource Connector providers list
Edit the file $LSF_TOP/conf/resource_connector/hostProviders.json and ensure the SoftLayer provider is in the list of providers. The following JSON object must be in the list:
{
"name": "softlayer",
"type": "softlayerProv",
"confPath": "resource_connector/softlayer",
"scriptPath": "resource_connector/softlayer"
}
For more information and examples of hostProviders.json options, see the reference in IBM Knowledge Center:
http://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_resource_connector/lsf_rc_hostproviders.dita
5.3.2 Configure the SoftLayer provider plugin
-------------------------
The SoftLayer plugin configuration file can be found in $LSF_TOP/conf/resource_connector/softlayer/conf/softlayerprov_config.json. You must
set the SOFTLAYER_CREDENTIAL_FILE property to point to a file that contains valid SoftLayer API credentials.
Sample configuration file is shown below. For more details, refer to the softlayerprov_config.json example in section 7 of this document.
{
"SOFTLAYER_CREDENTIAL_FILE": "/opt/lsf/conf/resource_connector/softlayer/conf/credentials",
"SOFTLAYER_DOMAIN_NAME": "hostfactory.com"
}
The credentials file is typically stored in the same plugin configuration directory, but can be anywhere in the file system. Its contents are two key-value pairs setting SoftLayer API username and secret key, for example:
softlayer_access_user_name = <your SoftLayer username>
softlayer_secret_api_key = XXXXXXX
You can find the user's API key in the SoftLayer console; select Account -> Users from the menu and click on the View link in the API Key column.
At a minimum the SoftLayer user whose credentials are configured for the plugin requires permissions to order and cancel virtual servers and access the necessary network and storage resources.
5.3.3 Configure the SoftLayer image templates
-------------------------
Create at least one template in the $LSF_TOP/conf/resource_connector/softlayer/conf/softlayerprov_templates.json file. A sample file with a single template is shown below. Ensure that your template attributes accurately reflect virtual server characteristics of the named image.
{
"templates": [
{
"templateId": "Template-1",
"maxNumber": 10,
"attributes": {
"type": ["String", "X86_64"],
"ncpus": ["Numeric", "4"],
"ncores": ["Numeric", "1"],
"nram": ["Numeric", "8192"],
"softlayercomp": ["Boolean", "1"],
"customattr": ["String", "somedata"]
},
"imageId": "lsfslavehosts",
"datacenter": "tor01",
"vlanNumber": "882",
"useHourlyPricing": true,
"localDiskFlag": false,
"privateNetworkOnlyFlag": false,
"dedicatedAccountHostOnlyFlag": false,
"postProvisionURL": "https://10.115.206.151/user_data_wmeta.sh",
"userData": "customattr=somedata"
}
]
}
Attributes "ncpus" and "softlayercomp" are mandatory; note that the resource name of "softlayercomp" must match the resource name you have defined in step 4.1.2.
For more details about the template properties refer to the softlayerprov_templates.json example in section 7 of this document.
5.3.4 Enable LSF Resource Connector
-------------------------
1) Configure the demand calculation scheduler module.
Modify the $LSF_ENVDIR/lsbatch/<cluster_name>/configdir/lsb.modules file to add the new module "schmod_demand" to the PluginModule section.
For example,
Begin PluginModule
SCH_PLUGIN RB_PLUGIN SCH_DISABLE_PHASES
...
schmod_demand () ()
End PluginModule
2) Define a new boolean resource to identify borrowed hosts.
Section 3.2.2 step 7 in this document describes how to define a new resource name provided by each SoftLayer virtual server. Add this new resource to the defined LSF resource list so that it is accepted by LSF.
On the LSF master host, modify the $LSF_TOP/conf/lsf.shared file to add the new boolean resource to the Resource section.
The following is an example using the "softlayercomp" resource name defined previously:
Begin Resource
RESOURCENAME TYPE INTERVAL INCREASING DESCRIPTION
...
softlayercomp Boolean () () (SoftLayer compute host)
End Resource
3) Enable the Resource Connector.
Define the LSB_RC_EXTERNAL_HOST_FLAG parameter in the $LSF_TOP/conf/lsf.conf file to enable the Resource Connector feature.
The value of the parameter must be set to the resource name created in step 2.
For example:
LSB_RC_EXTERNAL_HOST_FLAG=softlayercomp
You can define multiple virtual host templates with different boolean resource flags and assign them to LSB_RC_EXTERNAL_HOST_FLAG, separated by spaces.
4) Define RC_HOSTS in lsb.queues for a queue for which borrowing needs to be enabled.
RC_HOSTS = none | all | flag [flag..]
The flag is a boolean resource subset of the list of boolean resources
defined in LSB_RC_EXTERNAL_HOST_FLAG in lsf.conf. For example,
RC_HOSTS = softlayercomp
If RC_HOSTS is not defined, its default value is "none". That is, borrowing is disabled for that particular queue even if LSB_RC_EXTERNAL_HOST_FLAG is defined in lsf.conf. If the "RC_HOSTS" parameter is not defined in any queue, borrowing does not happen.
5) If LSF on the master host was not installed with the dynamic host enabled with the ENABLE_DYNAMIC_HOSTS = Y parameter in install.config at installation time, then set LSF_DYNAMIC_HOST_WAIT_TIME = <seconds> in lsf.conf
To ensure the smooth operation of the integration, set LSF_DYNAMIC_HOST_WAIT_TIME to a small value such as 1 or 2 seconds.
6) Enable registration of hosts that may not be resolvable via DNS.
Add the following configuration parameter to $LSF_TOP/conf/lsf.conf
LSF_REG_FLOAT_HOSTS=Y
This allows LIM on the server to accept cluster registration requests from hosts that run `lsreghost` to join the cluster.
7) This step is optional. You can define the variable LSF_HOST_ADDR_RANGE in $LSF_TOP/conf/lsf.cluster.<clustername> file for increased security.
Begin Parameters
LSF_HOST_ADDR_RANGE=*.*.*.* #IP address range of the SoftLayer virtual servers
End Parameters
Only hosts with IP addresses within the specified range can be added to or removed from the instance.
8) Restart LSF daemons on the master host for the changes to take effect.
lsadmin limrestart
badmin mbdrestart
For more details please review instructions in the Spectrum LSF Knowledge Center at https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_resource_connector/lsf_rc_update_lsfconfig.html
=========================
6. Operation
=========================
-------------------------
6.1. Check the Resource Connector status
-------------------------
On the LSF master host, verify that the 'ebrokerd' process is running after enabling the Resource Connector.
ps -ef | grep ebrokerd
Review ebrokerd and Resource Connector plugin log files in $LSF_TOP/log/ebrokerd.log.<hostname> and $LSF_TOP/log/softlayer-provider.<hostname>.log for error messages.
-------------------------
6.2. Submit jobs to request SoftLayer virtual servers
-------------------------
In this section, "slhost01" is a sample virtual server in SoftLayer.
1) Use bsub to submit jobs that require hosts provisioned by SoftLayer.
The following bsub command with no options submits a job that will trigger a borrow demand when there are no available resources in LSF cluster:
$ bsub myjob
Alternatively, you can use the "softlayercomp" resource in its select[] resource requirement. Since the "softlayercomp" resource is defined in a template as a boolean attribute, it will trigger a demand:
$ bsub -R "select[softlayercomp]" myjob
A virtual server order will be sent to SoftLayer, and after few minutes the new host will join the cluster. Note that SoftLayer provisioning process can take up to 20 minutes.
2) Use bhosts to monitor borrowed hosts. Host status becomes "ok" when it joins the LSF cluster as a dynamic host.
$ bhosts -a
HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV
lsfmaster ok - 1 0 0 0 0 0
slhost01 ok - 1 1 1 0 0 0
Verify that this job runs on slhost01.
3) LSF can dispatch subsequent jobs to an existing SoftLayer virtual server if there is demand and job resource requirements match the host template. If there is no additional demand for the SoftLayer compute host, LSF will relinquish the host and a cancel request will be sent to SoftLayer. After some time the virtual server will become unavailable.
$ bhosts -a
HOST_NAME STATUS JL/U MAX NJOBS RUN SSUSP USUSP RSV
lsfmaster ok - 1 0 0 0 0 0
slhost01 unavail - 1 0 0 0 0 0
=========================
7. Logging and troubleshooting
=========================
-------------------------
7.1 Log files for LSF
-------------------------
To change the log level or log classes for LSF, update the following parameters in the $LSF_TOP/conf/lsf.conf file:
* LSF_LOG_MASK
* LSB_DEBUG_MBD
* LSB_DEBUG_EBROKERD
For example, the log level is set to LOG_INFO, and the debugging log class for mbatchd and ebrokerd is set to LC2_COMM:
LSF_LOG_MASK=LOG_INFO
LSB_DEBUG_MBD="LC2_COMM"
LSB_DEBUG_EBROKERD="LC2_COMM"
-------------------------
7.2 Log files for the Resource Connector
-------------------------
To change the log level for the Resource Connector, update the LogLevel parameter in the $LSF_TOP/conf/resource_connector/softlayer/conf/softlayerprov_config.json file.
For example, to set the log level to DEBUG:
{
"LogLevel": "DEBUG",
}
Log files for the Resource Connector can be found in the $LSF_TOP/log directory.
-------------------------
7.3 Persistent files for the Resource Connector
-------------------------
The Resource Connector saves some state information for synchronization with LSF after failover or restart. This information is saved in persistence files, which are in the $LSB_SHAREDIR/<cluster_name>/resource_connector/ directory.
=========================
8. Configuration reference
=========================
-------------------------
softlayerprov_config.json
-------------------------
The softlayerprov_config.json file contains administrative settings for the Resource Connector SoftLayer plugin. The file is in the $LSF_TOP/conf/resource_connector/softlayer/conf/ directory.
The softlayerprov_config.json parameters are as follows (please note that parameter names are case-sensitive):
LogLevel
--------
The log level for this host provider. Valid log levels are: INFO, DEBUG, WARN.
SOFTLAYER_CREDENTIAL_FILE
--------
Required. Absolute path to the file containing SoftLayer API credentials.
SOFTLAYER_DOMAIN_NAME
--------
Optional. Sets the domain name of virtual hosts requested by the plugin.
UserDataTimeout
--------
Optional. Sets the timeout value, in seconds, for SoftLayer API calls to set user metadata. Default is 120 seconds.
-------------------------
softlayerprov_templates.json
-------------------------
The softlayerprov_templates.json file is the primary way to define the mapping between LSF resource demand requests and SoftLayer virtual servers. The template represents a set of hosts that share some attributes in common such as the number of CPUs, the amount of available memory, the installed software stack, operating system, and other attributes.
LSF requests resources from the Resource Connector by specifying the number of instances of a particular template that it requires to satisfy its demand. The Resource Connector uses the definitions in this file to map this demand into a set of virtual server orders on SoftLayer.
The softlayerprov_templates.json file is located in the $LSF_TOP/conf/resource_connector/softlayer/conf directory.
The file contains a JSON-defined list called 'templates'. Each templates list is an object containing the following parameters:
templateId
----
Required. A string that uniquely identifies the template.
attributes
----------
Required. A list of attributes representing the hosts in the template from the LSF point of view. LSF attempts to place its pending workload on hypothetical hosts matching these attributes in order to calculate how many instances of each template to request.
The format of each attribute string in the list is
"<attribute_name>": ["<attribute_type>", "<attribute_value>"]
"<attribute_name>" is the LSF resource name, (for example, "type" or "ncores"). The attribute name must either be a built-in resource (such r15s or type) or defined in the Resource section in the $LSF_TOP/conf/lsf.shared file on the LSF master host.
"<attribute_type>" can be either 'Boolean', 'String', or 'Numeric' and must correspond to the corresponding resource definition in lsf.shared.
"<attribute_value>" is the value of the resource provided by hosts. For Boolean resources, use '1' to define the presence of the resource and '0' to define its absence. For Numeric resources, specify a range using '[min:max]'
Ensure that the attributes accurately reflect actual configuration of the virtual server image used for the template. If, for example, the attribute definition specifies hosts with ncpus=4, but the actual virtual servers provided by SoftLayer have ncpus=2, LSF's demand calculation will not be accurate.
Notes:
* "type" attribute: Default value is LSF_RESOURCE_CONNECTOR_DEFAULT_HOST_TYPE (see in lsf.conf).
* "ncpus" attribute: Default value is 1.
imageId
----------
Required. Name of the SoftLayer image to be used for provisioning of virtual servers.
maxNumber
----------
Optional. An integer value determining the maximum number of virtual servers that can be provisioned using this template. This value limits the number of dynamic hosts that can be requested by LSF.
postProvisionURL
----------
Required. This HTTPS URL must point to the postprovision script defined by you.
userData
----------
Required. A string representing a list of keys and their values.
The format of userData is
<key>=<value_list>;<key>=<value_list>
<key> is key name of userData variable, such as "packages" or "volume" or "zone" or "templateName".
<value_list> is list of userData values separated by comma, for example, "package1, package2".
Once userData is defined, it is divided into keys/values and exported as the virtual server environment variables. For example, if userData is defined as:
packages=M,N;logfile=X
It is exported as the following environment variables in the instances:
packages=M,N
logfile=X
These variables can be read by the provisioning script in the instance as keys: "packages" and "logfile"
privateNetworkOnlyFlag
----------
Must be set to "false" to enable public network on the virtual host, if the LSF cluster master is not a SoftLayer host accessible via the SoftLayer internal VLAN.
datacenter
----------
Required. Indicates the SoftLayer data center where virtual servers should be provisioned.
vlanNumber
----------
Required. Must be set to the internal VLAN number in the appropriate data center.
-------------------------
lsf.conf
-------------------------
Add the following parameters and values to the $LSF_TOP/conf/lsf.conf file on the Resource Connector:
LSF_EXTERNAL_HOST_FLAG
-------------------------
Syntax
LSF_EXTERNAL_HOST_FLAG = String ...
Description
Setting this parameter enables the overall Resource Connector feature.
Specify a list of String resource names that identify borrowed hosts. Any hosts providing a resource from the list is initially closed by LSF at startup, and only opened when the Resource Connector informs LSF that the hosts have been allocated to it.
Run badmin mbdrestart for the changes to take effect.
Example
LSF_EXTERNAL_HOST_FLAG=slcompute
Default
Not defined
LSB_RSRC_CONNECTOR_DEFAULT_HOST_TYPE
-------------------------
Syntax
LSB_RSRC_CONNECTOR_DEFAULT_HOST_TYPE = string
Description
Specifies the default host type to use for a template if the 'type' attribute is not defined on a template in the softlayerprov_templates.json file.
Default
X86_64
LSB_RC_QUERY_INTERVAL
-------------------------
Syntax
LSB_RC_QUERY_INTERVAL = seconds
Description
The interval in seconds that the Resource Connector checks host status and asynchronous request results from providers.
Run badmin mbdrestart for the changes to take effect.
Default
30 seconds
=========================
9. Copyright
=========================
© Copyright IBM Corporation 2014,2017
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM®, the IBM logo and ibm.com® are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the
Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml