IBM Power Systems Scale-out LC Server Firmware

Applies to:  S922LC (9006-22C) and Elastic Storage Server (5104-22C)

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.

 

Contents

1.0 Systems Affected

1.1 Minimum ipmitool Code Level

1.2 Minimum Browser levels for BMC Web GUI

1.3 Fix level Information on IBM Open Power Components and Operating systems

2.0 Important Information

3.0 Firmware Information

3.1 Firmware Information and Description 

4.0 Operating System Information

4.1 Linux Operating System

4.2 How to Determine the Level of a Linux Operating System

4.3 How to Determine if the opal-prd (Processor Recovery Diagnostics) package is installed

5.0 How to Determine The Currently Installed Firmware Level

6.0 Downloading the Firmware Package

7.0 Installing the Firmware

7.1 IBM Power Systems Firmware maintenance

7.2 Updating the System Firmware with the pUpdate utility

7.3 Supporting Diagnostics

7.4 Installing ipmitool on Ubuntu

7.5 Updating the System Firmware using the BMC Web GUI

7.6 System I/O Firmware

8.0 System Management and Virtualization

8.1 BMC Service Processor IPMI and Web GUI Access

8.2 Open Power Abstraction Layer (OPAL)

8.3 Intelligent Platform Management Interface (IPMI)

8.4 Petitboot bootloader

8.5 KVM Options for these Power LC servers

9.0 Quick Start Guide for Installing Linux on these LC servers

10.0 Change History

 

1.0 Systems Affected

This package provides firmware for the Power Systems Scale-out LC S922LC (9006-22C) and Elastic Storage Server model (5104-22C).

 

The firmware level in this package is:

 

These are the following images:

BMC Firmware:     SMT_P9_114.bin

PNOR Firmware:     p9dsu01302018_IBM_sign.pnor

pUpdate Utility for powerpc:  pUpdate

 

 

Details on the package binaries are included in section 3.1

1.1 Minimum ipmitool Code Level

This section specifies the "Minimum ipmitool Code Level" required by the System Firmware to perform firmware installations and managing the system.  Open Power requires ipmitool level v1.8.15 to execute correctly on the V0.19 or later firmware.

 

Verify your ipmitool level on your linux workstation using the following commands:

 

bash-4.1$ ipmitool -V

ipmitool version 1.8.15

 

If you need to update or add impitool to your Linux workstation , you can compile ipmitool (current level 1.8.15) for Linux as follows from the Sourceforge:

 

1.1.1  Download ipmitool tar from http://sourceforge.net/projects/ipmitool/  to your linux system

1.1.2  Extract tarball on linux system

1.1.3  cd to top-level directory

1.1.4 ./configure

1.1.5  make

1.1.6  ipmitool will be under src/ipmitool        

 

You may also get the ipmitool package directly from your workstation linux packages such as Ubuntu 14.04.3:

 

sudo apt-get install ipmitool

 

1.2 Minimum Browser levels for BMC Web GUI

The BMC Web GUI is a web-based application that works within a browser.   Supported browser levels are shown below with Chrome being the preferred browser:

  1.  

1.3 Fix level Information on IBM Open Power Components and Operating systems

For specific fix level information on key components of IBM Power Systems servers and Linux operating systems, please refer to the documentation in the IBM Knowledge Center.

Here are the links for the LC servers:

9006-22C:   http://www.ibm.com/support/knowledgecenter/POWER9/p9hdx/9006_22c_landing.htm

5104-22C:   https://www.ibm.com/support/knowledgecenter/en/POWER9/p9hdx/5104_22c_landing.htm

2.0 Important Information


Downgrading firmware from any given release level to an earlier release level is not recommended. 

If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.

Concurrent Firmware Updates not available for LC servers.

Concurrent system firmware update is not supported on these LC servers.

3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For the LC server systems, the installation of system firmware is always disruptive.

 

3.1 Firmware Information and Description

The xxx.pnor  file updates the primary side of the PNOR.  The yyy.bin updates the primary side of the BMC only.  The golden sides are unchanged.

 

Filename

Size

Checksum

p9dsu01302018_IBM_sign.pnor

33554560

 0e90b0fb16072d3ba3cac6f83ca0dda5

SMT_P9_114.bin

33554432

 57bf02f42cb1ea201088beaf984a39b3

pUpdate

1058520

  26d6bc1db003700ffd13c40a953b1f0b

 

Note: The Checksum can be found by running the Linux/Unix/AIX md5sum command against the file (all 32 characters of the checksum are listed), ie: md5sum xxx.pnor.  

 

After a successful update to this firmware level, the PNOR components and BMC should be at the following levels.  The ipmitool "fru" command can be used to display FRU ID 47 and the ipmitool "mc info" command can be used to display the BMC level.

 

Note:  FRU information for the PNOR level does not show the updated levels via the fru command until the system has been booted once at the updated level.

 

PNOR firmware levels from FRU ID 47 inventory list for driver:  

 

 Product Name          : OpenPOWER Firmware

 Product Version       : open-power-SUPERMICRO-P9DSU-V0.24-20180130

 Product Extra         :        op-build-8a84c4b

 Product Extra         :        skiboot-v5.9.8-pc7e3644

 Product Extra         :        hostboot-89d9dff

 Product Extra         :        linux-4.13.16-openpower1-p255dad4

 Product Extra         :        petitboot-v1.6.3-p2cfc12f

 Product Extra         :        machine-xml-bbf0d7a

 Product Extra         :        occ-84f3564

 Product Extra         :        sbe-99e2fe2

 

 

BMC Level:                        

   

Display BMC firmware level using the "ipmitool mc info | grep Firmware" command:

 

Firmware Revision         : 1.14

 

 

 

 

V0.24-20180130
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url: 
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

V0.24-20180130

02/16/2018

 

Impact:  Security     Severity:  SPE

 

This Service Pack includes updates in response to Recent Security Vulnerabilities, New Features & Functions and System Firmware Updates.  

 

Details of each are below:

 

Response for Recent Security Vulnerabilities

 

In response to recently reported security vulnerabilities, this firmware update is being released to address Common Vulnerabilities and Exposures issue numbers CVE-2017-5715, CVE-2017-5753 and CVE-2017-5754.  Operating System updates are required in conjunction with this FW level for CVE-2017-5753 and CVE-2017-5754.

 

New features and functions (not related to above CVE’s)

 

Support for DD2.1 versions of the Power 9 processors

 

Support was added for compile time certificates to STUNNEL allow full 3650 days validity.  This change allows JAR to have the newest STUNNEL certificates for each build.

 

System firmware changes that affect all systems

 

A problem was fixed for the ipmitool dcmi power limit that was ignoring action states.  With the fix, the action is set as shown for this example:

"ipmitool dcmi power  set_limit action power_off"

    Current Limit State: Power Limit Active

    Exception actions:   Hard Power Off & Log Event to SEL

    Power Limit:         1500 Watts

    Correction time:     0 milliseconds

    Sampling period:     0 seconds

 

A problem was fixed for the ipmitool chassis poh (Power On Hours) counter getting reset to zero.  The counter should have been stored in non-volatile storage to increment the hour count whenever the system is powered up and it should not reset to zero when the system is powered off or the BMC is reset.

       

A problem was fixed for an intermittent hang in iKVM when used in full-screen mode.

   

A problem was fixed for improving the security of BMC SSH session by adding support for CBC Ciphers and enabling Weak MAC algorithms.

       

A problem was fixed for a checkstop and re-ipl occurring during a normal reboot of the system.

       

A problem was fixed for an intermittent hang on a re-IPL of a system.  This can occur in ISTEP 10.10  (proc_pcie_scominit) for around 600 seconds, which then recovers to complete the IPL.  This problem may also have a symptom of an ipmitool get power limit information fail with reason code 0x250a if attempted during the IPL stall state.  It is also possible the IPL could stall at ISTEP 21.1 during host_runtime_setup.

 

A problem was fixed for an unexpected reboot of the system and NTP is enabled from the BMC web interface.

 

A problem was fixed for intermittent hangs for PCIe adapters going through an error reset.  The wrong PE number was being configured for the device but the PE mapping has been corrected.  Larger PE numbers were truncated causing additional PE errors and freeze errors.

 

A problem was fixed for intermittent host checkstops caused by NCU and PCI timeout mismatches.  PCI timeouts that are longer than NCU timeouts may cause checkstops on the host.

 

A problem was fixed for incorrect guards on the MCA when there are faulty DIMMs.

 

A problem was fixed for not setting the guard/deconfiguration bits correctly on bad devices, allowing these to be accessed during the reconfiguration loop of the IPL.  This can lead to errors that fail the IPL.

 

A problem was fixed for an IPL failing in step 6.11 after encountering a checkstop.

 

A problem was fixed to disable out-of-store behavior in PCIe devices and improve MMIO packet rate performance

 

V0.19-20171030D

12/01/2017

Impact:  New       Severity:  New

GA Level

 

4.0 Operating System Information

OS levels supported by the LC servers:

 - Red Hat Enterprise Linux 7.4 for IBM Power LE (POWER9)

 -Red Hat Enterprise Linux 7.4 for IBM Power LE (POWER9) /RHEV 4.1

–Ubuntu 16.04.3

–Ubuntu 16.04.3 / KVM

–Ubuntu 17.04

–Ubuntu 17.04 / KVM

 

IBM Power LC servers supports Linux which provides a UNIX like implementation across many computer architectures.  Linux supports almost all of the Power System I/O and the configurator verifies support on order.  For more information about the software that is available on IBM Power Systems, see the Linux on IBM Power Systems website:

 http://www.ibm.com/systems/power/software/linux/index.html

 

 

4.1 Linux Operating System

The Linux operating system is an open source, cross-platform OS. It is supported on every Power Systems server IBM sells. Linux on Power Systems is the only Linux infrastructure that offers both scale-out and scale-up choices.  One supported version of Linux on the IBM Power LC is Ubuntu Server 17.04 for IBM POWER9.  For more information about Ubuntu Server for Ubuntu for POWER9 see the following website.

https://wiki.ubuntu.com/ppc64el

 

Another supported version of Linux on the Power LC servers is  Red Hat Enterprise Linux 7.4 LE.  For additional questions about the availability of this release and supported Power servers, consult the Red Hat Hardware Catalog at https://hardware.redhat.com.

 

For information about the PowerLinux Community, see the following website:

https://www.ibm.com/developerworks/group/tpl

 

For information about the features and external devices that are supported by Linux, see this website:

http://www.ibm.com/systems/power/software/linux/index.html

4.2 How to Determine the Level of a Linux Operating System

 

Use one of the following commands at the Linux command prompt to determine the current Linux level:

 

 

The output string from the command will provide the Linux version level.

 

4.3 How to Determine if the opal-prd (Processor Recovery Diagnostics) package is installed

The opal-prd package on the Linux system collects the OPAL Processor Recovery Diagnostics messages to log file /var/log/syslog.  It is recommended that this package be installed if it is not already present as it will help with maintaining the system processors by alerting the users to processor maintenance when needed.

 

On Ubuntu Linux, perform command, dpkg -l "opal-prd".  The output shows whether the package is installed on your system by marking it with ii (installed) and un (not installed).

 

This package provides a daemon to load and run the OpenPower firmware's Processor Recovery Diagnostics binary. This is responsible for run-time maintenance of Power hardware.  

If the package is not installed on your system, the following command can be run on Ubuntu to install it:

sudo apt-get install opal-prd

 

On Red Hat Linux, perform command "rpm -qa | grep -i opal-prd ".  The command output indicates the package is installed on your system if the rpm for opal-prd is found and displayed.  This package provides a daemon to load and run the OpenPower firmware's Processor Recovery Diagnostics binary. This is responsible for run-time maintenance of Power hardware.   If the package is not installed on your system, the following command can be run on Red Hat to install it:

        sudo yum update opal-prd

 

5.0 How to Determine The Currently Installed Firmware Level

 

Use the ipmtool "fru" command or the BMC Web GUI FRU option to look at product details of FRU 47.

 

ipmitool -I lanplus -H <bmc host IP address> -P admin -U ADMIN fru print 47

 

6.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

 

7.0 Installing the Firmware

7.1  IBM Power Systems Firmware maintenance

The updating and upgrading of system firmware depends on several factors, such as the current firmware that is installed, and what operating systems is running on the system.

These scenarios and the associated installation instructions are comprehensively outlined in the firmware section of Fix Central, found at the following website:

http://www.ibm.com/support/fixcentral/

 

Any hardware failures should be resolved before proceeding with the firmware updates to help insure the system will not be running degraded after the updates.

7.2 Updating the System Firmware with the pUpdate utility

The pUpdate utility is provided with the firmware update files from IBM Fix Central.  It can be used to perform in-band (from the host OS), in-band update recovery, and out-of-band updates by selecting either the "-i usb" , "-i bt" or  "-i lan" parameters, respectively on the command invocation. The code update needs to be done in two steps:  1) Update the BMC firmware and 2) Update the CEC PNOR for the hostboot and the OPAL components.  It is recommended that the BMC be updated first unless otherwise specified in the firmware install instructions.

 

Before using the pUpdate command on the host,  make sure that the ipmi driver is loaded in the kernel and the ipmi service is started.

 

Note: For updates that use the "usb" or "bt" pUpdate option, you must use the root user ID and password to log in to the host operating system. After you log in to the host operating system, ensure that the IPMI service is activated.

# chkconfig ipmi on

# service ipmi start

 

For more information about activating the IPMI service, see the OpenIPMI Driver: https://www.ibm.com/support/knowledgecenter/POWER8/p8eih/p8eih_ipmi_open_driver.htm

 

For in-band update, use the following "-i usb" invocation of pUpdate:

 

BMC update:  "pUpdate -f bmc.bin -i usb", where bmc.bin is the name and location of the BMC image file.

 

PNOR update:  "pUpdate -pnor pnor.bin -i usb", where pnor.bin is the name and location of the PNOR image file.

 

If the in-band update fails on the BMC, use the recovery option with the Block Transfer (bt) invocation of pUpdate:

 

BMC update: "pUpdate -f bmc.bin -i bt -r y" where bmc.bin is the name  and location of the BMC image file.

 

PNOR update:" pUpdate -pnor pnor.bin -i bt " where pnor.bin is the name and location of the PNOR image file.

 

For more information on BMC recovery steps, refer to the following link in the IBM Knowledge Center:

https://www.ibm.com/support/knowledgecenter/POWER8/p8eis/p8eis_console_problem.htm

 

If the host is not booted, a network connection can be made to the BMC and an out-of-band update done with the following LAN invocation from a Linux companion system:

 

BMC update: " pUpdate -f bmc.bin -i lan -h xx.xx.xx.xx  -u ADMIN -p ADMIN -r y" where bmc.bin is the name and location of the BMC image file, xx.xx.xx.xx is the IP address of the BMC.

 

PNOR update:  "pUpdate -pnor pnor.bin -i lan -h xx.xx.xx.xx -u ADMIN -p ADMIN " where pnor.bin is the name  and location of the PNOR image file and xx.xx.xx.xx is the IP address of the BMC.

 

For more details on how to use the pUpdate utility, refer to the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eit/p9eit_update_firmware_pupdate.htm

7.3 Supporting Diagnostics

You can use diagnostic utilities to diagnose adapter problems.

 

For more details on how to use the diagnostic utilities,  refer to the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eit/p9eit_diags_kickoff.htm

7.4  Installing ipmitool on Ubuntu

Open Power requires Source Forge ipmitool level v1.8.15 to execute correctly on the P9 V0.19-20171030D and later firmware.

7.5  Updating the System Firmware using the BMC Web GUI

Another method to update the system firmware is by using the baseboard management controller (BMC).

The system firmware is a combination of the BMC firmware and the PNOR firmware. To update the system firmware, update both the BMC firmware and the PNOR firmware by using the BMC.

Note: System firmware update from the BMC Web GUI is only supported on Google Chrome and Mozilla Firefox browsers.

 

Complete the following steps to update the BMC firmware:

1.        Log in to the BMC by entering the user name and password. Then, press Enter.

2.        From the Maintenance list on the BMC dashboard, select BMC Update.

3.        In the BMC Update window, select Enter Update Mode. Click OK.

4.        In the BMC Upload window, choose the .bin file from your local system folder and click Upload Firmware. Wait for the file to be uploaded. Then, click OK.

5.        The existing and new versions of the BMC firmware are displayed.  Ensure that the Preserve Configuration check box is selected and the Preserve SDR check box is not selected. Click Start Upgrade.

Note: You cannot perform other activities by using the BMC interface until the firmware update is complete.

6.        The upgrade progress of the firmware update is displayed. After the BMC update is complete, the system is restarted.

7.        After the restart of the system is complete, verify the firmware revision level in the System menu of the BMC dashboard.

 

Complete the following steps to update the PNOR firmware:

1.        Log in to the BMC by entering the user name and password. Then, press Enter.

2.        From the Maintenance list on the dashboard, select PNOR Update.

3.        In the PNOR Upload window, choose the .pnor file from your local system folder and click Upload PNOR. Wait for the file to be uploaded. Then, click OK.

4.        The existing and new dates of the PNOR firmware are displayed. Click Start Upgrade.

Note: You cannot perform other activities by using the BMC interface until the PNOR update is complete.

5.        The progress of the PNOR update is displayed. After the PNOR update is completed,  the system must be restarted to finish installation of the new PNOR firmware.

 

For more information on updating the firmware using the BMC, refer to the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eit/p9eit_update_firmware_bmc.htm

7.6  System I/O Firmware

System I/O devices have firmware that can be updated.  

Please see the appropriate IBM Knowledge Center for these servers for applicable I/O firmware update information:

 

9006-22C:

http://www.ibm.com/support/knowledgecenter/POWER9/p9hdx/9006_22c_landing.htm

5104-22C:

https://www.ibm.com/support/knowledgecenter/en/POWER9/p9hdx/5104_22c_landing.htm

8.0 System Management and Virtualization

The service processor, or baseboard management controller (BMC), provides a hypervisor and operating system-independent layer that uses the robust error detection and self-healing functions that are built into the POWER processor and memory buffer modules. Open power application layer (OPAL) is the system firmware in the stack of POWER processor-based Linux-only servers.

 

8.1  BMC Service Processor IPMI and Web GUI Access

The service processor, or baseboard management controller (BMC), is the primary control for autonomous sensor monitoring and event logging features on the LC server.

The BMC supports the Intelligent Platform Management Interface (IPMI) for system monitoring and management.  The BMC monitors the operation of the firmware during the boot process and also monitors the OPAL hypervisor for termination.  The firmware code update is supported through the BMC and Intelligent Platform Monitoring Interface (IPMI) and the BMC Web GUI  The GUI console is accessed using a web browser with a "http:" connection to port.  See section 1.2 for the supported browsers that can be used with BMC Web GUI.  

 

8.2 Open Power Abstraction Layer (OPAL)

The Open Power Abstraction Layer (OPAL) provides hardware abstraction and run time services to the running host Operating System.

For these LC servers,  only the OPAL bare-metal installs can be used.  A KVM can be used on top of the installed OS to run Linux virtual guest OS machines.

 

Find out more about OPAL skiboot here:

https://github.com/open-power/skiboot

 

8.3 Intelligent Platform Management Interface (IPMI)

The Intelligent Platform Management Interface (IPMI) is an open standard for monitoring, logging, recovery, inventory, and control of hardware that is implemented independent of the main CPU, BIOS, and OS. It is the default console to use when you configure PowerKVM.  These LC servers provide one 10M/100M baseT IPMI port.

The ipmitool is a utility for managing and configuring devices that support IPMI. It provides a simple command-line interface to the service processor.  You can install the ipmitool from the Linux distribution packages in your workstation, sourceforge.net, or another server (preferably on the same network as the installed server). For example, in Ubuntu, use this command:

$ sudo apt-get install ipmitool

For installing ipmitool from sourceforge, please see section 1.1 "Minimum ipmitool Code Level".

 

For more information about ipmitool, there are several good references for ipmitool commands:

 

  1. 1.The man page  

  2. 2.The built-in command line help provides a list of IPMItool commands:
    # ipmitool help 

  3. 3.You can also get help for many specific IPMItool commands by adding the word help after the command:
    # ipmitool channel help 

  4. 4.For a list of common ipmitool commands and help on each, you may use the following link:  
    www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabpcommonipmi.htm 

     

 

To connect to your host system with IPMI, you need to know the IP address of the server and have

a valid password. To power on the server with the ipmitool, follow these steps:

1. Open a terminal program.

2. Power on your server with the ipmitool:

ipmitool -I lanplus -H bmc_ip_address -P ipmi_password power on

3. Activate your IPMI console:

ipmitool -I lanplus -H bmc_ip_address -P ipmi_password sol activate

 

8.4 Petitboot bootloader

Petitboot is a kexec based bootloader used by IBM POWER9 systems for doing the bare-metal installs on these LC servers.

After the POWER9 system powers on, the petitboot bootloader scans local boot devices and network interfaces to find boot options that are available to the system. Petitboot returns a list of boot options that are available to the system. If you are using a static IP or if you did not provide boot arguments in your network boot server, you must provide the details to petitboot.  You can configure petitboot to find your boot with the following instructions:

https://www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitbootadvanced.htm

 

You can edit petitboot configuration options, change the amount of time before Petitboot automatically boots, etc. with these instructions:

https://www.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitbootconfig.htm

 

After you select to boot the ISO media for the Linux distribution of your choice, the installer wizard for that Linux distribution walks you through the steps to set up disk options, your root password, time zones, and so on.

You can read more about the petitboot bootloader program here:

https://www.kernel.org/pub/linux/kernel/people/geoff/petitboot/petitboot.html

 

 

8.5 KVM Options for these Power LC servers

Kernel-based Virtual Machine (KVM) is a cross-platform, open source hypervisor that provides enterprise-class performance, scalability and security to run Linux and other workloads on a range of processor architectures.  For these LC servers, Ubuntu KVM or Red Hat Enterprise Virtualization (RHEV)  may be installed on top of a bare-metal install of Ubuntu or RHEL, respectively.

 

Red Hat Enterprise Virtualization (RHEV) for IBM Power is an enterprise virtualization product produced by Red Hat, based on the KVM hypervisor. For more information, go to this link on the Red Hat portal:

 

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Release_Notes/chap-RHEV_for_IBM_Power.html

 

Ubuntu KVM is configured by installing the missing virtualization packages (qemu-user qemu-utils cloud-image-utils qemu-system-ppc qemu-slof libvirt-bin numactl); adding users in a KVM group; disabling the SMT mode of the cpu using the ppc64_cpu tool; and enabling the KVM module in the kernel.  For more information on how to complete these steps, refer to this link in the Ubuntu wiki: https://wiki.ubuntu.com/ppc64el/CommonQuestions#How_to_use_Ubuntu_as_a_hypervisor.3F  

 

IBM PowerKVM is not supported on these LC servers.

 

Note: These Power LC servers and their KVM options do not support AIX or IBM i guest VMs and cannot be managed by a HMC

 

For additional questions about the availability of this release and supported Power servers, consult the Red Hat Hardware Catalog:

https://hardware.redhat.com/.

 

9.0 Quick Start Guide for Installing Linux on these LC servers

This guide helps you install Ubuntu on a Linux on a Power Systems server.

Overview

Use the information found in http://www.ibm.com/support/knowledgecenter/linuxonibm/liabw/liabwkickoff.htm  to install Linux, in this case Ubuntu, on a non-virtualized (bare metal) IBM Power LC server.  Note that the choice of PowerKVM is offered in the link but that is not a supported OS for these LC servers.

 

 

 

10.0 Change History

 

Date

Description

02/16/2018

 V0.24-20180130 release

12/01/2017

 V0.19-20171030D release for Power 9 S922LC (9006-22C) and Elastic Storage Server (5104-22C)