IBM Power8 Systems S822LC Server Firmware

Applies to:  S822LC (8335-GCA) and S822LC (8335-GTA)

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.

 

Contents

1.0 Systems Affected

1.1 Minimum ipmitool Code Level

1.2 Minimum Browser levels for BMC ASM (Advanced System Management) Console

1.3 Fix level Information on IBM Open Power Components and Operating systems

1.4 Steps needed to keep Redundant Power Supply Policy Enabled on 8335-GCA systems

1.5 PowerKVM 3.1 with Ubuntu guests supported on 8335-GCA systems

1.6 "Deactivate" Power Limit fails at system runtime so use circumvention

1.7 Minimum Firmware level of OP810.31 needed for NVMe flash adapter cards #EC54 and #EC56

1.8 Minimum xCAT level 2.13.4 for use in firmware updates

2.0 Important Information

3.0 Firmware Information

3.1 Firmware Information and Description 

4.0 Operating System Information

4.1 Linux Operating System

4.2 How to Determine the Level of a Linux Operating System

4.3 How to Determine if the opal-prd (Processor Recovery Diagnostics) package is installed

5.0 How to Determine The Currently Installed Firmware Level

6.0 Downloading the Firmware Package

7.0 Installing the Firmware

7.1 IBM Power Systems Firmware maintenance

7.2 Updating the System Firmware with ipmitool

7.2.1 Return codes from the ipmitool "hpm upgrade" command

7.3 Installing ipmitool on Ubuntu

7.4 Updating the System Firmware using the BMC Advanced System Management (ASM)

8.0 System Management and Virtualization

8.1 BMC Service Processor IPMI and ASM Access

8.2 Open Power Abstraction Layer (OPAL) Hypervisor

8.3 Intelligent Platform Management Interface (IPMI)

8.4 Petitboot bootloader

8.5 IBM PowerKVM

8.5.1 Download IBM PowerKVM

8.5.2 Update PowerKVM

8.5.3 Upgrade PowerKVM

8.6 Kimchi management tool

8.7 IBM PowerVC

8.7.1 PowerVC Benefits

9.0 Quick Start Guide for Installing Linux on LC servers

10.0 Change History

 

1.0 Systems Affected

This package provides firmware for Power Systems LC S822LC (8335-GCA)  and S822LC(8335-GTA) servers only.

The firmware level in this package is:

 

1.1 Minimum ipmitool Code Level

This section specifies the "Minimum ipmitool Code Level" required by the System Firmware to perform firmware installations and managing the system.  Open Power requires ipmitool level v1.8.15 to execute correctly on the OP810/OP820 firmware, especially the ipmitool code update function.

If ipmitool is being run from Ubuntu, the minimum level is 1.8.13-1ubuntu0.5 and should be obtained directly from the Ubuntu packages and not Source Forge (if you want to do in-band firmware updates from Ubuntu) as shown in the second example below.  The Source Forge version v1.8.15 loaded on Ubuntu would not be able to do the firmware update.  However,  there are other problems in 1.8.13-1ubuntu0.5 that affect functions that control the system.  If these other functions are the priority, use the Source Forge version of ipmitool for Ubuntu.

 

 Verify your ipmitool level on your linux workstation using the following commands:

 

bash-4.1$ ipmitool -V

ipmitool version 1.8.15

 

If you are need to update or add impitool to your Linux workstation , you can compile ipmitool (current level 1.8.15) for Linux as follows from the Sourceforge:

 

1.1.1  Download impitool tar from http://sourceforge.net/projects/ipmitool/  to  your linux system

1.1.2  Extract tarball on linux system

1.1.3  cd to top-level directory

1.1.4 ./configure

1.1.5  make

1.1.6  ipmitool will be under src/ipmitool        

 

You may also get the ipmitool package directly from your workstation linux packages such as Ubuntu 14.04.3:

 

sudo apt-get install ipmitool

 

1.2 Minimum Browser levels for BMC ASM (Advanced System Management) Console

The BMC ASM is a  web-based application that works within a browser.   Supported browsers levels are shown below with Chrome being the preferred browser:

  1.  

1.3 Fix level Information on IBM Open Power Components and Operating systems

For specific fix level information on key components of IBM Power Systems LC and Linux operating systems,  please refer to the documentation in the IBM Knowledge Center for
the S822LC (8335-GCA) and the S822LC (8335-GTA):

 http://www-01.ibm.com/support/knowledgecenter/P8DEA/p8hdx/8335_gca_landing.htm

 http://www-01.ibm.com/support/knowledgecenter/P8DEA/p8hdx/8335_gta_landing.htm

 

1.4 Steps needed to keep Redundant Power Supply Policy Enabled on 8335-GCA systems

For OP810.10,  8335-GCA systems with redundant power supplies are shipped with redundant power supply policy as "Enabled".  However,  because of an error, this setting is lost on a reset of the BMC service processor or an AC power-cycle and the system reverts to "Disabled" and will IPL without power supply redundancy.  To circumvent this problem, re-apply redundancy after every reset of the BMC or AC power-cycle using the following ipmitool command (assuming "admin" is the ADMIN password on the BMC):

 

ipmitool -H <BMC IP address> -U ADMIN -P admin raw 0x3A 0x18 0x01 0x02

 

After running the above command, the user can IPL the system.  (If the system is already running, the command will not take effect until the next IPL of the system).

 

This policy is enabled by default for the 8335-GCA system and disabled for the 8335-GTA, which needs higher power for High Performance Computing.  When disabled, it will allow a higher power cap to be set with the trade-off not having a power supply in the backup role for redundancy.

 

To read the Redundant Power Supply Policy policy, use the following command: ipmitool -H <BMC IP address> -U ADMIN -P admin raw 0x3A 0x18 0x00

policy response: RC followed by the current policy followed by the policy on the next IPL (1 = disabled, 2 = enabled)

 

1.5 PowerKVM 3.1 with Ubuntu guests supported on 8335-GCA systems

 

For the 8335-GCA, PowerKVM 3.1 can be installed with Linux Ubuntu or Red Hat Enterprise guests or they may be run on OPAL Bare Metal (EC16).

 

For 8335-GTA,  only the OPAL Bare Metal (EC16) is available.  PowerKVM  3.1 is not supported on 8335-GTA.

 

1.6 "Deactivate" Power Limit fails at system runtime so use circumvention

 

"Deactivate" Power Limit at system runtime does not work.

 

-bash-4.1$ ipmitool -I lanplus -H IP -U ADMIN -a power status

Chassis Power is on

 

-bash-4.1$ ipmitool -I lanplus -H IP -U ADMIN -P admin dcmi power deactivate

 

    DCMI request failed because: Parameter out of range (c9)

 

 

Although the power limit cannot be deactivated while the system is up,  it can be changed to its maximum value which has the same effect on the system.

 

The reason to have a power limit is to manage the power consumption of all the systems in the server farm.  But a power limit can have the trade-off of lowering system performance if the workloads need more power.  So if the power limit needs to be adjusted at system runtime and the need is to remove the power limit, use the ipmitool command to set the power limit to the maximum Watts.  If the power supply redundancy policy is disabled, the maximum Watts that can be set on the power limit  is 2555 Watts.  If the power supply redundancy is enabled, the maximum watts that can be set on the power limit is 1833 Watts.

 

 ipmitool -I lanplus -H IP -U ADMIN -P admin dcmi power set_limit limit <limit value in Watts>

 

1.7 Minimum Firmware level of OP810.31 needed for NVMe flash adapter cards #EC54 and #EC56

 

In order to have a NVMe flash adapter card with Feature Code #EC54 or #EC56 plugged into a S822LC (8335-GCA) PCIe slot, the firmware on the 8335-GCA must be at least at level OP810.31.  At earlier firmware levels, the NVMe flash adapter cards would not be active on the system after an IPL.

1.8 Minimum xCAT level 2.13.4 for use in firmware updates

 

If using xCAT on the host OS to do firmware updates, the minimum xCAT level that should be used is 2.13.4 because it has stability improvements for the firmware update process.  See the xCAT 2.13.4 release notes below for more information.

 

https://github.com/xcat2/xcat-core/wiki/XCAT_2.13.4_Release_Notes

 

2.0 Important Information


Downgrading firmware from any given release level to an earlier release level is not recommended. 

If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.

Concurrent Firmware Updates not available for LC servers.

Concurrent system firmware update is not supported on LC servers.

 

3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For the LC server systems, the installation of system firmware is always disruptive.

 

3.1 Firmware Information and Description

The update.hpm file updates the primary side of the PNOR and the primary side of the BMC only, leaving the golden sides unchanged.

Filename

Size

Checksum

8335_820.1923.20190604n_update.hpm

67109473

55466b5ea6289581a4bdb6c4b79c33b5

 

 

 

 

Note: The Checksum can be found by running the Linux/Unix/AIX md5sum command against the Hardware Platform Management (hpm) file (all 32 characters of the checksum are listed), ie: md5sum <filename>

 

After a successful update to this new firmware level, the PNOR components and BMC should be at the levels listed below.  The ipmitool "fru" command can be used to display FRU ID 47:  "ipmitool -H bmc_ip_ipaddress -I lanplus -U ipmi_user -P ipmi_password fru print 47".

 

And the BMC command line command "cat" can be used to display the BMC level file:  "cat /proc/ractrends/Helper/FwInf".

 

Note:  FRU information for the PNOR level does not show the updated levels via the fru command until the system has been booted once at the updated level.

 

 Display pnor FW level using this ipmitool cmd       ipmitool -H bmc_ip_ipaddress -I lan -U ipmi_user -P ipmi_password fru print 47

 

 FRU Device Description : System Firmware (ID 47)

 

System Firmware:          

 

 Product Name          : OpenPOWER Firmware

 Product Version       : IBM-firestone-OP8_v1.12_2.98

 Product Extra         :     op-build-e481f57

 Product Extra         :     buildroot-81b8d98

 Product Extra         :     skiboot-5.4.8-5787ad3

 Product Extra         :     hostboot-2eb7706-dada8a4

 Product Extra         :     linux-4.4.92-openpower1-32c8e8f

 Product Extra         :     petitboot-v1.4.4-e414dbe

 Product Extra         :     firestone-xml-2494a43

 Product Extra         :     occ-d7efe30-5eb2419

 

BMC firmware level :  

        display BMC FW level via ssh session on the BMC , using this cmd            #  cat /proc/ractrends/Helper/FwInfo  

                                           

 

FW_VERSION=2.13.104536

FW_DATE=Sep 15 2017

FW_BUILDTIME=16:52:53 CDT

FW_DESC=8335 SRC BUILD RR9 09152017

FW_PRODUCTID=1

FW_RELEASEID=RR9

FW_CODEBASEVERSION=2.X

 

OP820
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url: 
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

8335_820.1923 / OP820.30

06/28/19

Impact:  Security      Severity:  SPE

 

New features and functions

In response to recently reported security vulnerabilities, this firmware update is being released to address Common Vulnerabilities and Exposures issue numbers CVE-2017-5715, CVE-2017-5753 and CVE-2017-5754. Operating System updates are required in conjunction with this FW level for CVE-2017-5753 and CVE-2017-5754.  This replaces an earlier firmware update for the same problem which was found to not be effective.

8335_820.1742 / OP820.21

01/12/18

Impact:  Security      Severity:  SPE

 

New features and functions

In response to recently reported security vulnerabilities, this firmware update is being released to address Common Vulnerabilities and Exposures issue numbers CVE-2017-5715, CVE-2017-5753 and CVE-2017-5754. Operating System updates are required in conjunction with this FW level for CVE-2017-5753 and CVE-2017-5754.

 

System firmware changes that affect all sytems

A problem was fixed for systems losing performance and going into Safe mode (a power mode with reduced processor frequencies intended to protect the system from over-heating and excessive power consumption). This happened because of an On-Chip Controller (OCC) internal queue overflow. The problem has only been observed for systems running heavy workloads with maximum memory configurations (where every DIMM slot is populated - size of DIMM does not matter), but this may not be required to encounter the problem. Recovery from Safe mode back to normal performance can be done with a re-IPL of the system.

A problem was fixed for correctly showing the Chassis Serial number in the "lshw" command output instead of incorreclty showing the Board Serial number. This regression error was introduced in the OP820.10 service pack.

A problem was fixed in OPAL skiroot for systems, configured to boot from a Non-Volatible Memory express (NVME) adapter, failing to reboot after the XCAT rpower reset command. Some of the failed systems will be present with a mon prompt showing a Program Check; other systems will complete the boot process but have no usable network interfaces. The systems can recovered by doing a power off and a IPL.

A problem was fixed on the BMC for java applet failures when using the BMC JViewer. To resolve the problem, the BMC JDK was updated. The applet failures had the following message: "Error: Unsigned application requesting unrestricted access to system. The following resource is signed with a weak signature algorithm MD5withRSA and is treated as unsigned: http://lc-pls1605c-con.wellsfargo.com:80/Java/release/JViewer.jar"

8335_820.1642 / OP820.10

05/19/17

Impact:  New      Severity:  New

 

New features and functions

Support was added for HTTP(S) proxies when downloading resources during the Petitboot. For example, this support allows the user to set HTTP(S) proxies for use when loading configuration or boot files.

Support for expanded time-out options for the BMC web gui/KVM sessions. The old time-out range (300 - 1800 seconds) has been changed as follows:

      • •.Increased the maximum time-out from 1800 seconds to 1296000 seconds, i.e., basically increased the maximum time-out to 15 days. 

      • •.Decreased the minimum time-out from 300 seconds to 0. If the time-out value is 0, then it is considered an infinite time-out. KVM will not time-out and that in-turn will also keep the BMC web gui running too. 

Support for the Red Hat Enterprise Linux 7.3 OS as a OPAL bare-metal install. For more information on the features delivered with RHEL7.3, see the Red Hat information portal at http://access.redhat.com/documentation/en/red-hat-enterprise-linux

Support was added to allow booting from a PCIe Non-Volatile Memory express (NVMe) adapter. The adapters affected are Feature code EC54 and EC56 NVMe flash adapters with CCIN 58CB and 58CC respectively.

Support for a OPAL raw console to receive output from the PowerPC boot EPAPR (Embedded Power Architecture Platform Requirements) wrapper. This allows decompression failures inside the wrapper caused by data corruption to be reported to the user.

Support for the CAPI Compression Accelerator Adapter with Feature Codes EJ1A and EJ1B and CCIN 2CF0. This CAPI FPGA (Field Programmable Gate Array) adapter acts as a co-processor for the POWER8 processor chip handling specialized, repetitive function extremely efficiently. The adapter is preloaded with a GZIP application and is intended to run as a gzip accelerator. The GZIP application maximum bandwidth is the PCIe3 interface bandwidth. Use of the EJ1A or EJ1B adapter requires one EC2A CAPI activation feature per system. This CAPI gzip feature does not run under PowerKVM but as a bare-metal install only for the following minimum Little Endian (LE) Linux distributions levels:

      • •.Red Hat Enterprise Linux 7.2 

      • •.Ubuntu 14.04.5 

      • •.Ubuntu 16.04.1 

Support for the CAPI NVMe (Non-Volatile Memory express) Flash Accelerator Adapter with feature code EJ1K. This feature provides a PCIe Gen3 adapter with an FPGA and 1.92 TB of low write latency, non-volatile flash memory. The adapter physically is a half-length x8 adapter, but requires a x16 PCIe CAPI-capable Gen3 slot in the system unit. The system connects to the FPGA using the CAPI interface. The FPGA connects to the flash memory using NVMe, which is a high-performance software interface to read/write this flash memory. Use of the EJ1K adapter requires one EC2A CAPI activation feature per system. This CAPI Flash Accelerator Adapter does not run under PowerKVM but is a bare-metal install only for the following minimum Little Endian (LE) Linux distribution level: Ubuntu 16.04.1 and RedHat 7.3. The 1.92 TB of flash memory consists of two 960 GB M.2 NVMe modules mounted on the card. These modules are not removable. Each of the modules is treated as a separate LUN. The adapter card is designed for read intensive workloads with light write activity. Approximately 1,370 TB of data can be written over the life of each of the 960 GB M.2 NVMe modules. There is a "fuel gauge" function provided as part of Linux utilities enabling users to view remaining write capacity on each of the modules. If the maximum write capability is achieved, the adapter's replacement or any NVMe M.2 module's replacement is not covered under IBM warranty or maintenance. This feature only pertains to the IBM Power System S822LC (8335-GCA) model.

Support was added for detecting and logging a SEL for power supply fan faults and turning on the system attention LED.

Support was enhanced in the BMC LDAP configuration:

      • •.Added support for allowing the hostname in any of LDAP fields that currently only allow IP addresses. 

      • •.Added support for BIND names greater than 64 characters. 

Support was enhanced for the configuration of the System and Audit Log settings:

      • •.Log settings were enhanced to support "local Logs" and "remote Logs" at the same time. 

      • •.The "Server Address" field was enhanced to have a custom port number in addition to the default 514 port. 

      • •.The Syslog System log was enhanced to support a TCP configuration in addition to the UDP configuration that was already supported. 

Support was changed to for the BMC SMTP configuration to remove the "machine name" field since a SMTP relay server cannot be configured.

Support was enhanced for the BMC SMTP configuration to allow the "Server Address" field to have a symbolic host name with a domain name or an IP address as was supported.

Support was enhanced for the BMC DNS configuration to allow host names to have a "." included in the name and to allow the host names to be greater than 15 characters.

System firmware changes that affect all systems

A problem was fixed for error handling in complete resets for the PCI Host Bridge (PHB). During a complete reset, there can be a time-out waiting for a pending transaction, resulting in the PHB being marked as broken and the reset is not completed, leaving the adapters in an error state. With the fix, the PHB is fenced and the Linux kernel can retry the complete reset.

  • A problem was fixed for a missing device discovery message and overly-verbose output messages during the boot. The output is now less verbose during the boot - only error-level messages are printed during Petitboot bootloader initialization. This means that there will be fewer messages printed as the system boots. Additionally, the Petitboot user interface is started earlier in the boot process. This means that the user will be presented with the user interface sooner, but it may still take time, potentially up to 30 seconds, for the user interface to be populated with boot options as storage and network hardware is being initialized. During this time, Petitboot will show the status message "Info: Waiting for device discovery". When Petitboot device discovery is completed, the following status message will be shown "Info: Connected to pb-discover!".

    A problem was fixed for Java error messages being displayed when logging into the BMC web gui. The Remote Console Preview window is no longer displayed on the dashboard, which was causing all the extra Java error messages.

    A problem has been fixed for the system MAC address being cleared with zeroes on an AC power cycle.

    A problem was fixed for "Preserve All Configuration" not working for the BMC web gui HPM update.

    A problem was fixed for not being able to change the IP address settings via Petitboot. Fixed a timing issue with the BCM5421 Controller. Without this fix, the timing issue would lead to BMC not getting any network even though all the network parameters/configuration are proper.

    A problem was fixed for PCIe Non-Volatile Memory express (NVMe) adapters not showing up after a power on of the system. The adapters affected are Feature code EC54 and EC56 NVMe flash adapters with CCIN 58CB and 58CC respectively. These flash adapters are not supported on the IBM Power System S822LC (8335-GTA).

    A problem was fixed for a system unrecoverable error that could occur if a CAPI adapter has an error exception signal and hangs when processing it. This is a rare problem that requires multiple error exceptions to the adapter in a short period of time, causing a deadlock in the adapter.

    A problem was fixed for CAPI adapter errors that caused a system processor to be called out and guarded instead of the CAPI adapter unit. The errors that cause this problem are the rare fatal adapter errors, so the problem should be infrequently seen. With the fix, the failing CAPI adapter is guarded after the checkstop instead of the system processor.

    A problem was fixed for looping error processing for some hardware failures where OPAL-PRD becomes unresponsive. The loop has been fixed to prevent repeated error messages and system slow-down. The error message "0xdeadbeefdeadbeef" was added so it is known when the error handling in OPAL-PRD has failed.

    An error message was changed for a problem where the SLW (Sleep Winkle) timer gets stuck and the firmware falls back to OPAL pollers. The previous error message was "Stuck with odd generation !". The new message to the SOL console is "SLW : timer stuck, falling back to OPAL pollers. You will likely have slower I2C and may have experienced increased jitter." These messages can be safely ignored at this time until a future firmware release resolves the issue of the stuck timer. The error only occurs when running test procedures that stress the hardware.

    A problem was fixed for ipmitool "mc reset cold" failing to reset the the BMC service processor. The ipmitool "mc reset cold" will stop working after the user issues power on when system is already on or after user tries to do power off when system is already powered off. The problem circumvention is to run the ipmitool "mc reset warm" command to the BMC which will restart the service processor IPMI process and clear internal flags that are preventing "mc reset cold" from working.

    A problem was fixed for slow IPMI Serial Over LAN (SOL) console connection to the server on the BMC. The problem was triggered by an incorrect handling of the definition bits for VGA and serial console output.

    A problem was fixed for the IPMI Serial Over LAN (SOL) console to the Petitboot user interface for the left and right arrow movements. When editing the command line for the kernel, the user could not go to the start of the line and then go forward one character at a time.

    A problem was fixed for the BMC integrated ethernet adapter BCM5421 connection speed being downgraded to 100 Mb/s instead of running at the expected 1000 Mb/s for the petitboot and the Linux OS. After the fix is applied, an A/C power cycle of the system is needed to activate the fix.

    A problem was fixed for a kexec-hardboot reboot of the system that caused USB devices to be lost. A system power cycle is needed to recover the USB devices when this error occurs.

    A problem was fixed for the shutdown of PCI devices that was causing spurious reboots of the system for a power off. The logical PCI devices are now removed during the shutdown.

    A problem was fixed for failures that happen when multiple Hypervisor Virtual Console (HVC) are active at the same time. On machines with more than one HVC console, any console after the first failed to register an interrupt handler since all consoles shared the same IRQ number.

    A problem was fixed for fundamental PCI resets at boot time causing the PCI adapters to not be usable in the Linux OS. No errors occur in the skiboot but the adapters are configurable once the OS is reached.

    A problem was fixed for time-out errors during the power off of PCI slots with " Timeout powering off slot... FIRENZE-PCI: Wrong state 00000000 on slot" error message during a power off of the system.

    A problem was fixed for the system remaining in "safe" mode after an On-Chip Controller (OCC) reset. In "safe" mode, the system is running at reduced processor frequencies, affecting system performance. The OCC reset is an error recovery command that can be requested by the BMC or OPAL for certain OCC errors.

    A problem has been fixed for Fault LEDs returning to the off state after an AC cycle when a FRU had failed and was guarded. With the fix, after an AC power cycle and next power on, the Fault LED will turn on again if the FRUs are guarded.

    A problem has been fixed for a BMC cold reset that was not moving the BIOS from the golden side to the primary side.

    A problem has been fixed for systems losing performance and going into safe mode because of On-Chip Controller (OCC) timeouts collecting Analog Power Subsystem Sweep (APSS) data. This data is used by OCC to tune the processor frequency. This problem occurs more frequently on systems with large configurations that are running heavy workloads.

    A problem was fixed for a security issue on the BMC login.

 

4.0 Operating System Information

IBM Power  LC servers supports Linux which provides a UNIX like implementation across many computer architectures.  Linux supports almost all of the Power System I/O and the configurator verifies support on order.  For more information about the software that is available on IBM Power Systems, see the Linux on IBM Power Systems website:

 http://www.ibm.com/systems/power/software/linux/index.html

 

4.1 Linux Operating System

 

The Linux operating system is an open source, cross-platform OS. It is supported on every Power Systems server IBM sells.  Linux on Power Systems is the only Linux infrastructure that offers both scale-out and scale-up choices.  The minimum supported version of Linux on the IBM Power LC server is Ubuntu Server 14.04.03 TLS for IBM POWER8 with 16.04/16.04.1 recommended..  For more information about Ubuntu Server for Ubuntu for POWER8 see the following website:

http://www.ubuntu.com/download/server/power8

 

Here is the list of supported Linux distributions for this LC server:

 

1) Red Hat Enterprise Linux 8.0 for POWER8

2) Red Hat Enterprise Linux 7.3, any subsequent updates

3) Ubuntu 16.04.1, and subsequent updates and releases

4) Ubuntu 18.04, and subsequent updates and releases

 

See the following distribution web sites for detailed information about the distributions:

 

For information about the PowerLinux Community, see the following website:

https://www.ibm.com/developerworks/group/tpl

 

For information about the features and external devices that are supported by Linux, see this website:

http://www.ibm.com/systems/power/software/linux/index.html

 

4.2 How to Determine the Level of a Linux Operating System

 

Use one of the following commands at the Linux command prompt to determine the current Linux level:

 

 

The output string from the command will provide the Linux version level.

 

4.3 How to Determine if the opal-prd (Processor Recovery Diagnostics) package is installed

The opal-prd package on the Linux system collects the OPAL Processor Recovery Diagnostics messages to log file /var/log/syslog.  It is recommended that this package be installed if it is not already present as it will help with maintaining the system processors by alerting the users to processor maintenance when needed.

 

On Ubuntu Linux, perform command  dpkg -l "opal-prd".  The output shows whether the package is installed on your system by marking it with ii (installed) and un (not installed).

This package provides a daemon to load and run the OpenPower firmware's Processor Recovery Diagnostics binary. This is responsible for run-time maintenance of Power hardware.  

If the package is not installed on your system, the following command can be run on Ubuntu to install it:

sudo apt-get install opal-prd

 

On Red Hat Linux, perform command "rpm -qa | grep -i opal-prd ".  The command output indicates the package is installed on your system if the rpm for opal-prd is found and displayed.  This package provides a daemon to load and run the OpenPower firmware's Processor Recovery Diagnostics binary. This is responsible for run-time maintenance of Power hardware.   If the package is not installed on your system, the following command can be run on Red Hat to install it:

        sudo yum update opal-prd

 

5.0 How to Determine The Currently Installed Firmware Level

 

Use the ipmtool "fru" command or the BMC Advanced System Management (ASM) FRU option to look at product details of FRU 47.

 

ipmitool -I lanplus -H <bmc host IP address> -P admin -U ADMIN fru

 

6.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

 

7.0 Installing the Firmware

7.1  IBM Power Systems Firmware maintenance

The updating and upgrading of system firmware depends on several factors, such as the current firmware that is installed, and what operating systems is running on the system.

These scenarios and the associated installation instructions are comprehensively outlined in the firmware section of Fix Central, found at the following website:

http://www.ibm.com/support/fixcentral/

 

Any hardware failures should be resolved before proceeding with the firmware updates to help insure the system will not be running degraded after the updates.

 

Run this command "hpm check" before starting an update to understand what is going to be updated:

 

 ipmitool -H <BMC IP> -U ADMIN -I lanplus -P admin hpm check <xxxxx.hpm>  where xxxx.hpm is the firmware update file name.

 

 

7.2 Updating the System Firmware with ipmitool

 Firmware update steps for the LC servers can be managed via the command line with ipmitool command.

  1) Power off the machine - install code from Standby Power state:

– ipmitool -H <hostname> -I lanplus -U ADMIN -P admin chassis power off

  2) Issue bmc reset (establish stable the starting point)

– ipmitool -H <BMC IP> -I lanplus -U ADMIN -P admin mc reset cold

  3) Run the following commands to flash the BMC and firmware (the command below protects BMC memory content; avoids losing network settings) :

3.1)  ipmitool -H <BMC IP> -I lanplus -U ADMIN -P admin raw 0x32 0xba 0x18 0x00

 

The fru print command can be used  to  display the current version of the firmware, which will help determine the appropriate update/upgrade steps to be followed.

 

For example, for FRU id 47:

  "ipmitool -H bmc_ip_ipaddress -I lanplus -U ipmi_user -P ipmi_password fru print 47"

  Product Version       : IBM-firestone-ibm-OP8_v1.12_2.98  

 

In the example above, the "_1" value in the "_1.62" string indicates the firmware is at the OP810.xx release level.

If this value were "_2", it would indicate the firmware is at the OP820.xx release level.

 

If you are updating within the same release level  (OP810.xx to OP810.yy,  or  OP820.xx to OP820.yy),  run the following command (step 3.2a), and then skip to step 4.

  3.2a)  ipmitool -H <BMC IP> -U ADMIN -I lanplus -P admin -z 30000 hpm upgrade <xxxxx.hpm> force

 

If you are  upgrading/down-grading from one release level to another (ie, from OP810.xx to OP820.xx or  OP820.xx to OP810.xx),  do the following four steps:

  3.2b):

# Flash BMC  Boot and App components 0 and 1

3.2b-1)  ipmitool -H <BMC_IP> -U ADMIN  -I lanplus -P admin -z 30000 hpm upgrade <xxxxx.hpm> component 0 force

3.2b-2)  ipmitool -H <BMC_IP> -U ADMIN  -I lanplus  -P admin -z 30000 hpm upgrade <xxxxx.hpm> component 1 force

 

# Wait for BMC to reboot  (It takes about 2-5 minutes for BMC to reach ready state. The 5 minute wait is recommended)..

3.2b-3) ipmitool -H <BMC_IP> -I lan -U ADMIN -P admin raw 0x3a 0x0a — If it returns 0x00 then BMC is at ready state otherwise it is not yet ready to continue with next step

 

# Flash BIOS/PNOR component

3.2b-4) ipmitool -H <BMC_IP>  -U ADMIN -I lanplus  -P admin -z 30000 hpm upgrade <xxxxx.hpm> component 2 force

 

  4) If the BMC network settings get lost it is possible to restore them with the following command line steps:

     a) Get serial connect to the BMC

     b) Login and execute the following commands to setup the network:

– /usr/local/bin/ipmitool -H 127.0.0.1 -I lanplus -U ADMIN -P admin lan set 1 ipsrc static

 – /usr/local/bin/ipmitool -H 127.0.0.1 -I lanplus  -U ADMIN -P admin lan set 1 ipaddr x.x.x.x

 – /usr/local/bin/ipmitool -H 127.0.0.1 -I lanplus -U ADMIN -P admin lan set 1 netmask 255.255.x.x

 – /usr/local/bin/ipmitool -H 127.0.0.1 -I lanplus -U ADMIN -P admin lan set 1 defgw ipaddr x.x.x.x

 

  5) Attention: if you experience a segmentation fault error during the code update - try the command again and change the block size 30000 to 25000.

  6) Power on and IPL the machine:

 – ipmitool -H <hostname> -I lanplus -U ADMIN -P admin chassis power on

7.2.1 Return codes from the ipmitool "hpm upgrade" command

 

The "hpm upgrade"  returns a "0" return code on success and a "-1" return code for any type of failure.  To get more error information, the user must go to the /var/log/notice.log on the system where the ipmitool was run and there will be an error message that corresponds to the "-1" returned in the ipmitool.  Below are possible error messages that can be generated for a failure in the command:

 

HpmfwupgValidateImageIntegrity:  Validate Image failure = "Invalid MD5 signature" or "Invalid image signature" or  "Unrecognized image version" or " Invalid header checksum".

 

HpmfwupgPreparationStage:  Performing Preparation Stage =  "Invalid image file for manufacturer",  Invalid image file for product ,  Invalid device ID

             Version not compatible for upgrade ="Version: Major x1,  Minor: y1  Not compatible with  Version: Major: x2 Minor:  y2"

 

HpmfwupgPreUpgradeCheck:  Pre-upgrade check

 

HpmfwupgUpgradeStage:  Upgrade Stage - Activation = "Self test failed:  Result1 = xx, Result2 = yy"

 

 

7.3  Installing ipmitool on Ubuntu

Open Power requires Source Forge ipmitool level v1.8.15 to execute correctly on the OP810 firmware.  This works fine unless there is a need to in-band firmware update from the Ubuntu host.  For this, the level of ipmitool from the Ubuntu packages must be installed.  The next step shows how to install ipmitool 1.8.13-1ubuntu0.5  from Ubuntu 14.04.3 to enable in-band code update support for the LC servers:

 

 sudo apt-get install ipmitool

7.4  Updating the System Firmware using the BMC Advanced System Management (ASM)

One method to update the System Firmware on the LC server is to use the Advanced System Management browser GUI.  The Chrome or Firefox browser must be used for this method as there are problems in this release with using IE where the firmware update will fail.

1)  First you have to connect to the BMC Service Processor Interface. Use your browser and access the BMC service processor with it's configured IP address.

2)  After the successful login, the "Advanced System Management Dashboard" will be displayed. This is the common screen for multiple activities that can be performed such as configuration, FRU information and firmware updates. General information regarding the current power consumption, sensor monitoring, event logs is displayed.

3)  The next step is to select the Firmware Update Menu.

4)  Then select the correct firmware update image type. Please select the HPM type for firmware updates. This is the only type that will be provided by the IBM Fix Central site which would have been downloaded to your workstation earlier.

5)  Now select the firmware update file from where it was stored when down loaded to the web browser.

6)  When the correct firmware image is selected,  the GUI will show a list of components that will be updated. By default all the components will be selected.  Press Proceed button to perform the firmware update. However, when updating between the OP810 and OP820 levels, the update must be done in three steps:

6.1)  Select the  Boot component and  proceed with update;

6.2)  Select the App component and proceed with update; and

6.3)  Select the BIOS component and proceed with update.

7)  After the firmware update is completed, the System will perform a reboot.

 

 

8.0 System Management and Virtualization

The service processor, or baseboard management controller (BMC), provides a hypervisor and operating system-independent layer that uses the robust error detection and self-healing functions that are built into the POWER8 processor and memory buffer modules. Open power application layer (OPAL) is the system firmware in the stack of POWER8 processor-based Linux-only servers. IBM PowerKVM technology offers key capabilities that can help consolidate and simplify the IT environment.  PowerKVM servers can be managed by open source Linux tools that use the libvirt API such as the Kimchi point-to-point administration tool and PowerVC.  QEMU is a generic and open source machine emulator and virtualizer that hosts the virtual machines on a KVM hypervisor. It is the software that manages and monitors the virtual machines.

IBM PowerVC delivers easy-to-use advanced virtualization management capabilities that are virtualized by IBM PowerKVM.   PowerVC manages PowerKVM Virtual Machines (VMs) within a resource pool and enables the capture, deployment, and inventory of VM images.

 

8.1  BMC Service Processor IPMI and ASM Access

The service processor, or baseboard management controller (BMC), is the primary control for autonomous sensor monitoring and event logging features on the LC server.

The BMC supports the Intelligent Platform Management Interface (IPMI) for system monitoring and management.  The BMC monitors the operation of the firmware during the boot process and also monitors the OPAL hypervisor for termination.  The firmware code update is supported through the BMC and Intelligent Platform Monitoring Interface (IPMI) and the Advanced System Management (ASM) console.  The ASM console is accessed using a web browser with a "http:" connection to port.  See section 1.2 for the supported browsers that can be used with ASM.  For more information on using the BMC ASM, see the IBM Redbook PDF file for the  IBM Power System S822LC Technical Overview and Introduction - section 3.4 Serviceability" ,  http://www.redbooks.ibm.com/redpieces/pdfs/redp5283.pdf.

 

8.2 Open Power Abstraction Layer (OPAL) Hypervisor

The Open Power Abstraction Layer (OPAL) provides hardware abstraction and run time services to the running host Operating System.

For the 8335-GCA,  PowerKVM 3.1 can be installed with Ubuntu guests or Ubuntu may be run on OPAL Bare Metal (EC16).

For the 8335-GTA,  only the OPAL Bare Metal (EC16) is available.  PowerKVM  3.1 is not supported on 8335-GTA.

 

Find out more about OPAL skiboot here:

https://github.com/open-power/skiboot

 

8.3 Intelligent Platform Management Interface (IPMI)

The Intelligent Platform Management Interface (IPMI) is an open standard for monitoring, logging, recovery, inventory, and control of hardware that is implemented independent of the main CPU, BIOS, and OS. It is the default console to use when you configure PowerKVM.  The LC server provides one 10M/100M baseT IPMI port.

The ipmitool is a utility for managing and configuring devices that support IPMI. It provides a simple command-line interface to the service processor.  You can install the ipmitool from the Linux distribution packages in your workstation, sourceforge.net, or another server (preferably on the same network as the installed server). For example, in Ubuntu, use this command:

$ sudo apt-get install ipmitool

For installing ipmitool from sourceforge, please see section 1.1 "Minimum ipmitool Code Level".

 

For more information about ipmitool, there are several good references for ipmitool commands:

 

1.  The man page

2.  The built-in command line help provides a list of IPMItool commands:
 # ipmitool help

3.  You can also get help for many specific IPMItool commands by adding the word help after the command:  
 # ipmitool channel help

4.  For  a list of common ipmitool commands and help on each, you may use the following link:  
www-01.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabpcommonipmi.htm

 

 

To connect to your host system with IPMI, you need to know the IP address of the server and have

a valid password. To power on the server with the ipmitool, follow these steps:

1. Open a terminal program.

2. Power on your server with the ipmitool:

ipmitool -I lanplus -H fsp_ip_address -P ipmi_password power on

3. Activate your IPMI console:

ipmitool -I lanplus -H fsp_ip_address -P ipmi_password sol activate

For more help with configuring IBM PowerKVM on a Linux on Power Systems server see the

following:

https://www-01.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabpusingipmi.htm

Also, see the Quick Start Guide for Configuring IBM PowerKVM on Power Systems here:

https://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/linuxonibm/liabq/kvmquickstart_guide.pdf

 

8.4 Petitboot bootloader

Petitboot is a kexec based bootloader used by IBM POWER8 systems configured with PowerKVM.

After the POWER8 system powers on, the petitboot bootloader scans local boot devices and network interfaces to find boot options that are available to the system. Petitboot returns a list of boot options that are available to the system. If you are using a static IP or if you did not provide boot arguments in your network boot

server, you must provide the details to petitboot. You can configure petitboot to find your boot with the following instructions:

https://www-01.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitbootadvanced.htm

 

You can edit petitboot configuration options, change the amount of time before Petitboot

automatically boots, etc. with these instructions:

https://www-01.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabppetitbootconfig.htm

 

After you select to boot the PowerKVM installer, the installer wizard walks you through the steps to set up disk options, your root password, time zones, and so on.

You can read more about the petitboot bootloader program here:

https://www.kernel.org/pub/linux/kernel/people/geoff/petitboot/petitboot.html

 

8.5 IBM PowerKVM

Kernel-based Virtual Machine (KVM) is a cross-platform, open source hypervisor that provides enterprise-class performance, scalability and security to run Linux and other

workloads on a range of processor architectures. For the Linux-only scale-out systems with POWER8 technology this mechanism is ported to Power Systems and called IBM PowerKVM (5765-KVM). The publication "IBM PowerKVM Configuration and Use SG24-8231" can be found here:

http://publib-b.boulder.ibm.com/abstracts/sg248231.html

 

IBM PowerKVM supports Big Endian (BE) and Little Endian (LE) mode for Ubuntu 14.04.03 LTS.

Note: Power LC servers and PowerKVM do not support AIX or IBM i guest VMs and cannot be managed by an HMC

 

IBM PowerKVM supports the Power 8335-GTA server with the Red Hat Enterprise Linux 7.2.  For additional questions about the availability of this release and supported Power servers, consult the Red Hat Hardware Catalog at https://hardware.redhat.com.

 

For a complete list of supported virtualization options for IBM Power Systems Linux-onlyservers see this link:

http://www-01.ibm.com/support/knowledgecenter/linuxonibm/liaam/liaamvirtoptions.htm

 

8.5.1 Download IBM PowerKVM

If your system is not preconfigured with PowerKVM, you need to download the installer file.

PowerKVM installation files are available from the Entitled System Support site at:

http://www-304.ibm.com/servers/eserver/ess/index.wss

 

8.5.2 Update PowerKVM

If you want to update your system with the latest PowerKVM package, you can find the updates through Fix Central or the IBM yum repository for PowerKVM.  If your system has

Internet access, you can use Kimchi or yum. If you do not have Internet access, use the ibm-update-system utility.

https://www-01.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabpupdatesystem.htm

 

8.5.3 Upgrade PowerKVM

If you want to upgrade PowerKVM to a new release, see the publication Upgrade PowerKVM:

https://www-01.ibm.com/support/knowledgecenter/linuxonibm/liabp/liabpupgrade.htm

 

8.6 Kimchi management tool

Kimchi is an Apache-licensed project hosted on GitHub. It is an HTML5-based management tool that can get you started with IBM PowerKVM. It runs as a daemon on the hypervisor host and interfaces with underlying libvirt, QEMU, and KVM components. You use Kimchi to create and manage guests, monitor your host system, create networking interfaces, add storage, and update packages. To use Kimchi, open a browser and point it to "https://ip_address:8001" where ip_address is the IP address of your KVM system. Log in using the admin user name and password.

Note: When you connect to Kimchi, make sure that you enable SSL connections in your browser. For Firefox browsers, you might also be required to connect to

"https://ip_address:64667" where ip_address is the IP address of your host KVM system and accept the self-signed certificate. Connect using the HTTP secure (HTTPS).

Kimchi represents a simple, open source, multi-platform virtualization tool to manage PowerKVM. You can learn more about the Kimchi project here:

https://github.com/kimchi-project/kimchi

 

8.7 IBM PowerVC

The IBM Power Virtualization Center (5765-VCS) is an advanced enterprise virtualization management offering for Power Systems based on the OpenStack technology. OpenStack is an open source software that controls large pools of server, storage, and networking resources throughout a data center. IBM PowerVC Version 1.3.0 was announced in October, 2015 and is built on OpenStack (liberty). This comprehensive virtualization management offering enables virtual machine setup and management.

 

8.7.1 PowerVC Benefits

PowerVC includes the following features and benefits:

 

For full details about hardware and operating system support for PowerVC hosts., refer to Hardware and Software Requirements here:

http://www-01.ibm.com/support/knowledgecenter/SSXK2N_1.2.3/com.ibm.powervc.kvm.help.doc/powervc_hwandsw_reqs_kvm.html

 

9.0 Quick Start Guide for Installing Linux on LC servers

This guide helps you install Ubuntu on a Linux on Power Systems server.

Overview

Use the information found in http://www-01.ibm.com/support/knowledgecenter/linuxonibm/liabw/liabwkickoff.htm to install Linux, in this case Ubuntu, on a non-virtualized or bare metal IBM Power LC server. The Ubuntu installer is available for download for specific Linux on Power Systems. For information about which systems support Ubuntu, see Supported Linux distributions for POWER8 Linux on Power systems at https://ibm.biz/BdEH5t.

 

 

10.0 Change History

 

Date

Description

10/30/2015

New for LC server OP810.00 release

11/24/2015

OP810.10 release

03/18/2016

OP810.30 release

06/16/2016

OP810.31 release - update to BMC firmware only (PNOR firmware level unchanged from OP810.30)

07/28/2016

OP810.32 limited release

09/09/2016

OP810.33 limited release

05/19/2017

OP820.10 release

01/12/2018

OP820.21 release

06/28/2019

OP820.30 release