Power9 System Firmware

Applies to:   9008-22L; 9009-22A; 9009-41A; 9009-42A; 9223-22H; and 9223-42H.

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.

Contents

READ ME FIRST

Important Information applicable only for systems at VL930_035 (FW930.00) with VIOS partitions or IBM i HSM:
If you have VL930_035 (FW 930.00) on your system, take the following steps to both resolve any current problem you may be experiencing due to changed location codes, and to avoid future problems due to location codes reverting to their original format in later firmware levels.
1) Apply an ifix corresponding to your VIOS level.  The fixes and a README to identify which ifix you need can be found here:
ftp://aix.software.ibm.com/aix/ifixes/FW930/
2) Power down the system
3) Apply FW 930.01 or later
4) Power on the system.
The VIOS will restore the correct mapping when the system is booted on FW 930.01 or later with the ifix installed.
5) Remove the ifix after booting on FW 930.01 or later after verifying the mappings are restored

If FW930.00 was installed, and IBM i partitions ran on it, there will be extraneous resources defined in the IBM i partitions following the update to FW930.01 or later.  These warnings can be ignored.  After the IPL of the IBM i partition on FW930.01 or later, the unneeded resources should be removed, using the following steps given in APAR MB04215:
1. STRSST and sign in
2. Option 1. Start a service tool
3. Option 7. Hardware service manager
4. Option 4.  Failed and non-reporting hardware resources
5. Put a '4' (Remove) against any BCxx (Fabric I/O Bridge) resources that are listed, and then press Enter to remove, and Enter again to confirm the removal.
6. If there are any SRIOV shared mode resources listed, those can be removed also.

NOTE: Customers upgrading from FW910.xx levels to FW930.01 or later are not affected by this problem!

1.0 Systems Affected

This package provides firmware for Power Systems L922 (9008-22L), Power System S922 (9009-22A), Power System S914 (9009-41A), Power System S924 (9009-42A), Power System H922 (9223-22H) and Power System H924 (9223-42H. ) servers only.

The firmware level in this package is:

1.1 Minimum HMC Code Level

This section is intended to describe the "Minimum HMC Code Level" required by the System Firmware to complete the firmware installation process. When installing the System Firmware, the HMC level must be equal to or higher than the "Minimum HMC Code Level" before starting the system firmware update.  If the HMC managing the server targeted for the System Firmware update is running a code level lower than the "Minimum HMC Code Level" the firmware update will not proceed.

The Minimum HMC Code levels for this firmware for HMC x86,  ppc64 or ppc64le are listed below.

x86 -  This term is used to reference the legacy HMC that runs on x86/Intel/AMD  hardware for both the 7042 Machine Type appliances and the Virtual HMC that can run on the Intel hypervisors (KVM, VMWare, Xen).
ppc64 or ppc64le - describes the Linux code that is compiled to run on Power-based servers or LPARS (Logical Partitions)

For information concerning HMC releases and the latest PTFs,  go to the following URL to access Fix Central:
http://www-933.ibm.com/support/fixcentral/

For specific fix level information on key components of IBM Power Systems running the AIX, IBM i and Linux operating systems, we suggest using the Fix Level Recommendation Tool (FLRT):
http://www14.software.ibm.com/webapp/set2/flrt/home


NOTES:
                -You must be logged in as hscroot in order for the firmware installation to complete correctly.
                - Systems Director Management Console (SDMC) does not support this System Firmware level

1.2 IBM i Minimum Levels

Reference the following URL for IBM i Support: Recommended fixes:

https://www-01.ibm.com/support/docview.wss?uid=nas8N1021194

1.3 Late Breaking Features

A new option was added to ASMI to provide customer control over speculative execution in response to CVE-2017-5753 and CVE-2017-5715 (collectively known as Spectre) and CVE-2017-5754 (known as Meltdown)  for POWER9 9009-42A, 9223-42H, 9009-41A, 9009-22A, 9223-22H, and 9008-22L.  More information on this option can be found in the IBM KnowledgeCenter at the following link:   https://www.ibm.com/support/knowledgecenter/POWER9/p9hby/p9hby_speculative_execution_control.htm.

2.0 Important Information

Possible partition crash when using Live Partition Mobility (LPM) or partition hang when doing a concurrent firmware update:

A very intermittent issue has been found in the IBM lab when using partition mobility or firmware update.  For a mobility operation, the issue can result in a partition crash if the mobility target system is FW930.00, FW930.01 or FW930.02.  For a code update operation, the partition may hang.  The recovery is to reboot the partition after the crash or hang.  This problem is fixed in service pack FW930.03.

Downgrading firmware from any given release level to an earlier release level is not recommended.

Firmware downgrade warning:  Adapter feature codes (i#EC2R/#EC2S/#EC2T/#EC2U and #EC3E/#EC3F/#EC3L/#EC3M/#EC3T/#EC3U) when configured in SR-IOV shared mode in FW930, even if originally configured in shared mode in a pre-FW930 release, may not function properly if the system is downgraded to a pre-FW930 release. The adapter should be configured in dedicated mode first (i.e. take the adapter out of SR-IOV shared mode) before downgrading to a pre-FW930 release.

If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.

2.1 IPv6 Support and Limitations

IPv6 (Internet Protocol version 6) is supported in the System Management Services (SMS) in this level of system firmware. There are several limitations that should be considered.

When configuring a network interface card (NIC) for remote IPL, only the most recently configured protocol (IPv4 or IPv6) is retained. For example, if the network interface card was previously configured with IPv4 information and is now being configured with IPv6 information, the IPv4 configuration information is discarded.

A single network interface card may only be chosen once for the boot device list. In other words, the interface cannot be configured for the IPv6 protocol and for the IPv4 protocol at the same time.

2.2 Concurrent Firmware Updates

Concurrent system firmware update is supported on HMC Managed Systems only.

2.3 Memory Considerations for Firmware Upgrades

Firmware Release Level upgrades and Service Pack updates may consume additional system memory.
Server firmware requires memory to support the logical partitions on the server. The amount of memory required by the server firmware varies according to several factors.
Factors influencing server firmware memory requirements include the following:
Generally, you can estimate the amount of memory required by server firmware to be approximately 8% of the system installed memory. The actual amount required will generally be less than 8%. However, there are some server models that require an absolute minimum amount of memory for server firmware, regardless of the previously mentioned considerations.

Additional information can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9hat/p9hat_lparmemory.htm

2.4 SBE Updates

Power 9 servers contain SBEs (Self Boot Engines) and are used to boot the system.  SBE is internal to each of the Power 9 chips and used to "self boot" the chip.  The SBE image is persistent and is only reloaded if there is a system firmware update that contains a SBE change.  If there is a SBE change and system firmware update is concurrent, then the SBE update is delayed to the next IPL of the CEC which will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  If there is a SBE change and the system firmware update is disruptive, then SBE update will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  During the SBE update process, the HMC or op-panel will display service processor code C1C3C213 for each of the SBEs being updated.  This is a normal progress code and system boot should be not be terminated by the user. Additional time estimate can be between 6-10 minutes.

3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For systems that are not managed by an HMC, the installation of system firmware is always disruptive.

Note: The concurrent levels of system firmware may, on occasion, contain fixes that are known as Deferred and/or Partition-Deferred. Deferred fixes can be installed concurrently, but will not be activated until the next IPL. Partition-Deferred fixes can be installed concurrently, but will not be activated until a partition reactivate is performed. Deferred and/or Partition-Deferred fixes, if any, will be identified in the "Firmware Update Descriptions" table of this document. For these types of fixes (Deferred and/or Partition-Deferred) within a service pack, only the fixes in the service pack which cannot be concurrently activated are deferred.

Note: The file names and service pack levels used in the following examples are for clarification only, and are not necessarily levels that have been, or will be released.

System firmware file naming convention:

01VLxxx_yyy_zzz

NOTE: Values of service pack and last disruptive service pack level (yyy and zzz) are only unique within a release level (xxx). For example, 01VL900_040_040 and 01VL910_040_045 are different service packs.

An installation is disruptive if:

            Example: Currently installed release is 01VL900_040_040, new release is 01VL910_050_050.

            Example: VL910_040_040 is disruptive, no matter what level of VL910 is currently installed on the system.

            Example: Currently installed service pack is VL910_040_040 and new service pack is VL910_050_045.

An installation is concurrent if:

The release level (xxx) is the same, and
The service pack level (yyy) currently installed on the system is the same or higher than the last disruptive service pack level (zzz) of the service pack to be installed.

Example: Currently installed service pack is VL910_040_040, new service pack is VL910_041_040.

3.1 Firmware Information and Description

 
Filename Size Checksum md5sum
01VL930_040_040.rpm 130365311 17775
 4246816cec2ac287fa6a9c53ae11fcff

Note: The Checksum can be found by running the AIX sum command against the rpm file (only the first 5 digits are listed).
ie: sum 01VL930_040_040.rpm

VL930
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

The complete Firmware Fix History for this Release Level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VL-Firmware-Hist.html
VL930_040_040 / FW930.01

05/31/19		 Impact: Availability       Severity: HIPER

System firmware changes that affect certain systems
  • HIPER/Pervasive: DISRUPTIVE:  For systems at VL930_035 (FW930.00) with LPARs using a VIOS for virtual adapters or using VIOS Shared Storage Pool (SSP) clusters, a problem was fixed for a possible network hang in the LPAR after upgrading from FW910.XX to FW930.00.  Systems with IBM i partitons and HSM may see extraneous resources, with the old ones non-reporting.  With the problem, the I/O adapter port location code names had changed from that used for the FW910 release, and this difference in the naming convention prevents the VIOS virtual adapters from working in some cases.  For example, NPIV (N_PORT ID Virtualization) Fibre Channel (FC) adapters fail to work because of the I/O mappings are different.  With the fix, the I/O adapter location code naming convention reverts back to that used for FW910.    There could be other impacts not described here to operating system views of IO devices and IBM strongly recommends that customers at FW930.00 update to this new level or higher.  Customers upgrading from FW910.xx levels to FW930.01 or later are not affected by this problem.
    The following is an example of the original location code format and the problem version of the location code name (notice that "001" was replaced by "ND1"):
    Original format: "fcs0 U78D2.001.WZS000W-P1-C6 Available 01-00 8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)"
    Problem format: "fcs0 U78D2.ND1.WZS000W-P1-C6 Available 01-00 8Gb PCI Express Dual Port FC Adapter (df1000f114108a03)"
    If you have a FW930.00 (VL930_035) installed, please follow the steps outlined in the "Important Information applicable only for systems at VL930_035 (FW930.00)  with VIOS partitions or IBM i HSM" section of the README. 
VL930_035_035 / FW930.00

05/17/19		 Impact:  New      Severity:  New

All features and fixes from the FW910.30 service pack (and below) are included in this release.

New Features and Functions
  • Support was added to allow the FPGA soft error checking on the PCIe I/O expansion drawer (#EMX0) to be disabled with the help of IBM support using the hypervisor "xmsvc" macro.  This new setting will persist until it it is changed by the user or IBM support.  The effect of disabling FPGA soft error checking is to eliminate the FPGA soft error recovery which causes a recoverable PCIe adapter outage.  Some of the soft errors will be hidden by this change but others may have unpredictable results, so this should be done only under guidance of IBM support.
  • Support for the PCIe3 expansion drawer  (#EMX0) I/O drawer clock enhancement so that a reset of the drawer does not affect the reference clock to the adapters so the PCIe lanes for the PCIe adapters can keep running through an I/O drawer FPGA reset.  To use this support, new cable cards, fanout modules, and optical cables are needed after this support is installed: PCIe Six Slot Fan out module(#EMXH) - only allowed to be connected to converter adapter cable card;  PCIe X16 to CXP Optical or CU converter adapter for the expansion drawer (#EJ1R); and new AOC cables with feature/part number of #ECCR/78P6567, #ECCX/78P6568, #ECCY/78P6569, and #ECCZ/78P6570. These parts cannot be install concurrently, so a scheduled outage is needed to complete the migration.
  • Support added for RDMA Over Converged Ethernet (RoCE) for SR-IOV adapters.
  • Support added for SMS menu to enhance the  I/O information option to have "vscsi" and "network" options.  The information shown for "vscsi" devices is similar to that provided for SAS and Fibre Channel devices.  The "network" option provides connectivity information for the adapter ports and shows which can be used for network boots and installs.
  • Support for an IBM i system with a SAN boot feature code that can be ordered without an internal DASD backplane, SAS cables, or SAS adapters.
  • Support added to allow integrated USB ports to be disabled.  This is available via an Advanced System Management Interface (ASMI) menu option:  "System Configuration -> Security -> USB Policy".  The USB disable policy, if selected, does not apply to pluggable USB adapters plugged into PCIe slots such as the 4-Port USB adapter (#EC45/#EC46), which are always enabled.
System firmware changes that affect all systems
  • A problem was fixed for a system IPLing with an invalid time set on the service processor that causes partitions to be reset to the Epoch date of 01/01/1970.  With the fix, on the IPL, the hypervisor logs a B700120x when the service processor real time clock is found to be invalid and halts the IPL to allow the time and date to be corrected by the user.  The Advanced System Management Interface (ASMI) can be used to correct the time and date on the service processor.  On the next IPL, if the time and date have not been corrected, the hypervisor will log a SRC B7001224 (indicating the user was warned on the last IPL) but allow the partitions to start, but the time and date will be set to the Epoch value.

4.0 How to Determine The Currently Installed Firmware Level

You can view the server's current firmware level on the Advanced System Management Interface (ASMI) Welcome pane. It appears in the top right corner. Example: VL910_123.

5.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

Note: If your HMC is not internet-connected you will need to download the new firmware level to a USB flash memory device or ftp server.

6.0 Installing the Firmware

The method used to install new firmware will depend on the release level of firmware which is currently installed on your server. The release level can be determined by the prefix of the new firmware's filename.

Example: VLxxx_yyy_zzz

Where xxx = release level

HMC Managed Systems:

Instructions for installing firmware updates and upgrades on systems managed by an HMC can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eh6/p9eh6_updates_sys.htm


NovaLink Managed Systems:

A NovaLink managed system does not have a HMC attached and is managed either by PowerVM Novalink or PowerVC using PowerVM Novalink.
Instructions for installing firmware updates and upgrades on systems managed by PowerVM NovaLink can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eig/p9eig_updating_firmware.htm


HMC and NovaLink Co-Managed Systems:

 A co-managed system is managed by HMC and NovaLink, with one of the interfaces in the co-management master mode.
Instructions for installing firmware updates and upgrades on systems co-managed by an HMC and Novalink is the same as above for a HMC managed systems since the firmware update must be done by the HMC in the co-management master mode.  Before the firmware update is attempted, one must be sure that HMC is set in the master mode using the steps at the following IBM KnowledgeCenter link for NovaLink co-managed systems:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eig/p9eig_kickoff.htm

Then the firmware updates can proceed with the same steps as for the HMC managed systems:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eh6/p9eh6_updates_sys.htm


Systems not Managed by an HMC or NovaLink:

Power Systems:

Instructions for installing firmware on systems that are not managed by an HMC can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9ha5/fix_serv_firm_kick.htm


Systems running Ubuntu operating system:

If  Ubuntu will be used to update the system firmware, please follow these instructions to extract the installable binary and update/upgrade the firmware:

1) Download the .gz (tarball) from Fix Central to your Ubuntu system (ie, to /tmp/fwupdate).

2) Extract the .gz file to /tmp/ on the Ubuntu system:

Example:
    tar -xzf /tmp/fwupdate/01VL910_040_040.tar.gz -C /tmp/fwupdate

3) Use update_flash -v -f <extracted file name> to verify the package.

4) Update your firmware using update_flash:

/usr/sbin/update_flash  -f <extracted file name>

System will reboot during the firmware update. When the system reaches Ubuntu run-time state, you can then commit or reject the firmware update:
Commit: /usr/sbin/update_flash -c
Reject:   /usr/sbin/update_flash -r

IBM i Systems:

For information concerning IBM i Systems, go to the following URL to access Fix Central: 
http://www-933.ibm.com/support/fixcentral/

Choose "Select product", under Product Group specify "System i", under Product specify "IBM i", then Continue and specify the desired firmware PTF accordingly.

7.0 Firmware History

The complete Firmware Fix History (including HIPER descriptions)  for this Release level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VL-Firmware-Hist.html

8.0 Change History

Date
 Description
August 19, 2019 Added note to section "2.0 Important Information" of this description file about a possible partition crash when using Live Partition Mobility (LPM) or partition hang when doing a concurrent firmware update.
July 03, 2019
 Added warning about special handling that may be needed for adapters configured in SR-IOV shared mode when downgrading from any FW930 level to a pre-FW930 level to section "2.0 Important Information" of this description file.
June 03, 2019 Fix description update for firmware level VL930_040_040 / FW930.01