Power9 System Firmware

Applies to:   9008-22L; 9009-22A; 9009-41A; 9009-42A; 9223-22H; and 9223-42H.

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.

Contents

1.0 Systems Affected

This package provides firmware for Power Systems L922 (9008-22L), Power System S922 (9009-22A), Power System S914 (9009-41A), Power System S924 (9009-42A), Power System H922 (9223-22H) and Power System H924 (9223-42H. ) servers only.

The firmware level in this package is:

1.1 Minimum HMC Code Level

This section is intended to describe the "Minimum HMC Code Level" required by the System Firmware to complete the firmware installation process. When installing the System Firmware, the HMC level must be equal to or higher than the "Minimum HMC Code Level" before starting the system firmware update.  If the HMC managing the server targeted for the System Firmware update is running a code level lower than the "Minimum HMC Code Level" the firmware update will not proceed.

The Minimum HMC Code levels for this firmware for HMC x86,  ppc64 or ppc64le are listed below.

x86 -  This term is used to reference the legacy HMC that runs on x86/Intel/AMD hardware for both the 7042 Machine Type appliances and the Virtual HMC that can run on the Intel hypervisors (KVM, VMWare, Xen).
ppc64 or ppc64le - describes the Linux code that is compiled to run on Power-based servers or LPARS (Logical Partitions)

For information concerning HMC releases and the latest PTFs,  go to the following URL to access Fix Central:
http://www-933.ibm.com/support/fixcentral/

For specific fix level information on key components of IBM Power Systems running the AIX, IBM i and Linux operating systems, we suggest using the Fix Level Recommendation Tool (FLRT):
http://www14.software.ibm.com/webapp/set2/flrt/home


NOTES:
                -You must be logged in as hscroot in order for the firmware installation to complete correctly.
                - Systems Director Management Console (SDMC) does not support this System Firmware level

1.2 IBM i Minimum Levels

Reference the following URL for IBM i Support: Recommended fixes:

https://www-01.ibm.com/support/docview.wss?uid=nas8N1021194

2.0 Important Information

NovaLink with partitions running certain SR-IOV capable adapters is NOT supported at this firmware release

NovaLink does not support IO adapter FCs EC2R/EC2S, EC2T/EC2U, EC3L/EC3M, EC66/EC67 with FW930 and later.   If the adapter was already in use with FW910/920, upgrading to FW930/940 will result in errors in NovaLink and PowerVC which causes the loss of any management operation via NovaLink / PowerVC combination.  Upgrading systems in this configuration is not supported.  If the system is required to be at FW930/940 or was shipped with FW930/940 call Support for further assistance.

Boot adapter microcode requirement

Update all adapters which are boot adapters, or which may be used as boot adapters in the future, to the latest microcode from IBM Fix Central.  The latest microcode will ensure the adapters support the new Firmware Secure Boot feature of Power Systems. This requirement applies when updating system firmware from a level prior to FW940 to levels FW940 and later.
The latest adapter microcode levels include signed boot driver code. If a boot-capable PCI adapter is not installed with the latest level of adapter microcode, the partition which owns the adapter will boot, but error logs with SRCs BA5400A5 or BA5400A6 will be posted.  Once the adapter(s) are updated, the error logs will no longer be posted.

Downgrading firmware from any given release level to an earlier release level is not recommended

Firmware downgrade warnings:
1) Adapter feature codes (#EC2R/#EC2S/#EC2T/#EC2U and #EC3L/#EC3M and #EC66/EC67) when configured in SR-IOV shared mode in FW930 or later, even if originally configured in shared mode in a pre-FW930 release, may not function properly if the system is downgraded to a pre-FW930 release. The adapter should be configured in dedicated mode first (i.e. take the adapter out of SR-IOV shared mode) before downgrading to a pre-FW930 release.
2) If partitions have been run in POWER9 compatibility mode in FW940, a downgrade to an earlier release (pre-FW940) may cause a problem with the partitions starting.  To prevent this problem, the "server firmware" settings must be reset by rebooting partitions in "Power9_base" before doing the downgrade.

If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.

2.1 IPv6 Support and Limitations

IPv6 (Internet Protocol version 6) is supported in the System Management Services (SMS) in this level of system firmware. There are several limitations that should be considered.

When configuring a network interface card (NIC) for remote IPL, only the most recently configured protocol (IPv4 or IPv6) is retained. For example, if the network interface card was previously configured with IPv4 information and is now being configured with IPv6 information, the IPv4 configuration information is discarded.

A single network interface card may only be chosen once for the boot device list. In other words, the interface cannot be configured for the IPv6 protocol and for the IPv4 protocol at the same time.

2.2 Concurrent Firmware Updates

Concurrent system firmware update is supported on HMC Managed Systems only.

Ensure that there are no RMC connections issues for any system partitions prior to applying the firmware update.  If there is a RMC connection failure to a partition during the firmware update, the RMC connection will need to be restored and additional recovery actions for that partition will be required to complete partition firmware updates.

2.3 Memory Considerations for Firmware Upgrades

Firmware Release Level upgrades and Service Pack updates may consume additional system memory.
Server firmware requires memory to support the logical partitions on the server. The amount of memory required by the server firmware varies according to several factors.
Factors influencing server firmware memory requirements include the following:
Generally, you can estimate the amount of memory required by server firmware to be approximately 8% of the system installed memory. The actual amount required will generally be less than 8%. However, there are some server models that require an absolute minimum amount of memory for server firmware, regardless of the previously mentioned considerations.

Additional information can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9hat/p9hat_lparmemory.htm

2.4 SBE Updates

Power 9 servers contain SBEs (Self Boot Engines) and are used to boot the system.  SBE is internal to each of the Power 9 chips and used to "self boot" the chip.  The SBE image is persistent and is only reloaded if there is a system firmware update that contains a SBE change.  If there is a SBE change and system firmware update is concurrent, then the SBE update is delayed to the next IPL of the CEC which will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  If there is a SBE change and the system firmware update is disruptive, then SBE update will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  During the SBE update process, the HMC or op-panel will display service processor code C1C3C213 for each of the SBEs being updated.  This is a normal progress code and system boot should be not be terminated by the user. Additional time estimate can be between 6-10 minutes.

3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For systems that are not managed by an HMC, the installation of system firmware is always disruptive.

Note: The concurrent levels of system firmware may, on occasion, contain fixes that are known as Deferred and/or Partition-Deferred. Deferred fixes can be installed concurrently, but will not be activated until the next IPL. Partition-Deferred fixes can be installed concurrently, but will not be activated until a partition reactivate is performed. Deferred and/or Partition-Deferred fixes, if any, will be identified in the "Firmware Update Descriptions" table of this document. For these types of fixes (Deferred and/or Partition-Deferred) within a service pack, only the fixes in the service pack which cannot be concurrently activated are deferred.

Note: The file names and service pack levels used in the following examples are for clarification only, and are not necessarily levels that have been, or will be released.

System firmware file naming convention:

01VLxxx_yyy_zzz

NOTE: Values of service pack and last disruptive service pack level (yyy and zzz) are only unique within a release level (xxx). For example, 01VL900_040_040 and 01VL910_040_045 are different service packs.

An installation is disruptive if:

            Example: Currently installed release is 01VL900_040_040, new release is 01VL910_050_050.

            Example: VL910_040_040 is disruptive, no matter what level of VL910 is currently installed on the system.

            Example: Currently installed service pack is VL910_040_040 and new service pack is VL910_050_045.

An installation is concurrent if:

The release level (xxx) is the same, and
The service pack level (yyy) currently installed on the system is the same or higher than the last disruptive service pack level (zzz) of the service pack to be installed.

Example: Currently installed service pack is VL910_040_040, new service pack is VL910_041_040.

3.1 Firmware Information and Description

 
Filename Size Checksum md5sum
01VL940_034_027.rpm 134340831 39175 1012411b6bb36353f68ddb0530219116

Note: The Checksum can be found by running the AIX sum command against the rpm file (only the first 5 digits are listed).
ie: sum 01VL940_034_027.rpm

VL940
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

The complete Firmware Fix History for this Release Level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VL-Firmware-Hist.html
VL940_034_027 / FW940.01

01/09/20
 Impact:  Security      Severity:  SPE

New features and functions

  • Support was added for improved security for the service processor password policy.  For the service processor, the "admin", "hmc" and "general" password must be set on first use for newly manufactured systems and after a factory reset of the system.  The IPMI interface has been changed to be disabled by default in these scenarios.  The REST/Redfish interface will return an error saying the user account is expired.  This policy change helps to enforce the service processor is not left in a state with a well known password.  The user can change from an expired default password to a new password using the Advanced System Management Interface (ASMI).
  • Support was added for real-time data capture for PCIe3 expansion drawer (#EMX0) cable card connection data via resource dump selector on the HMC or in ASMI on the service processor.  Using the resource selector string of "xmfr -dumpccdata" will non-disruptively generate an RSCDUMP type of dump file that has the current cable card data, including data from cables and the retimers.
  • Improvements to link stack algorithms.

System firmware changes that affect all systems

  • A problem was fixed for an intermittent IPMI core dump on the service processor.  This occurs only rarely when multiple IPMI sessions are starting and cleaning up at the same time.  A new IPMI session can fail initialization when one of its session objects is cleaned up.  The circumvention is to retry the IPMI command that failed.
  • A problem was fixed for system hangs or incomplete states displayed by HMC(s) with SRC B182951C logged.  The hang can occur during operations that require a memory relocation for any partition such as Dynamic Platform Optimization (DPO), memory mirroring defragmentation, or memory guarding that happens as part of memory error recovery during normal operations of the system.
  • A problem was fixed for possible unexpected interrupt behavior for partitions running in POWER9 processor compatibility mode.  This issue can occur during the boot of a partition running in POWER9 processor compatibility mode with an OS level that supports the External Interrupt Virtualization Engine (XIVE) exploitation mode.  For more information on compatibility modes, see the following two articles in the IBM Knowledge Center:
    1) Processor compatibility mode overview:   https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcm.htm
    2) Processor compatibility mode definitions:  https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcmdefs.htm
  • A problem was fixed for an intermittent IPL failure with SRC B181E540 logged with fault signature " ex(n2p1c0) (L2FIR[13]) NCU Powerbus data timeout".  No FRU is called out.  The error may be ignored and the re-IPL is successful.  The error occurs very infrequently.  This is the second iteration of the fix that has been released.  Expedient routing of the Powerbus interrupts did not occur in all cases in the prior fix, so the timeout problem was still occurring.
System firmware changes that affect certain systems
  • DEFERRED:  For systems using the Feature Code #EPIM 8-core processor, a problem was fixed for a slightly degraded UtlraTurbo maximum frequency (approximately 3% less) compared to what is expected for this processor chip.  The fix requires new Workload Optimized Frequency (WOF) tables for the processor, so the system must be re-IPLed for the installed fix to be active.  The WOF UltraTurbo maximum frequency can only be achieved when turning cores off or operating below the 50% workload capacity of the system.
  • On systems running IBM i partitions configured as Restricted I/O partitions that are also running in either P7 or P8 processor compatibility mode, a problem was fixed for a likely hang during boot with BA210000 and BA218000 checkpoints and error logs after migrating to FW940.00 level system firmware.  The trigger for the problem is booting IBMi partitions configured as Restricted I/O partitions in P7 or P8 compatibility mode on FW940.00 system firmware.  Such partitions are usually configured this way so that they can be used for live partition migration (LPM) to and
    from P7/P8 systems. Without the fix, the user can do either of the following as circumventions for the boot failure of the IBM i partition:
    1) Move the partition to P9 compatibility mode
    2) Or remove the 'Restricted I/O Partition' property
VL940_027_027 / FW940.00

11/25/19		 Impact:  New      Severity:  New

GA Level with key features included listed below
  • All features and fixes from the FW930.11. service pack (and below) are included in this release.  At the time of the FW940.00 release, the FW930.11 is a future FW930 service pack scheduled for the fourth quarter of 2019.
New Features and Functions
  • User Mode NX Accelerator Enablement for PowerVM.  This enables the access of NX accelerators such as the gzip engine through user mode interfaces.  The IBM Virtual HMC (vHMC) 9.1.940 provides a user interface to this feature.  The LPAR must be running in POWER9 compatibility mode to use this feature.  For more information on compatibility modes, see the following two articles in the IBM Knowledge Center:
    1) Processor compatibility mode overview:    https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcm.htm
    2) Processor compatibility mode definitions:  https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcmdefs.htm
  • Support for SR-IOV logical ports in IBM i restricted I/O mode.
  • Support for user mode enablement of the External Interrupt Virtualization Engine (XIVE).  This user mode enables the management of interrupts to move from the hypervisor to the operating system for improved efficiency.  Operating systems may also have to be updated to enable this support.  The LPAR must be running in POWER9 compatibility mode to use this feature.  For more information on compatibility modes, see the following two articles in the IBM Knowledge Center:
    1) Processor compatibility mode overview:    https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcm.htm
    2) Processor compatibility mode definitions:  https://www.ibm.com/support/knowledgecenter/POWER9/p9hc3/p9hc3_pcmdefs.htm
  • Extended support for PowerVM Firmware Secure Boot.  This feature restricts access to the Open Firmware prompt and validates all adapter boot driver code. Boot adapters, or adapters which may be used as boot adapters in the future, must be updated to the latest microcode from IBM Fix Central.  The latest microcode will ensure the adapters support the Firmware Secure Boot feature of Power Systems. This requirement applies when updating system firmware from a level prior to FW940 to levels FW940 and later.  The latest adapter microcode levels include signed boot driver code.  If a boot-capable PCI adapter is not installed with the latest level of adapter microcode, the partition which owns the adapter will boot, but error logs with SRCs BA5400A5 or BA5400A6 will be posted. Once the adapter(s) are updated, the error logs will no longer be posted.
  • Linux OS support was added for PowerVM LPARs for the PCIe4 2x100GbE ConnectX-5 RoCE adapter with feature codes of #EC66/EC67 and CCIN 2CF3.  Linux versions RHEL 7.5 and SLES 12.3 are supported.
System firmware changes that affect all systems
  • A problem was fixed for incorrect call outs for PowerVM hypervisor terminations with SRC B7000103 logged.  With the fix, the call outs are changed from SVCDOCS, FSPSP04, and FSPSP06 to FSPSP16.  When this type of termination occurs, IBM support requires the dumps be collected to determine the cause of failure.
  • A problem was fixed for an IPL failure with the following possible SRCs logged:  11007611, 110076x1, 1100D00C, and 110015xx.  The service processor may reset/reload for this intermittent error and end up in the termination state.

4.0 How to Determine The Currently Installed Firmware Level

You can view the server's current firmware level on the Advanced System Management Interface (ASMI) Welcome pane. It appears in the top right corner. Example: VL910_123.

5.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

Note: If your HMC is not internet-connected you will need to download the new firmware level to a USB flash memory device or ftp server.

6.0 Installing the Firmware

The method used to install new firmware will depend on the release level of firmware which is currently installed on your server. The release level can be determined by the prefix of the new firmware's filename.

Example: VLxxx_yyy_zzz

Where xxx = release level

HMC Managed Systems:

Instructions for installing firmware updates and upgrades on systems managed by an HMC can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eh6/p9eh6_updates_sys.htm


NovaLink Managed Systems:

A NovaLink managed system does not have a HMC attached and is managed either by PowerVM Novalink or PowerVC using PowerVM Novalink.
Instructions for installing firmware updates and upgrades on systems managed by PowerVM NovaLink can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eig/p9eig_updating_firmware.htm


HMC and NovaLink Co-Managed Systems:

 A co-managed system is managed by HMC and NovaLink, with one of the interfaces in the co-management master mode.
Instructions for installing firmware updates and upgrades on systems co-managed by an HMC and Novalink is the same as above for a HMC managed systems since the firmware update must be done by the HMC in the co-management master mode.  Before the firmware update is attempted, one must be sure that HMC is set in the master mode using the steps at the following IBM KnowledgeCenter link for NovaLink co-managed systems:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eig/p9eig_kickoff.htm

Then the firmware updates can proceed with the same steps as for the HMC managed systems:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9eh6/p9eh6_updates_sys.htm


Systems not Managed by an HMC or NovaLink:

Power Systems:

Instructions for installing firmware on systems that are not managed by an HMC can be found at:
https://www.ibm.com/support/knowledgecenter/9009-22A/p9ha5/fix_serv_firm_kick.htm


Systems running Ubuntu operating system:

If Ubuntu will be used to update the system firmware, please follow these instructions to extract the installable binary and update/upgrade the firmware:

1) Download the .gz (tarball) from Fix Central to your Ubuntu system (ie, to /tmp/fwupdate).

2) Extract the .gz file to /tmp/ on the Ubuntu system:

Example:
    tar -xzf /tmp/fwupdate/01VL910_040_040.tar.gz -C /tmp/fwupdate

3) Use update_flash -v -f <extracted file name> to verify the package.

4) Update your firmware using update_flash:

/usr/sbin/update_flash  -f <extracted file name>

System will reboot during the firmware update. When the system reaches Ubuntu run-time state, you can then commit or reject the firmware update:
Commit: /usr/sbin/update_flash -c
Reject:   /usr/sbin/update_flash -r

IBM i Systems:

For information concerning IBM i Systems, go to the following URL to access Fix Central: 
http://www-933.ibm.com/support/fixcentral/

Choose "Select product", under Product Group specify "System i", under Product specify "IBM i", then Continue and specify the desired firmware PTF accordingly.

7.0 Firmware History

The complete Firmware Fix History (including HIPER descriptions)  for this Release level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VL-Firmware-Hist.html