Power9 System Firmware

Applies to:   9080-M9S

This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.


Contents


1.0 Systems Affected

This package provides firmware for Power Systems E980 (9080-M9S) servers only.

The firmware level in this package is:

1.1 Minimum HMC Code Level

This section is intended to describe the "Minimum HMC Code Level" required by the System Firmware to complete the firmware installation process. When installing the System Firmware, the HMC level must be equal to or higher than the "Minimum HMC Code Level" before starting the system firmware update.  If the HMC managing the server targeted for the System Firmware update is running a code level lower than the "Minimum HMC Code Level" the firmware update will not proceed.

The Minimum HMC Code levels for this firmware for HMC x86,  ppc64 or ppc64le are listed below.

x86 -  This term is used to reference the legacy HMC that runs on x86/Intel/AMD hardware for both the 7042 Machine Type appliances and the Virtual HMC that can run on the Intel hypervisors (KVM, VMWare, Xen).
ppc64 or ppc64le - describes the Linux code that is compiled to run on Power-based servers or LPARS (Logical Partitions)

For information concerning HMC releases and the latest PTFs,  go to the following URL to access Fix Central:
http://www-933.ibm.com/support/fixcentral/

For specific fix level information on key components of IBM Power Systems running the AIX, IBM i and Linux operating systems, we suggest using the Fix Level Recommendation Tool (FLRT):
http://www14.software.ibm.com/webapp/set2/flrt/home


NOTES:

                -You must be logged in as hscroot in order for the firmware installation to complete correctly.
                - Systems Director Management Console (SDMC) does not support this System Firmware level.

2.0 Important Information

NovaLink levels earlier than "NovaLink 1.0.0.16 Feb 2020 release" with partitions running certain SR-IOV capable adapters is NOT supported at this firmware release

NovaLink levels earlier than "NovaLink 1.0.0.16 Feb 2020 release" do not support IO adapter FCs EC2R/EC2S, EC2T/EC2U, EC3L/EC3M, EC66/EC67 with FW930 and later.   If the adapter was already in use with FW910/920 at an older NovaLink level, upgrading to FW930/940 will result in errors in NovaLink and PowerVC which causes the loss of any management operation via NovaLink / PowerVC combination.  Upgrading systems in this configuration is not supported at the older NovaLink levels.  If the system is required to be at FW930/940 or was shipped with FW930/940, NovaLink must first be updated to "NovaLink 1.0.0.16 Feb 2020 release"  or later.

Possible partition crash when using Live Partition Mobility (LPM) or partition hang when doing a concurrent firmware update

A very intermittent issue has been found in the IBM lab when using partition mobility or firmware update.  For a mobility operation, the issue can result in a partition crash if the mobility target system is FW930.00, FW930.01 or FW930.02.  For a code update operation, the partition may hang.  The recovery is to reboot the partition after the crash or hang.  This problem is fixed in service pack FW930.03.

Downgrading firmware from any given release level to an earlier release level is not recommended

Firmware downgrade warnings:
1) Adapter feature codes (#EC2R/#EC2S/#EC2T/#EC2U and #EC3L/#EC3M and #EC66/EC67) when configured in SR-IOV shared mode in FW930 or later, even if originally configured in shared mode in a pre-FW930 release, may not function properly if the system is downgraded to a pre-FW930 release. The adapter should be configured in dedicated mode first (i.e. take the adapter out of SR-IOV shared mode) before downgrading to a pre-FW930 release.
2) If partitions have been run in POWER9 compatibility mode in FW940, a downgrade to an earlier release (pre-FW940) may cause a problem with the partitions starting.  To prevent this problem, the "server firmware" settings must be reset by rebooting partitions in "Power9_base" before doing the downgrade.

If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.

2.1 IPv6 Support and Limitations

IPv6 (Internet Protocol version 6) is supported in the System Management Services (SMS) in this level of system firmware. There are several limitations that should be considered.

When configuring a network interface card (NIC) for remote IPL, only the most recently configured protocol (IPv4 or IPv6) is retained. For example, if the network interface card was previously configured with IPv4 information and is now being configured with IPv6 information, the IPv4 configuration information is discarded.

A single network interface card may only be chosen once for the boot device list. In other words, the interface cannot be configured for the IPv6 protocol and for the IPv4 protocol at the same time.

2.2 Concurrent Firmware Updates

Concurrent system firmware update is supported on HMC Managed Systems only.

Ensure that there are no RMC connections issues for any system partitions prior to applying the firmware update.  If there is a RMC connection failure to a partition during the firmware update, the RMC connection will need to be restored and additional recovery actions for that partition will be required to complete partition firmware updates.

2.3 Memory Considerations for Firmware Upgrades

Firmware Release Level upgrades and Service Pack updates may consume additional system memory.
Server firmware requires memory to support the logical partitions on the server. The amount of memory required by the server firmware varies according to several factors.
Factors influencing server firmware memory requirements include the following:
Generally, you can estimate the amount of memory required by server firmware to be approximately 8% of the system installed memory. The actual amount required will generally be less than 8%. However, there are some server models that require an absolute minimum amount of memory for server firmware, regardless of the previously mentioned considerations.

Additional information can be found at:
https://www.ibm.com/support/knowledgecenter/9080-M9S/p9hat/p9hat_lparmemory.htm

2.4 SBE Updates

Power 9 servers contain SBEs (Self Boot Engines) and are used to boot the system.  SBE is internal to each of the Power 9 chips and used to "self boot" the chip.  The SBE image is persistent and is only reloaded if there is a system firmware update that contains a SBE change.  If there is a SBE change and system firmware update is concurrent, then the SBE update is delayed to the next IPL of the CEC which will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  If there is a SBE change and the system firmware update is disruptive, then SBE update will cause an additional 3-5 minutes per processor chip in the system to be added on to the IPL.  During the SBE update process, the HMC or op-panel will display service processor code C1C3C213 for each of the SBEs being updated.  This is a normal progress code and system boot should be not be terminated by the user. Additional time estimate can be between 12-20 minutes per drawer or up to 48-80 minutes for maximum configuration.

The SBE image is only updated with this service pack if the starting firmware level is less than FW930.11.


3.0 Firmware Information

Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.

For systems that are not managed by an HMC, the installation of system firmware is always disruptive.

Note: The concurrent levels of system firmware may, on occasion, contain fixes that are known as Deferred and/or Partition-Deferred. Deferred fixes can be installed concurrently, but will not be activated until the next IPL. Partition-Deferred fixes can be installed concurrently, but will not be activated until a partition reactivate is performed. Deferred and/or Partition-Deferred fixes, if any, will be identified in the "Firmware Update Descriptions" table of this document. For these types of fixes (Deferred and/or Partition-Deferred) within a service pack, only the fixes in the service pack which cannot be concurrently activated are deferred.

Note: The file names and service pack levels used in the following examples are for clarification only, and are not necessarily levels that have been, or will be released.

System firmware file naming convention:

01VHxxx_yyy_zzz

NOTE: Values of service pack and last disruptive service pack level (yyy and zzz) are only unique within a release level (xxx). For example, 01VH900_040_040 and 01VH910_040_045 are different service packs.

An installation is disruptive if:

            Example: Currently installed release is 01VH900_040_040, new release is 01VH910_050_050.

            Example: VH910_040_040 is disruptive, no matter what level of VH910 is currently installed on the system.

            Example: Currently installed service pack is VH910_040_040 and new service pack is VH910_050_045.

An installation is concurrent if:

The release level (xxx) is the same, and
The service pack level (yyy) currently installed on the system is the same or higher than the last disruptive service pack level (zzz) of the service pack to be installed.

Example: Currently installed service pack is VH910_040_040, new service pack is VH910_041_040.

3.1 Firmware Information and Description

 
Filename Size Checksum md5sum
01VH930_101_035.rpm 141781189 17851
a93862d14477de44caefd322722d41f2

Note: The Checksum can be found by running the AIX sum command against the rpm file (only the first 5 digits are listed).
ie: sum 01VH930_101_035.rpm

VH930
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

The complete Firmware Fix History for this Release Level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VH-Firmware-Hist.html
VH930_101_035 / FW930.20

02/27/20
Impact: Availability       Severity: HIPER

New features and functions
  • Support was added for real-time data capture for PCIe3 expansion drawer (#EMX0) cable card connection data via resource dump selector on the HMC or in ASMI on the service processor.  Using the resource selector string of "xmfr -dumpccdata" will non-disruptively generate an RSCDUMP type of dump file that has the current cable card data, including data from cables and the retimers.
  • Support was added for redundant VPD EEPROMs.  If the primary module VPD EEPROM fails, the system will automatically change to the backup module.
System firmware changes that affect all systems
  • HIPER/Pervasive:  A problem was fixed for a possible system crash and HMC "Incomplete" state when a logical partition (LPAR) power off after a dynamic LPAR (DLPAR) operation fails for a PCIe adapter.  This scenario is likely to occur during concurrent maintenance of PCIe adapters or for #EMX0 components such as PCIe3 Cable adapters, Active Optical or copper cables, fanout modules, chassis management cards, or midplanes.  The DLPAR fail can leave page table mappings active for the adapter, causing the problems on the power down of the LPAR.  If the system does not crash, the DLPAR will fail if it is retried until a platform IPL is performed.
  • HIPER/Pervasive:  A problem was fixed for an HMC "Incomplete" state for a system after the HMC user password is changed with ASMI on the service processor.  This problem can occur if the HMC password is changed on the service processor but not also on the HMC, and a reset of the service processor happens.  With the fix, the HMC will get the needed "failed authentication" error so that the user knows to update the old password on the HMC.
  • DEFERRED:  A problem was fixed for a processor core failure with SRCs B150BA3C and BC8A090F logged that deconfigures the entire processor for the current IPL.  A re-IPL of the system will recover the lost processor with only the bad core guarded.
  • A problem was fixed for certain SR-IOV adapters that can have an adapter reset after a mailbox command timeout error.
    This fix updates the adapter firmware to 11.2.211.39  for the following Feature Codes and CCINs: #EN15/EN16 with CCIN 2CE3, #EN17/EN18 with CCIN 2CE4, #EN0H/EN0J with CCIN 2B93, #EN0M/EN0N with CCIN 2CC0, and #EN0K/EN0L with CCIN 2CC1.
    The SR-IOV adapter firmware level update for the shared-mode adapters happens under user control to prevent unexpected temporary outages on the adapters.  A system reboot will update all SR-IOV shared-mode adapters with the new firmware level.  In addition, when an adapter is first set to SR-IOV shared mode, the adapter firmware is updated to the latest level available with the system firmware (and it is also updated automatically during maintenance operations, such as when the adapter is stopped or replaced).  And lastly, selective manual updates of the SR-IOV adapters can be performed using the Hardware Management Console (HMC).  To selectively update the adapter firmware, follow the steps given at the IBM Knowledge Center for using HMC to make the updates:   https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
    Note: Adapters that are capable of running in SR-IOV mode, but are currently running in dedicated mode and assigned to a partition, can be updated concurrently either by the OS that owns the adapter or the managing HMC (if OS is AIX or VIOS and RMC is running).
  • A problem was fixed for an SR-IOV adapter failure with B400FFxx errors logged when moving the adapter to shared mode.  This is an infrequent race condition where the adapter is not yet ready for commands and it can also occur during EEH error recovery for the adapter.  This affects the SR-IOV adapters with the following feature codes and CCINs:   #EC2R/EC2S with CCIN 58FA; #EC2T/EC2U with CCIN 58FB; #EC3L/EC3M with CCIN 2CEC; and #EC66/EC67 with CCIN 2CF3.
  • A problem was fixed for an IPL failure with the following possible SRCs logged:  11007611, 110076x1, 1100D00C, and 110015xx.  The service processor may reset/reload for this intermittent error and end up in the termination state.
  • A problem was fixed for a failed clock card causing a node to be guarded during the IPL of a multi-node system.  With the fix, the redundant clock card allows all the nodes to IPL in the case of a single clock card failure.
  • A problem was fixed for the green power LED on the System Control Unit  (SCU) not being lit even though the system is powered on.  Without the fix, the LED is always in the off state.
  • A problem was fixed for delayed interrupts on a Power9 system following a Live Partition Mobility operation from a Power7 or Power8 system.  The delayed interrupts could cause device time-outs, program dispatching delays, or other device problems on the target Power9 system.
  • A problem was fixed for processor cores not being able to be used by dedicated processor partitions if they were DLPAR removed from a dedicated processor partition.  This error can occur if there was a firmware assisted dump or a Live Partition Mobility (LPM) operation after the DLPAR of the processor.  A re-IPL of the system will recover the processor cores.
  • A problem was fixed for a B7006A96 fanout module FPGA corruption error that can occur in unsupported PCIe3 expansion drawer(#EMX0) configurations that mix an enhanced PCIe3 fanout module (#EMXH) in the same drawer with legacy PCIe3 fanout modules (#EMXF, #EMXG, #ELMF, or #ELMG).  This causes the FPGA on the enhanced #EMXH to be updated with the legacy firmware and it becomes a non-working and unusable fanout module.  With the fix, the unsupported #EMX0 configurations are detected and handled gracefully without harm to the FPGA on the enhanced fanout modules.
  • A problem was fixed for lost interrupts that could cause device time-outs or delays in dispatching a program process.  This can occur during memory operations that require a memory relocation for any partition such as mirrored memory defragmentation done by the HMC optmem command, or memory guarding that happens as part of memory error recovery during normal operations of the system.
  • A problem was fixed for extraneous informational logging of SRC B7006A10 ("Insufficient SR-IOV resources available") with a 1306 PRC.  This SRC is logged whenever an SR-IOV adapter is moved from dedicated mode to shared mode.  This SRC with the 1306 PRC should be ignored as no action is needed and there is no issue with SR-IOV resources.
  • A problem was fixed for a hypervisor error during system shutdown where a B7000602 SRC is logged and the system may also briefly go "Incomplete" on the HMC but the shutdown is successful.  The system will power back on with no problems so the SRC can be ignored if it occurred during a shutdown.
  • A problem was fixed for possible dispatching delays for partitions running in POWER8, POWER9_base or POWER9 processor compatibility mode.
  • A problem was fixed for extraneous B400FF01 and B400FF02 SRCs logged when moving cables on SR-IOV adapters.  This is an infrequent error that can occur if the HMC performance monitor is running at the same time the cables are moved.  These SRCs can be ignored when accompanied by cable movement.
System firmware changes that affect certain systems
  • On systems with an IBM i partition, a problem was fixed that occurs after a Live Partition Mobility (LPM) of an IBM i partition that may cause issues including dispatching delays and the inability to do further LPM operations of that partition.  The frequency of this problem is rare.  A partition encountering this error can be recovered with a reboot of the partition.
  • On systems with an IBM i partition, a problem was fixed for a D-mode IPL failure when using a USB DVD drive in an IBM 7226 multimedia storage enclosure.  Error logs with SRC BA16010E, B2003110, and/or B200308C can occur.  As a circumvention, an external DVD drive can be used for the D-mode IPL.
  • On systems with an IBM i partition, a problem was fixed for a possibly incorrect number of Memory COD (Capacity On Demand) resources shown when gathering performance data with IBM i Collection Services. Memory resources activated by Power Enterprise Pools (PEP) 1.0 will be missing from the data.  An error was corrected in the IBM i MATMATR option 0X01F6 that retrieves the Memory COD information for the Collection Services.
  • On systems with Integrated Facility for Linux ( IFL) processors and Linux-only partitions, a problem was fixed for Power Enterprise Pools (PEP) 1.0 not going back into "Compliance" when resources are moved from Server 1 to Server 2, causing an expected "Approaching Out Of Compliance", but not automatically going back into compliance when the resources are no longer used on Server 1.  As a circumvention, the user can do an extra "push" and "pull" of one resource to make the Pool discover it is back in "Compliance".
VH930_093_035 / FW930.11

12/11/19
Impact: Availability       Severity: SPE

System firmware changes that affect all systems
  • DEFERRED: PARTITION_DEFERRED:  A problem was fixed for vHMC having no useable local graphics console when installed on FW930.00 and later partitions.
  • DEFERRED:  A problem was fixed for not being able to do a HMC exchange FRU for the PCIe cassette in P1-C3 if a PCIe to USB conversion card (CCIN 6B6C) is not installed in P1-C13.  In this situation, the P1-C3 location is not provided in the FRU selection list.  An alternative procedure to accomplish the same task would be to do an exchange FRU on the PCIe adapter P1-C3-C1 in the PCIe cassette.
  • DEFERRED:  A problem was fixed for rare system checkstops triggered by SMP cable failure or when one of the cables is not properly secured in place.
  • A problem was fixed for the Advanced System Management Interface (ASMI) showing an "Unknown" in the Deconfiguration records if a SMP group (SMPGROUP) unit is guarded.  With the fix, "OBUS End Point" will be displayed instead of "Unknown".
  • A problem was fixed for the Advanced System Management Interface (ASMI) menu for "PCIe Hardware Topology/Reset link" showing the wrong value.  This value is always wrong without the fix.
  • A problem was fixed for a local clock card (LCC)  failure that results in a failed service processor failover and a system that does not IPL or takes several hours to IPL.  With the fix, missing local clock card data is made available to the backup service processor so that the failover can succeed, allowing the system to IPL.
  • A problem was fixed for PLL unlock error with SRC B124E504 causing a secondary error of PRD Internal Firmware Software Fault with SRC B181E580 and incorrect FRU call outs.
  • A problem was fixed for an initialization failure of certain SR-IOV adapter ports during its boot, causing a B400FF02 SRC to be logged.  This is a rare problem and it recovers automatically by the reboot of the adapter on the error.  This problem affects the SR-IOV adapters with the following feature codes and CCINs: #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC;  and #EC66/EC67 with CCIN 2CF3.
  • A problem was fixed for the SR-IOV Virtual Functions (VFs) when the multi-cast promiscuous flag has been turned on for the VF.  Without the fix,  the VF device driver sometimes may erroneously fault when it senses that the multi-cast promiscuous mode had not been achieved although it had been requested.
  • A problem was fixed for SR-IOV adapters to provide a consistent Informational message level for cable plugging issues.  For transceivers not plugged on certain SR-IOV adapters, an unrecoverable error (UE) SRC B400FF03 was changed to an Informational message logged.  This affects the SR-IOV adapters with the following feature codes and CCINs: #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC;  and #EC66/EC67 with CCIN 2CF3.
    For copper cables unplugged on certain SR-IOV adapters, a missing message was replaced with an Informational message logged.  This affects the SR-IOV adapters with the following feature codes and CCINs: #EN17/EN18 with CCIN 2CE4, and #EN0K/EN0L with CCIN 2CC1.
  • A problem was fixed for incorrect DIMM callouts for DIMM over-temperature errors.  The error log for the DIMM over temperature will have incorrect FRU callouts, either calling out the wrong DIMM or the wrong DIMM controller memory buffer.
  • A problem was fixed for an Operations Panel hang after using it set LAN Console as the console type for several iterations.  After several iterations, the operations panel may hang with "Function 41" displayed on the operations panel.  A hot unplug and plug of the operations panel can be used to recover it from the hang. 
  • A problem was fixed for shared processor pools where uncapped shared processor partitions placed in a pool may not be able to consume all available processor cycles.  The problem may occur when the sum of the allocated processing units for the pool member partitions equals the maximum processing units of the pool.
  • A problem was fixed for Novalink failing to activate partitions that have names with character lengths near the maximum allowed character length.  This problem can be circumvented by changing the partition name to have 32 characters or less.
  • A problem was fixed where a Linux or AIX partition type was incorrectly reported as unknown.  Symptoms include: IBM Cloud Management Console (CMC) not being able to determine the RPA partition type (Linux/AIX) for partitions that are not active; and HMC attempts to dynamically add CPU to Linux partitions may fail with a HSCL1528 error message stating that there are not enough Integrated Facility for Linux ( IFL) cores for the operation.
  • A problem was fixed for a possible activation code memory conversion sequence number error when creating a Power Enterprise Pool (PEP) 1.0 Pool for a set of servers.  This can happen if Perm Memory activations were purchased local to a server but then needed to be converted from Perm MEM to Mobile PEP Mem state for pool use.  The deployment of the PEP fails with the following messages on the HMC:
    1)  HSCL9017 HSCL0521 A Mobile CoD memory conversion code to convert 100 GB of permanently activated memory to Mobile CoD memory on the managed system has been entered. The sequence number of the CoD code indicates that this code has been used before. Obtain a new CoD code and try again.
    2)  HSCL9119 The Mobile CoD memory activation code for the Power enterprise pool was not entered because a permanent to Mobile CoD memory conversion code for a server could not be entered.
    To recover from this error,  request a new XML file from IBM with an updated Memory Conversion activation code.
  • A problem was fixed for a hypervisor hang that can occur on the target side when doing a Live Partition Mobility (LPM) migration from a system that does not support encryption and compression of LPM data.  If the hang occurs, the HMC will go to an "Incomplete" state for the target system.  The problem is rare because the data from the source partition must be in a very specific pattern to cause the failure.  When the failure occurs, a B182951C will be logged on the target (destination) system and the HMC for the source partition will issue the following message:  "HSCLA318 The migration command issued to the destination management console failed with the following error: HSCLA228 The requested operation cannot be performed because the managed system <system identifier> is not in the Standby or Operating state.".  To recover, the target system must be re-IPLed.
  • A problem was fixed for performance collection tools not collecting data for event-based options.  This fix pertains to perfpmr and tprof on AIX, and Performance Explorer (PEX) on IBM i.
  • A problem was fixed for a SRC reminder that keeps repeating for B150F138 even after a UPIC cable has been repaired or replaced.  Without the fix, a hot-plug of a UPIC cable, while the system is running, will not get verified as the cable being fixed until the system is re-IPLed, so the initial error SRC for the missing or bad UPIC cable will be posted repeatedly until the re-IPL occurs.
  • A problem was fixed a Live Partition Mobility (LPM) migration of a large memory partition to a target system that causes the target system to crash and for the HMC to go to the "Incomplete" state.  For servers with the default LMB size (256MB), if the partition is >=16TB and if the desired memory is different than the maximum memory, LPM may fail on the target system.  Servers with LMB sizes less than the default could hit this problem with smaller memory partition sizes.  A circumvention to the problem is to set the desired and maximum memory to the same value for the large memory partition that is to be migrated.
  • A problem was fixed for certain SR-IOV adapters with the following issues:
    1) If the SR-IOV logical port's VLAN ID (PVID) is modified while the logical port is configured, the adapter will use an incorrect PVID for the Virtual Function (VF).  This problem is rare because most users do not change the PVID once the logical port is configured, so they will not have the problem.
    2) Adapters with an SRC of B400FF02 logged.
    This fix updates the adapter firmware to 11.2.211.38  for the following Feature Codes and CCINs: #EN15/EN16 with CCIN 2CE3,  #EN17/EN18 with CCIN 2CE4, #EN0H/EN0J with CCIN 2B93, #EN0M/EN0N with CCIN 2CC0, and #EN0K/EN0L with CCIN 2CC1.
    The SR-IOV adapter firmware level update for the shared-mode adapters happens under user control to prevent unexpected temporary outages on the adapters.  A system reboot will update all SR-IOV shared-mode adapters with the new firmware level.  In addition, when an adapter is first set to SR-IOV shared mode, the adapter firmware is updated to the latest level available with the system firmware (and it is also updated automatically during maintenance operations, such as when the adapter is stopped or replaced).  And lastly, selective manual updates of the SR-IOV adapters can be performed using the Hardware Management Console (HMC).  To selectively update the adapter firmware, follow the steps given at the IBM Knowledge Center for using HMC to make the updates:   https://www.ibm.com/support/knowledgecenter/en/POWER9/p9efd/p9efd_updating_sriov_firmware.htm.
    Note: Adapters that are capable of running in SR-IOV mode, but are currently running in dedicated mode and assigned to a partition, can be updated concurrently either by the OS that owns the adapter or the managing HMC (if OS is AIX or VIOS and RMC is running).
  • A problem was fixed for certain SR-IOV adapters where after some error conditions the adapter may hang with no messages or error recovery.  This is a rare problem for certain severe adapter errors.  This problem affects the SR-IOV adapters with the following feature codes:   #EC66/EC67 with CCIN 2CF3.  This problem can be recovered by removing the adapter from SR-IOV mode and putting it back in SR-IOV mode, or the system can be re-IPLed.
  • A problem was fixed for an initialization failure of certain SR-IOV adapters when changed into SR-IOV mode.  This is an infrequent problem that most likely can occur following a concurrent firmware update when the adapter also needs to be updated. This problem affects the SR-IOV adapter with the following feature codes and CCINs: #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC;  and #EC66/EC67 with CCIN 2CF3.  This problem can be recovered by removing the adapter from SR-IOV mode and putting it back in SR-IOV mode, or the system can be re-IPLed.
  • A problem was fixed for a missing IBM Power Enterprise Pools 2.0 (PEP 2.0) message when the PEP 2.0 subscription expires and the performance of the system is throttled.  Also, at the time of the throttle, the "remaining days" of the subscription still shows as "remaining 1 day".  This problem can only occur at least 90 days after PEP 2.0 has been started and if the PEP 2.0 subscription is allowed to expire.
  • A problem was fixed for an IBM Power Enterprise Pools 2.0 (PEP 2.0) extra performance "throttle ended" message when performance throttling is removed.  The extra "throttle ended" message, which shares nearly the same timestamp of the first, can be ignored.
  • A problem was fixed for a bad SMP cable causing a B114DA62 SRC to be logged without a correct FRU call out.  Without the fix, IBM support may be needed to isolate the bad part that needs to be replaced.
  • A problem was fixed for a rare SMP link initialization failure during an IPL reported with either SRC B114DA62 during IPL or BC14E540 immediately after IPL.  Without the fix, a system re-IPL is required to recover the operation of the SMP cable.
  • A problem was fixed for persistent high fan speeds in the system after a service processor failover.  To restore the fans to normal speed without re-IPLing the system requires the following steps:
    1) Use ASMI to perform a soft reset of the backup service processor.
    2) When the backup service processor has completed its reset, use the HMC to do an administrative failover, so that the reset service processor becomes the primary.
    3) Use ASMI to perform a soft reset on the new backup service processor.  When this has completed, system fan speeds should be back to normal.
  • A problem was fixed for a rare IPL failure with SRCs BC8A090F and BC702214 logged caused by an overflow of VPD repair data for the processor cores.  A re-IPL of the system should recover from this problem.
  • A problem was fixed for a false memory error that can be logged during the IPL with SRC BC70E540 with the description "mcb(n0p0c1) (MCBISTFIR[12]) WAT_DEBUG_ATTN" but with no hardware call outs.  This error log can be ignored.
  • A problem was fixed for an IPL failure after installing DIMMs of different sizes, causing memory access errors.  Without the fix, the memory configuration should be restored to only use DIMMs of the same size.
  • A problem was fixed for a memory DIMM plugging rule violation that causes the IPL to terminate with an error log with RC_GET_MEM_VPD_UNSUPPORTED_CONFIG IPL that calls out the memory port but has no DIMM call outs and no DIMM deconfigurations are done.  With the fix, the DIMMs that violate the plugging rules will be deconfigured and the IPL will complete.  Without the fix, the memory configuration should be restored to the prior working configuration to allow the IPL to be successful.
  • A problem was fixed for a B7006A22 Recoverable Error for the enhanced PCIe3 expansion drawer (#EMX0) I/O drawer with fanout PCIe Six Slot Fan Out modules (#EMXH) installed. This can occur up to two hours after an IPL from power off.   This can be a frequent occurrence on an IPL for systems that have the PCIe Six Slot Fan Out module (#EMXH).  The error is automatically recovered at the hypervisor level.  If an LPAR fails to start after this error, a restart of the LPAR is needed.
  • A problem was fixed for degraded memory bandwidth on systems with memory that had been dynamically repaired with symbols to mark the bad bits.
  • A problem was fixed for processor or memory VRM power faults that cause a guard of a node without a deconfiguration of the node.  This causes the degraded IPL to go to termination when the faulty node is accessed during the IPL.  The bad node can be manually deconfigured to allow the IPL to succeed.
  • A problem was fixed for an intermittent IPL failure with SRC B181E540 logged with fault signature " ex(n2p1c0) (L2FIR[13]) NCU Powerbus data timeout".  No FRU is called out.  The error may be ignored since the automatic re-IPL is successful.  The error occurs very infrequently.  This is the second iteration of the fix that has been released.  Expedient routing of the Powerbus interrupts did not occur in all cases in the prior fix, so the timeout problem was still occurring.
  • A problem was fixed for a loss of service processor redundancy after a failover to the backup on a Hostboot IPL error.  Although the failover is successful to the backup service processor, the primary service processor may terminate.  The service processor can be recovered from termination by using a soft reset from ASMI.  This problem only can occur during the IPL, not at run time.  This problem is very rare and has only happened with IBM internal testing using firmware error injection to cause a TI during the Hostboot IPL.
System firmware changes that affect certain systems
  • On systems with PCIe3 expansion drawers(feature code #EMX0),  a problem was fixed for a concurrent exchange of a PCIe expansion drawer cable card, although successful, leaves the fault LED turned on.
  • On systems with 16TB or more of memory, a problem was fixed for certain SR-IOV adapters not being able to start a Virtual Function (VF) if "I/O Adapter Enlarged Capacity" is enabled and VF option 0  has been selected for the number of supported VFs .   This problem affects the SR-IOV adapters with the following feature codes and CCINs: #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC;  and #EC66/EC67 with CCIN 2CF3.  This problem can be circumvented by the following action:  change away from VF option 0.  VF option 1 is the default option and it will work.
  • On systems with 64TB of memory, a problem was fixed for certain SR-IOV adapters not being able to start a Virtual Function (VF) if "I/O Adapter Enlarged Capacity" is enabled.  This problem affects the SR-IOV adapters with the following feature codes and CCINs: #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC;  and #EC66/EC67 with CCIN 2CF3. This problem can be circumvented by changing the configuration such that the adapter does not have "I/O Adapter Enlarged Capacity" enabled.
  • On systems with 16GB huge-pages, a problem was fixed for certain SR-IOV adapters with all or nearly all memory assigned to them preventing a system IPL.  This affects the SR-IOV adapters with the following feature codes and CCINs: #EC2R/EC2S with CCIN 58FA;  #EC2T/EC2U with CCIN 58FB;  #EC3L/EC3M with CCIN 2CEC;  and #EC66/EC67 with CCIN 2CF3.  The problem can be circumvented by powering off the system and turning off all the huge-page allocations.
  • On systems running IBM i partitions, a problem was fixed for a NVME (Non-Volatile Memory Express) device load source not being found for an IBM i boot partition.  This can occur on a system with multiple load source candidates when the needed load source is not the first entry in the namespace ID list.  This problem can be circumvented by directing the System License Internal Code (SLIC) Bootloader to a specific namespace and bypassing a search.
  • On systems with IBM i partitions,  a problem was fixed for Live Partition Mobility (LPM) migrations that could have incorrect hardware resource information (related to VPD) in the target partition if a failover had occurred for the source partition during the migration.  This failover would have to occur during the Suspended state of the migration, which only lasts about a second, so this should be rare.  With the fix, at a minimum, the migration error will be detected to abort the migration so it can be restarted.  And at a later IBMi OS level, the fix will allow the migration to complete even though the failover has occurred during the Suspended state of the migration.
  • On systems running IBM i partitions, a problem was fixed for IBM i collection services that may produce incorrect instruction count results.
  • On systems running IBM i partitions, a performance improvement was made for the cache memory management of workloads that utilize heavy I/O operations.
  • On systems with IBM i partitions, a rare problem was fixed for an intermittent failure of a DLPAR remove of an adapter.  In most cases, a retry of the operation will be successful.
  • On systems with IBM i partitions, a problem was fixed that was allowing V7R1 to boot on or be migrated to POWER9 servers.  As documented in the System Software maps for IBM i (https://www-01.ibm.com/support/docview.wss?uid=ssm1platformibmi), V7R1 IBM i software is not supported on POWER9 servers.
  • On systems with IBM i partitions, a problem was fixed for a LPAR restart error after a DLPAR of an active adapter was performed and the LPAR was shut down.  A reboot of the system will recover the LPAR so it will start.
VH930_068_035 / FW930.03

08/22/19
Impact:  Data                  Severity:  HIPER

System firmware changes that affect all systems
  • HIPER/Pervasive:  A change was made to fix an intermittent processor anomaly that may result in issues such as operating system or hypervisor termination, application segmentation fault, hang, or undetected data corruption.  The only issues observed to date have been operating system or hypervisor terminations.
  • A problem was fixed for a very intermittent partition error when using Live Partition Mobility (LPM) or concurrent firmware update.  For a mobility operation, the issue can result in a partition crash if the mobility target system is FW930.00, FW930.01 or FW930.02.  For a code update operation, the partition may hang.  The recovery is to reboot the partition after the crash or hang.
VH930_048_035 / FW930.02

06/28/19
Impact: Availability       Severity: SPE

New Features and Functions
  • Support added for IBM Power Enterprise Pools 2.0.  This is a new IBM Power E980 offering designed to deliver enhanced multisystem resource sharing and by-the-minute consumption of on-premises Power E980 compute resources to clients deploying and managing a private cloud infrastructure.  Each Power Enterprise Pool (2.0) is monitored and managed from a Cloud Management Console in the IBM Cloud.  Capacity Credits may be purchased from IBM, an authorized IBM Business Partner, or online through the IBM Entitled System Support website, https://www.ibm.com/servers/eserver/ess/index.wss, where available.  Clients may more easily identify capacity usage and trends across their Power E980 systems in a pool by viewing web-accessible aggregated data without spreadsheets or custom analysis tools.
  • Support was added for a new P9 six core processor with CCIN 5C33.
System firmware changes that affect all systems
  • A problem was fixed for a bad link for the PCIe3 expansion drawer (#EMX0) I/O drawer with the clock enhancement causing a system failure with B700F103.  This error could occur during an IPL or a concurrent add of the link hardware.
  • A problem was fixed for On-Chip Controller (OCC) power capping operation time-outs with SRC B1112AD3 that caused the system to enter safe mode, resulting in reduced performance.  The problem only occurred when the system was running with high power consumption, requiring the need for OCC power capping.
  • A problem was fixed for the "PCIe Topology " option to get cable information in the HMC or ASMI that was returning the wrong cable part numbers if the PCIe3 expansion drawer (#EMX0) I/O drawer clock enhancement was configured.  If cables with the incorrect part numbers are used for an enhanced PCIe3 expansion drawer configuration, the hypervisor will log a B7006A20 with PRC 4152 indicating an invalid configuration - https://www.ibm.com/support/knowledgecenter/9080-M9S/p9eai/B7006A20.htm.
  • A problem was fixed for a drift in the system time (time lags and the clock runs slower than the true value of time) that occurs when the system is powered off to the service processor standby state.  To recover from this problem, the system time must be manually corrected using the Advanced System Management Interface (ASMI) before powering on the system.  The time lag increases in proportion to the duration of time that the system is powered off.
VH930_035_035 / FW930.00

05/17/19
Impact:  New      Severity:  New

All features and fixes from the FW920.30 service pack (and below) are included in this release.

New Features and Functions
  • Support was added to allow the FPGA soft error checking on the PCIe I/O expansion drawer (#EMX0) to be disabled with the help of IBM support using the hypervisor "xmsvc" macro.  This new setting will persist until it it is changed by the user or IBM support.  The effect of disabling FPGA soft error checking is to eliminate the FPGA soft error recovery which causes a recoverable PCIe adapter outage.  Some of the soft errors will be hidden by this change but others may have unpredictable results, so this should be done only under guidance of IBM support.
  • Support for the PCIe3 expansion drawer (#EMX0) I/O drawer clock enhancement so that a reset of the drawer does not affect the reference clock to the adapters so the PCIe lanes for the PCIe adapters can keep running through an I/O drawer FPGA reset.  To use this support, new cable cards, fanout modules, and optical cables are needed after this support is installed: PCIe Six Slot Fan out module(#EMXH) - only allowed to be connected to converter adapter cable card;  PCIe X16 to CXP Optical or CU converter adapter for the expansion drawer (#EJ19); and new AOC cables with feature/part number of #ECCR/78P6567, #ECCX/78P6568, #ECCY/78P6569, and #ECCZ/78P6570. These parts cannot be install concurrently, so a scheduled outage is needed to complete the migration.
  • Support added for RDMA Over Converged Ethernet (RoCE) for SR-IOV adapters.
  • Support added for SMS menu to enhance the I/O information option to have "vscsi" and "network" options.  The information shown for "vscsi" devices is similar to that provided for SAS and Fibre Channel devices.  The "network" option provides connectivity information for the adapter ports and shows which can be used for network boots and installs.
  • Support added to monitor the thermal sensors on the NVMe SSD drives (feature codes #EC5J, #EC5K, #EC5L) and use that information to adjust the speed of the system fans for improved cooling of the SSD drives.
  • Support added to allow integrated USB ports to be disabled.  This is available via an Advanced System Management Interface (ASMI) menu option:  "System Configuration -> Security -> USB Policy".  The USB disable policy, if selected, does not apply to pluggable USB adapters plugged into PCIe slots such as the 4-Port USB adapter (#EC45/#EC46), which are always enabled.
System firmware changes that affect all systems
  • A problem was fixed for a clock card failure with SRC B158CC62 logged calling out the wrong clock card and not calling out the cable and system backplane as needed.  This fix does not add processors to the callout but in some cases the processor has also been identified as the cause of the clock card failure.
  • A problem was fixed for a system IPLing with an invalid time set on the service processor that causes partitions to be reset to the Epoch date of 01/01/1970.  With the fix, on the IPL, the hypervisor logs a B700120x when the service processor real time clock is found to be invalid and halts the IPL to allow the time and date to be corrected by the user.  The Advanced System Management Interface (ASMI) can be used to correct the time and date on the service processor.  On the next IPL, if the time and date have not been corrected, the hypervisor will log a SRC B7001224 (indicating the user was warned on the last IPL) but allow the partitions to start, but the time and date will be set to the Epoch value.
  • A problem was fixed for a possible boot failure from a ISO/IEC 13346 formatted image, also known as Universal Disk Format (UDF).
    UDF is a profile of the specification known as ISO/IEC 13346 and is an open vendor-neutral file system for computer data storage for a broad range of media such as DVDs and newer optical disc formats.  The failure is infrequent and depends on the image.  In rare cases, the boot code erroneously fails to find a file in the current directory.  If the boot fails on a specific image, the boot of that image will always fail without the fix.
  • A problem was fixed for broadcast bootp installs or boots that fail with a UDP checksum error.
  • A problem was fixed for failing to boot from an AIX mksysb backup on a USB RDX drive with SRCs logged of BA210012, AA06000D, and BA090010.  The boot error does not occur if a serial console is used to navigate the SMS menus.

4.0 How to Determine The Currently Installed Firmware Level

You can view the server's current firmware level on the Advanced System Management Interface (ASMI) Welcome pane. It appears in the top right corner. Example: VH920_123.


5.0 Downloading the Firmware Package

Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.

Note: If your HMC is not internet-connected you will need to download the new firmware level to a USB flash memory device or ftp server.


6.0 Installing the Firmware

The method used to install new firmware will depend on the release level of firmware which is currently installed on your server. The release level can be determined by the prefix of the new firmware's filename.

Example: VHxxx_yyy_zzz

Where xxx = release level

Instructions for installing firmware updates and upgrades can be found at https://www.ibm.com/support/knowledgecenter/9080-M9S/p9eh6/p9eh6_updates_sys.htm

IBM i Systems:

For information concerning IBM i Systems, go to the following URL to access Fix Central: 
http://www-933.ibm.com/support/fixcentral/

Choose "Select product", under Product Group specify "System i", under Product specify "IBM i", then Continue and specify the desired firmware PTF accordingly.

7.0 Firmware History

The complete Firmware Fix History (including HIPER descriptions)  for this Release level can be reviewed at the following url:
http://download.boulder.ibm.com/ibmdl/pub/software/server/firmware/VH-Firmware-Hist.html

8.0 Change History

Date
Description
June 8, 2020
Fix descripton additions for level VH930_093_035 / FW930.11.
April 10, 2020 Added firmware level statement to section "2.4 SBE Updates".