From FC

To FC

Return Parts?

EP0K - 16-core 2.60 GHz

(3.09 GHz Turbo) POWER9 Processor

EP0P - 16-core 2.7 GHz

(3.3 GHz Turbo) POWER9 Processor

Yes

EP0M - 20-core 2.0 GHz

(2.87 GHz Turbo) POWER9 Processor

EP0R - 20-core 2.4 GHz

(3.0  GHz Turbo) POWER9 Processor

Yes

Filename

Size

Checksum

obmc-witherspoon-op940.40.ubi.mtd.tar

22937600

13a4ba15b3eaac71a537f688845ad347

witherspoon-IBM-OP9-v2.4.1-4.52_prod.pnor.squashfs.tar

30136320

be593205ab51e0ee710e5d900ce6c620

OP940
For Impact, Severity and other Firmware definitions, Please refer to the below 'Glossary of firmware terms' url:
http://www14.software.ibm.com/webapp/set2/sas/f/power5cm/home.html#termdefs

OP940.40

PNOR OP9-v2.4.1-4.52

with BMC op940.40

07/06/2022

Impact: Data        Severity:  HIPER

New features and functions

Support was added in Petitboot for a "-s" parameter on the "kexec" command. " kexec" now supports a "-s" option to perform a kexec_file_load syscall (in place of a kexec_load). This is triggered through the "-s" argument to "kexec".

Support was added in Petitboot GRUB2 to allow empty paths in the (device)/path syntax. Also added support for "test -e" with an empty path - this tests for the presence of the device itself.

Support was added for Workload Optimized Frequencies (WOF) for 24-core processors.

System firmware changes that affect all systems

HIPER/Pervasive: A problem was fixed for error recovery for an intermittent BMC hang that causes a flash side switch on reset of the BMC with loss of profile settings (including network settings). The intermittent BMC hang can occur while the host is rebooting as part of a host firmware update. With the fix, the recovery for a BMC hang is a BMC reset that is done without switching the BMC flash side, allowing the BMC to reboot cleanly.

A security problem was fixed for the BMC OpenSSL certificate parsing where a specially crafted certificate with invalid parameters could cause an infinite loop, and result in a denial of service condition on the BMC. The recovery is to reboot the BMC. The Common Vulnerability and Exposure number for this problem is CVE-2022-0778.

A problem was fixed in Petitboot discovery for crashing on a bad URL path for a DHCP server, leaving the system with only a shell running.

A problem was fixed in Petitboot discovery to log a warning message in the status log when a duplicate filesystem (FS) for a device is detected. The message is needed to alert the user that the new device will be ignored, as Petitboot suppresses the second (and subsequent identical) FS.

A problem was fixed for a Broadcom network adapter being shown as link down in Petitboot and not configurable, but in the OS the adapter works correctly. The error message logged in Petitboot is "bnx2x: [bnx2x_init_firmware:13480(enP52p1s0f0)]Can't load firmware file bnx2x/bnx2x-e2-7.13.15.0.fw". The adapter worked correctly for the OP940.20 firmware level but fails for OP940.30. The Linux 5.10 kernel requires a newer version of Broadcom bnx2x firmware. With the fix, the Broadcom bnx2x firmware version is updated to 7.13.15.0.

A problem was fixed for a failed correctable error recovery for a DIMM that causes a flood of SRC BC81E580 error logs and also can prevent dynamic memory deallocation from occurring for a hard memory error. This is a very rare problem caused by an unexpected number of correctable error symbols for the DIMM in the per-symbol counter registers.

A problem was fixed for system hangs with SRC BC13090F logged caused by CPU lockups. This problem is intermittent and random but it can be consistently recreated over a period of a week. The Core Management Engine (CME) Processor State General Purpose Engine (PGPE) heartbeat was being lost because of a priority timing issue, causing the error.

A problem was fixed for an IPL failure related to boot errors in the On-Chip Controller (OCC) and Core Management Engine (CME) with SRC BC8A090F and an "RC_STOP_GPE_INIT_TIMEOUT" error logged.  This is an intermittent IPL failure. The system can be recovered by retrying the IPL. This fix reduces the frequency of the error but it may still rarely occur. If it does occur, the retry of the IPL will be successful to recover the system.

OP940.30

PNOR OP9-v2.4.1-4.37 with BMC op940.30

10/14/2021

Impact: Data        Severity:  HIPER

New features and functions

Support added to increase the host console log size to 256K

Linux-aspeed updated to version v5.4.119.

System firmware changes that affect all systems

HIPER/Non-Pervasive:  A problem was fixed for a potential problem with I/O adapters that could result in undetected data corruption.

A problem was fixed to prevent a system crash from a denial of service attack when starting a guest.  With the fix, if a corrupt guest request occurs with an invalid XIVE VP structure, only the new guest will abort, not the whole system.

A problem was fixed for XSCOM read/write errors being written into OPAL log as PR_ERROR instead of PR_INFO as the severity of these logs should be informational.

A problem was fixed for excessive and continuous Self Boot Engine (SBE) timer requests that prevent the SBE from processing normal operation messages. With this fix, the number of continuous timer updates is limited and there is a wait for the timer expiry interrupt to restart sending timer requests.

A problem was fixed for Self Boot Engine (SBE) timer requests expiring immediately when re-trying canceled timer requests. This can cause delays in SBE operations and excessive messaging to the SBE.

A problem was fixed for not checking for busy timer messages to the Self Boot Engine (SBE) that can result in lost timer messages, causing the need to resend messages to the SBE.

A problem was fixed for a system going to Safe mode during the IPL with SRCs logged of BC8A2AD3 and BC702616 when the On-Chip Controller (OCC) did not reach active state. The fans will be running at full speed in this mode. The error is triggered by a storm of updates to the Power Management Control Register (PMCR) during the IPL that cause a timeout between the PGPE engine and OCC for a heartbeat message. The problem is intermittent, so a possible recovery is to re-IPL the system.

A problem was fixed for systems missing GPU sensor temperature data from the sensor log data. This can occur if the BMC has been running for more than six months. The problem can be circumvented by resetting the BMC.

A problem was fixed to change the factory reset to not clear the secure boot file.

A security problem was fixed for the BMC for a flaw in ICMP packets in the Linux kernel that may allow an attacker to quickly scan open UDP ports. This flaw allows an off-path remote attacker to effectively bypass source port UDP randomization. This vulnerability is CVE-2020-25705.

A security problem was fixed for the BMC that allows remote attackers to make observations that help to obtain sensitive information about the internal state of the network Random Number Generator (RNG).  This vulnerability is CVE-2020-16166.

A security problem was fixed for the BMC OpenSSL X509 certificates to prevent a possible denial of service (DOS) attack with a NULL pointer de-reference and BMC crash. This vulnerability is CVE-2020-1971.

A security problem was fixed for the BMC OpenSSL TLS to prevent a maliciously crafted Client Hello message from causing a possible denial of service (DOS) attack with a NULL pointer de-reference and BMC crash. This vulnerability is CVE-2021-3449.

A security problem was fixed for the BMC HTTPS web server that could allow an unauthenticated user to obtain sensitive information. This Common Vulnerabilities and Exposures issue number is CVE 2021-38960.

OP940.22

PNOR OP9-v2.4.1-4.31 with BMC op940.22

09/30/2021

Impact: Security       Severity:   HIPER

System firmware changes that affect all systems

HIPER/Pervasive:  A security problem was fixed that allowed a network attacker to use specially crafted IPMI messages to bypass authentication and gain full control of the system.  This is security vulnerability CVE-2021-39296.

OP940.21

PNOR OP9-v2.4.1-4.31 with BMC op940.20

03/31/2021

Impact: Availability        Severity:  HIPER

System firmware changes that affect all systems

HIPER/Pervasive:   A problem was fixed for a processor core checkstop with SRC BC70E540 logged with Signature Description " ex(n0p1c4) (NCUFIR[11]) NCU no response to snooped TLBIE".  This problem is intermittent and random but occurs with relatively high frequency for certain workloads.  The trigger for the failure is one core of a fused core pair going into a stopped state while the other core of the pair continues running.

OP940.20

PNOR OP9-v2.4.1-4.24 with BMC op940.20

11/20/2020

Impact: Availability       Severity: SPE

New features and functions

Host firmware support for anti-rollback protection. This feature implements firmware rollback protection as described in NIST SP 800-147B “BIOS Protection Guidelines for Servers”. All host firmware is signed with a "secure version".  The secure boot verification process will block any firmware secure version that is less than the "minimum secure version" that is maintained in the processor hardware.  During the system power on the host firmware will update the "minimum secure version" to match the currently running firmware.

WARNING:  Once this service pack is installed the system will not boot successfully with any service packs prior to this one.  The system can be recovered by reloading this or any newer service pack

System firmware changes that affect all systems

A problem was fixed to be able to detect a failed PFET sensing circuit in a core at runtime, and prevent a system fail with an incomplete state when a core fails to wake up. The failed core is detected on the subsequent IPL. With the fix, a core is called out with the PFET failure with SRC BC13090F and hardware description "CME detected malfunctioning of PFET headers." to isolate the error better with a correct callout.

A problem was fixed for mixing memory DIMMs with different timings (different vendors) under the same memory controller that fail with a SRC BC20E504 error and DIMMs deconfigured. This is a "MCBIST_BRODCAST_OUT_OF_SYNC" error.  The loss of memory DIMMs can result in an IPL failure.  This problem can happen if the memory DIMMs have a certain level of timing differences.  If the timings are not compatible, the failure will occur on the IPL during the memory training. To circumvent this problem, each memory controller should have only memory DIMMs from same vendor plugged.

A problem was fixed for PCIe resources under a deconfigured PCIe Host Bridge (PHB) being shown on the OS host as available resources when they should be shown as deconfigured. While this fix can be applied concurrently, a re-IPL of the system is needed to correct the state of the PCIe resources if a PHB had already been deconfigured.

A problem was fixed for a possible host hang if the BMC becomes unresponsive, such as during a BMC reboot.  With the fix, the uart will drop console write message data to the BMC if the BMC does not respond to avoid a host hang.

A problem was fixed for a re-IPL after a system crash failing with a processor core in an incorrect state.  This can happen if the core is in a STOP state to save power at the time of the initial crash and then this stopped core cannot be initialized correctly on the re-IPL.

A problem was fixed for non-optimal On-Chip Controller (OCC) processor frequency adjustments when system power limits or user power caps are exceeded. When a workload causes power limits or caps to be exceeded, there can be large frequency swings for the processors and a processor chip can get stuck at minimum frequency. With the fix, the OCC now waits for new power readings when changing the processor frequency and uses a master power capping frequency to keep all processors at the same frequency.  As a work-around for this problem, do not set a power cap or run a workload that would exceed the system power limit.

A problem was fixed for an intermittent mismatch in the number of fan sensors versus the actual number of system fans. The problem can be circumvented by a re-IPL of the system.  Fan sensors may be missing or there could be extra fan sensors when the problem happens.

A problem was fixed for the "Firmware Revision" showing as "0.00" when displayed by the "mc info" IPMI command. The rest commands show the "Firmware Revision" correctly.

A problem was fixed for a checkstop not generating any eSels or guards of hardware.  Without the fix, there is no data to debug the system failure when this happens.

A problem was fixed for BMC recovery from errors that cause directory paths to be deleted on the BMC.  This problem has pervasive Common.Error.InternalFailure errors during BMC boots because of missing directories.

OP940.10

PNOR OP9-v2.4-4.49 with BMC op940.10

05/05/2020

Impact: Availability       Severity: SPE

New features and functions

Support was added for a 16 Gb DDR4 single rank DIMM with feature code #8A2643. This memory DIMM can be used in place of  #EM61, a dual rank 16Gb DIMM, for faster memory performance.

Support was added to allow adapters with Radeon cards to be the petitboot console. This specifically includes the graphics adapter with IBM feature code #EC51. The default is for the system to use the embedded AST graphics card for the console, so to change to the Radeon card one of the following two steps must be taken by the user to manually change the Skiroot kernel command line and append commands that instruct which console to use. On the Petitboot shell do one of the following:

1) Use a command to disable the default AST graphics card by blacklisting the driver for the embedded AST graphics card:

# nvram -p ibm,skiboot --update-config  bootargs="`cat /proc/cmdline` module_blacklist=ast"

2) Or use a command to map the Framebuffer console to the Radeon driver (considering AST graphics is still active):

# nvram -p ibm,skiboot --update-config bootargs="`cat /proc/cmdline` fbcon=map:N" where "N" corresponds to the mapping function of currently active framebuffer devices (see 'Documentation/fb/fbcon.rst' in the Linux Kernel source for more information).

The user can configure the console output back to the embedded AST graphics card by removing any 'module_blacklist=ast' or 'fbcon=map:1' entries from the  'bootargs' nvram setting, or by resetting it to the default:

# nvram -p ibm,skiboot --update-config bootargs=""

System firmware changes that affect all systems

A problem was fixed for security vulnerability CVE-2020-14156.  With this vulnerability, a weakness in file permissions allows a remote authenticated attacker to recover the cleartext password of IPMI users.  Without the fix, the problem can be avoided by changing the file permissions to restrict access to the root user only using the following procedure:
1) Login to the BMC command shell as the root user.  For example, use command "ssh root@$BMCIP" where "$BMCIP" is the IP address of the BMC.
2) Then change the file permissions with the following command: "chmod 600 /etc/ipmi_pass".

A problem was fixed for failing fans not being called out in an error log and they do not get marked non-functional in inventory.  Without the fix, this problem can be circumvented by manually starting the fan monitor service on the BMC twenty seconds after a power on has been requested, using the following BMC command:

"systemctl start phosphor-fan-monitor@0.service".

A problem was fixed for the REST AccountService/Accounts APIs to provide more feedback to guide the user on why a local account creation or password change has failed.  If the password is not good enough, the service will indicate that result with the exact reason provided from the Linux PAM modules.  Creating a new local user account and updating the password of an existing local user account both require a password to be supplied.  Without the fix, when this password is not acceptable (complexity rules, etc), the request fails with a suggestion to use a better password, but a diagnostic message from PAM is not provided to help the user.

A problem was fixed for handling On-Chip Controller (OCC) UE errors so that the OCC can reset without terminating the system.  Without the fix, the system will checkstop on any OCC UE that requires a reset of the OCC.  A re-ipl of the system will recover from this error.

A problem was fixed for a system HMI that can occur if a GPU Address Translation Request (ATR) exceeds the time-out period.  With the fix, the time-out period was extended to allow the worst-case extra time for memory accesses if the data was not in the cache.

A security vulnerability was fixed to block access to unknown I2C devices on the TPM bus.  With the fix, bus traffic is allowed only to the I2C bus address explicitly defined in the firmware.  If an unknown address is used by the host Linux, an I2C timeout error is returned to the OS without forwarding the request to the TPM bus.  

A problem was fixed for an OPAL boot hang if OPAL keeps aborting early in the boot path. With the fix, an OPAL Terminate Immediate (TI) is used to signal the BMC about the OPAL termination.  This allows the BMC to quiesce the system for debug if the reboot tries have exceeded the reboot counter limit in the AutoReboot policy.

A problem was fixed for a failure in a GPU that can occur if the GPU is reset at run time.

A problem was fixed for allocation of memory on the LPC bus that is not a power of 2 being truncated.  Using memory space that is not allocated can cause unexpected system failure as un-owned memory becomes corrupted.  This problem is rare because it can only occur for requests of memory that are near the maximum memory size for an allocation.

A problem was fixed for a spurious error message when Linux is first booted:  " LPC[000]: Got SYNC no-response error. Error address reg: 0xd0010082".  Without the fix, this error message can be ignored as it just indicates a boot progress indicator was not implemented by the BMC.

A problem was fixed for the I2C signal used to reset OpenCapi adapters so that when the slot is powered off, the reset pin will not be kept high by the I2C controller any more.  This eliminates applying a voltage to part of the FPGA, which can send the FPGA into a bad state or even damage the card.

A problem was fixed for a GPU being reset from an error state remaining fenced and not usable.

A problem was fixed for not reporting OpenCapi slot reset failures.  Although resets were attempted to recover from errors, any resets that failed on adapters were not reported to the OS. This issue could result in hangs in operations directed at broken adapters.  With the fix, a message is logged for the broken hardware indicating the PHY state could not be reset to working and the OS does not attempt to use it.

System firmware changes that affect certain systems

For systems with deconfigured cores, a rare problem was fixed for an incorrect voltage/frequency setting for the processors during heavy workloads with high ambient temperature.  This error could impact power usage, expected performance, or system availability if a processor fault occurs.  This problem can be avoided by not using CPU frequency scaling when there are cores deconfigured in the system.  Instead,  use the "powersave" governor to run the CPUs at the minimum frequency or the "userspace" governor to run the CPUs at a user-specified fixed frequency.

OP940.01

PNOR OP9-v2.4-4.35 with BMC op940.01

01/08/2020

Impact:  Security    Severity:  SPE

New features and functions:

Improved BMC password policy.   For the BMC, the root password must be set on first use for newly-manufactured systems and after a factory reset of the system. This policy change helps to enforce the BMC is not left in a state with a well-known password. In firmware level OP940.01 and later, the root password is expired by default. The BMC administrator must change the password before you can access the functions of the BMC. If you are upgrading from a previous OpenBMC firmware level, you do not have to change the password.

The administrator can change from the default password to a new password using the interfaces described below:

1.  Web GUI

2. To change your expired password from a network interface, you can use Redfish APIs.

3. To change your expired password from the OpenBMC tool command, you can use the openbmctool set_password subcommand to change your password.

Examples of using these interfaces for expired password recovery can be found in the following IBM Knowledge Center article:  https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_setpassword.htm#p9eih_openbmc_setpassword__section_o14_qfj_yjb

Support was added for the following GRUB2 enhancements for petitboot:

1) 'source' command is now supported.

2) UUID and label are now supported in the 'search' command.

System firmware changes that affect all systems

A problem was fixed for an intermittent IPL failure with SRC B181E540 logged with fault signature " ex(n2p1c0) (L2FIR[13]) NCU Powerbus data timeout".  No FRU is called out.  The error may be ignored and the re-IPL is successful.  The error occurs very infrequently.  This is the second iteration of the fix that has been released.  Expedient routing of the Powerbus interrupts did not occur in all cases in the prior fix, so the timeout problem was still occurring.

A problem was fixed for RHEL OpenShift boot failures caused by dependencies on unsupported features in petitboot GRUB2.  Functionality in the GRUB2 parser was extended to allow RHEL OpenShift to boot correctly.  Certain Red Hat cloud image installations were failing without the fix.

A problem was fixed for an LDAP Bind failure on the first configuration attempt from the BMC GUI.  This problem applies to systems that have had a factory reset or to systems that are newly shipped from manufacturing when using LDAP for the first time from the BMC GUI. The BMC GUI is not setting the LDAP Bind password.  The Bind password is needed for LDAP Authentication with the LDAP server.  Without the fix, the work around to this problem is to use openbmctool to configure LDAP.

An example of using the openbmctool command for LDAP configuration creation:

openbmctool  -U root  -P "0penBmc"  -H <BMC IP address>  ldap enable -a "<LDAP server URI>" -B "<bind DN of the LDAP server>" -b "<base DN of the LDAP server>" -p "<bind password of the LDAP server>" -S sub -t OpenLDAP

Attempting login...

{

  "data": null,

  "message": "200 OK",

  "status": "ok"

}

User root has been logged out

OP940.00

PNOR OP9-v2.4-4.31 with BMC op940.00

11/22/2019

Impact:  New    Severity:  New

New features and functions:

Support for the Bittware 250-SoC 2x100Gb PCIe x16 OpenCAPI FPGA Adapter with Feature Code #EC4W.

Support was added for using DD2.2 and DD2.3 version P9 processors in the same system.

Support was added for Redfish APIs on the BMC.  OpenBMC-based systems can be managed by using the DMTF Redfish APIs.

   Redfish is a REST API used for platform management and is standardized by the Distributed Management Task Force, Inc. (http://www.dmtf.org/standards/redfish).  For more information, see the IBM Knowledge Center at the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_redfish.htm

Support was added for SSL certificate upload and generation.  For more information, see the IBM Knowledge Center at the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_ssl.htm

Support was added for IPMI version 2.0 on the BMC. IPMI network access can be disabled.  The command to disable IPMI network access is: "ipmitool lan set 1 access off".  You can use in-band IPMI if you want to re-enable IPMI network access: "ipmitool lan set 1 access on".

Support was added for Lightweight Directory Access Protocol (LDAP) on the BMC.  For more information on using LDAP,  see the  IBM Knowledge Center at the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_ldap.htm

Support was added for creating multiple local user accounts on the BMC.  For more information on managing user accounts to add or remove new users, modify user settings, manage user account policy settings, and view privilege role descriptions,  see the IBM Knowledge Center at the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_usermanage.htm

Support was added for KVM (Keyboard, Video and Mouse) consoles  on the BMC .  For more information on how to use  the remote KVM console.,  see the  IBM Knowledge Center at the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_kvm.htm

Support was added for virtual media devices on the BMC .  For more information on how to use virtual media device to start a session,  see the  IBM Knowledge Center at the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_virtualmedia.htm

Support was added for remote logging in the BMC gui.

Support was added for boot options in the BMC gui.

Support was added for SNMP traps and alerts on the BMC.  For more information on SNMP settings on the BMC,  see the  IBM Knowledge Center at the following link:

https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_snmpsettings.htm

New thermal modes added to support  PCIe4 adapters with feature codes #EC62, #EC64, #EC6E, and #EC6G  for both air and water cooled systems.  Please refer to IBM  Knowledge Center for selection of the appropriate thermal mode for the PCIe4  adapter, system models, and cable types:

https://www.ibm.com/support/knowledgecenter/en/POWER9/p9ei3/p9ei3_thermal_mode.htm.  

This article  provides the guidance for the thermal mode to select and how to use the openbmctool to select it.  Be aware that the thermal modes needed to supply additional cooling are not the system default and they have to be manually selected.  Once the new thermal mode is selected at  with the system powered off or on,  it is persistent until either changed by the user or a factory reset of the system  is done.  v1.14  of the openbmctool is the minimum version necessary to select the thermal mode and the recommended version is v1.17. There is no functional difference between these versions for thermal modes but error messages were updated in  v.1.17 to reflect that with OP940  release level systems and later do not have to be powered on to set the thermal mode as was required in earlier releases.  The thermal modes that can be selected include "DEFAULT", "CUSTOM", "HEAVY_IO", and "MAX_BASE_FAN_FLOOR".  The system comes pre-selected with "DEFAULT" which may not provide enough cooling for some adapters and configurations.

Note:  If your system has one of the affected PCIe4 adapters and you are using optical cables for that adapter, a thermal mode other than "DEFAULT" must be selected with openbmctool to get the proper cooling for the system.

Date

Description

07/06/2022

OP940.40 release

10/14/2021

OP940.30 release

09/30/2021

OP940.22 release

03/31/2021

OP940.21 release

11/20/2020

OP940.20 release

05/05/2020

OP940.10 release

05/05/2020

Added one more entry to the fix list for OP940.01

01/08/2020

OP940.01 release (with addition of HMC minimum level section)

11/22/2019

New for AC922 LC servers for the OP940.00 release