IBM HMC on Power Firmware
Applies to: 7063-CR2
This document provides information about the installation of Licensed Machine or Licensed Internal Code, which is sometimes referred to generically as microcode or firmware.
1.2 Fix level Information on IBM Power HMC Components and Operating systems |
This package provides firmware for IBM HMC on Power 7063-CR2 only with minimum HMC version level of HMC V9 R2 M950 or later.
The firmware level in this package is:
•OP940.70 / PNOR OP9_v2.5_4.127 and OpenBMC op940.hmc-29
Note: A rare timing event during boot of the 7063-CR2 HMC can result in different conditions depending on the version of BMC FW currently functional on the HMC.Please refer here for workarounds and symptoms: https://www.ibm.com/support/pages/node/6520826
This section specifies the "Minimum ipmitool Code Level" required by the System Firmware for managing the system. OpenPOWER requires ipmitool level v1.8.15 or later to execute correctly on the OP910 and later firmware. It must be capable of establishing a IPMI v2 session with the ipmi support on the BMC.
Verify your ipmitool level on your Linux workstation using the following command:
bash-4.1$ ipmitool -V
ipmitool version 1.8.15
If you are need to update or add impitool to your Linux workstation , you can compile ipmitools (current level 1.8.15) for Linux as follows from Sourceforge:
1.1.1 Download impitool tar from http://sourceforge.net/projects/ipmitool/ to your linux system
1.1.2 Extract tarball on Linux system
1.1.3 cd to top-level directory
1.1.4 ./configure
1.1.5 make
1.1.6 ipmitool will be under src/ipmitool
You may also get the ipmitool package directly from your workstation Linux packages.
1.2 Fix level Information on IBM Power HMC Components and Operating systems
For specific fix level information on key components of IBM HMC model 7063-CR2 and the HMC software, please refer to the documentation in the IBM Knowledge Center.
https://www.ibm.com/docs/en/power9/0000-FUL?topic=code-upgrading-your-hmc-software
Downgrading firmware from any given release level to an earlier release level is not recommended.
If you feel that it is necessary to downgrade the firmware on your system to an earlier release level, please contact your next level of support.
Concurrent Firmware Updates not available for Power HMC.
Concurrent system firmware update is not supported on the Power HMC.
Use the following examples as a reference to determine whether your installation will be concurrent or disruptive.
For the Power HMC, the installation of system firmware is always disruptive.
The BMC and PNOR image tar files are used to update the primary side of the PNOR and the primary side of the BMC only, leaving the golden sides unchanged.
Filename | Size | Checksum |
obmc-mowgli-op940.hmc-29.ubi.mtd.tar
| 22712320 | 9c9bc59d6603acaadfc58a292a1caf73 |
mowgli-IBM-OP9_v2.5_4.127_prod.pnor.squashfs.tar
| 33341440 | 8b75e2f3078b8a6b80f6ccd8a61d17c3 |
Note: The Checksum can be found by running the Linux/Unix/AIX md5sum command against the Hardware Platform Management (hpm) file (all 32 characters of the checksum are listed), ie: md5sum <filename>
After a successful update to this firmware level, the PNOR components and BMC should be at the following levels.
To display the PNOR level, use the following BMC command: "cat /var/lib/phosphor-software-manager/pnor/ro/VERSION | grep -A 12 IBM"
And the BMC command line command "cat" can be used to display the BMC level: "cat /etc/os-release".
Note: FRU information for the PNOR level does not show the updated levels via the fru command until the system has been booted once at the updated level.
PNOR firmware level:
IBM-mowgli-OP9_v2.5_4.127-prod
op-build-v2.3-rc2-1464-g2e9eeab
buildroot-2021.02.10-4-g2b611ab
skiboot-v7.0
hostboot-f9065b1-pda915e3
occ-9047e57
linux-5.10.108-openpower1-p3c734f2
petitboot-v1.13-pd33bcc0
machine-xml-59f3878
hostboot-binaries-hw101520a.op940
capp-ucode-p9-dd2-v4
sbe-47abe2a-pdfd26c0
hcode-hw062722a.op940
BMC firmware level:
id: openbmc-phosphor
name: Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)
version: op940.hmc-29
version_id: op940.hmc-29-0-g11d2cfce07
pretty_name: Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) op940.hmc-29
build_id: op940.hmc-29
BMC Primary side version:
op940.hmc-29
Alternatively access the OpenBMC UI by directly pointing a browser to the IP or hostname of the BMC:
Example: https://<ip or hostname>
In the “Server overview” panel, locate the “FIRMWARE VERSION” fields under “Server information”(PNOR), and “BMC information (BMC).
OP940 | |
PNOR OP9_v2.5_4.127 with OpenBMC op940.hmc-29 OP940.70
11/17/2023
| Impact: Availability Severity: SPE
New features and functions
Fixes a problem where even though there is no DNS configuration on BMC, systemd-resolved service still uses the precompiled list of google DNS for queries. Updated to Linux kernel version 5.10.108 to address security issues found in previous released level 5.4.107. A problem was fixed in Petitboot discovery for crashing on a bad URL path for a DHCP server, leaving the system with only a shell running. A problem was fixed for a Broadcom network adapter being shown as link down in Petitboot and not configurable, but in the OS the adapter works correctly. The error message logged in Petitboot is "bnx2x: [bnx2x_init_firmware:13480(enP52p1s0f0)]Can't load firmware file bnx2x/bnx2x-e2-7.13.15.0.fw". The Linux 5.10 kernel requires a newer version of Broadcom bnx2x firmware. With the fix, the Broadcom bnx2x firmware version is updated to 7.13.15.0. A problem was fixed for a failed correctable error recovery for a DIMM that causes a flood of SRC BC81E580 error logs and also can prevent dynamic memory deallocation from occurring for a hard memory error. This is a very rare problem caused by an unexpected number of correctable error symbols for the DIMM in the per-symbol counter registers.
|
PNOR OP9_v2.5_4.125 with OpenBMC op940.hmc-27 OP940.60
10/12/2023
| Impact: Security Severity: SPE The Common Vulnerabilities and Exposures issue numbers for these problems are CVE-2022-4304 and CVE-2022-4450.
A fix was made to an algorithm used in processing data from the power supply.
|
PNOR OP9_v2.5_4.125 with OpenBMC op940.hmc-25 OP940.50
12/07/2022
|
|
PNOR OP9_v2.5_4.125 with OpenBMC op940.hmc-21 OP940.40
05/20/2022
| New features and functions
A problem was fixed for an issue with the clock date not rolling over correctly at the end of the day when a BMC reset occurred at the same time. For example, if the date/time was set forward to 5/21/2032 23:59:00 and a BMC reset was performed, the date afterward may show 6/5/2032 instead of the expected 5/22/2032 00:00:10.
|
PNOR OP9_v2.5_4.124 with OpenBMC op940.hmc-16 OP940.16
12/07/2021
| New features and functions
Support added to increase the host console log size to 256K.
System firmware changes that affect all systems
HIPER/Pervasive: A problem was fixed for error recovery for an intermittent BMC hang that causes a flash side switch on reset of the BMC with loss of profile settings (including network settings). The intermittent BMC hang can occur while the host is rebooting as part of a host firmware update. With the fix, the recovery for a BMC hang is a BMC reset that is done without switching the BMC flash side, allowing the BMC to reboot cleanly.
A problem was fixed for a system going to Safe mode during the IPL with SRCs logged of BC8A2AD3 and BC702616 when the On-Chip Controller (OCC) did not reach the active state. The fans will be running at full speed in this mode. The error is triggered by a storm of updates to the Power Management Control Register (PMCR) during the IPL that cause a timeout between the PGPE engine and OCC for a heartbeat message. The problem is intermittent, so a possible recovery is to re-IPL the system.
A problem was fixed to change the factory reset to not clear the secure boot file.
A security problem was fixed for a flaw in ICMP packets in the Linux kernel that may allow an attacker to quickly scan open UDP ports. This flaw allows an off-path remote attacker to effectively bypass source port UDP randomization. This vulnerability is CVE-2020-25705.
A security problem was fixed that allows remote attackers to make observations that help to obtain sensitive information about the internal state of the network Random Number Generator (RNG). This vulnerability is CVE-2020-16166.
A security problem was fixed for OpenSSL X509 certificates to prevent a possible denial of service (DOS) attack with a NULL pointer de-reference and BMC crash. This vulnerability is CVE-2020-1971.
A security problem was fixed for OpenSSL TLS to prevent a maliciously crafted Client Hello message from causing a possible denial of service (DOS) attack with a NULL pointer de-reference and BMC crash. This vulnerability is CVE-2021-3449.
|
PNOR OP9_v2.5_4.123 with OpenBMC op940.hmc-11.1 OP940.11
09/30/2021
| Impact: Security Severity: HIPER
New features and functions
Support removed for multiple KVM sessions. The maximum number of KVM sessions is 1.
Support added for "Model" to the asset interface
Support was changed to limit the maximum PCI Host Bridge (PHB) speed to PCIe3 for the first PHB on the device tree.
System firmware changes that affect all systems
HIPER/Pervasive: A security problem was fixed that allowed a network attacker to use specially crafted IPMI messages to bypass authentication and gain full control of the system. This is security vulnerability CVE-2021-39296.
HIPER/Non-Pervasive: A problem was fixed for a potential problem with I/O adapters that could result in undetected data corruption.
A number of important fixes for bounds checks and data validation were made for the secure boot (verified boot) feature that checks that OS kernels are valid before allowing them to boot. These fixes include corrections for memory leaks, over reads of data, improved error checking, more thorough input data validation, and removal of redundant print error messages for certificate chain errors.
A problem was fixed to prevent EEH recovery on an I/O adapter from causing an escalated PCI Host Bridge (PHB) reset on some adapters where it is not needed.
A problem was fixed during the OPAL boot to only remove the nest In-Memory Collection (IMC) counter devices from the device tree if the microcode state is not running or paused. Without the fix, all of the IMC devices are removed from the device tree if a pause of the microcode fails.
A problem was fixed to prevent In-Memory Collection (IMC) counter devices from getting removed from the device tree on larger systems if there is a delay for the microcode download from the On-Chip Controller (OCC).
A problem was fixed for "CPU Hardlock UP", memory re-free, and kernel crash errors during the boot when the BT (IPMI message link from OPAL to the BMC) timer is preempted by the BMC, causing the following error log: [29006114.163785853,3] BT: seq 0x81 netfn 0x0a cmd 0x23: Timeout sending message [29006114.288029290,3] BT: seq 0x81 netfn 0x0b cmd 0x23: Timeout sending message [29006114.288917798,3] IPMI: Incorrect netfn 0x0b in response
A problem was fixed for Self Boot Engine (SBE) timer requests expiring immediately when re-trying canceled timer requests. This can cause delays in SBE operations and excessive messaging to the SBE.
A problem was fixed for excessive and continuous Self Boot Engine (SBE) timer requests that prevent the SBE from processing normal operation messages. With this fix, the number of continuous timer updates is limited and there is a wait for the timer expiry interrupt to restart sending timer requests.
A problem was fixed for not checking for busy timer messages to the Self Boot Engine (SBE) that can result in lost timer messages, causing the need to resend messages to the SBE.
A problem was fixed for XSCOM read/write errors being written into OPAL log as PR_ERROR instead of PR_INFO as the severity of these logs should be informational.
A problem was fixed to prevent a system crash from a denial of service attack when starting a guest. With the fix, if a corrupt guest request occurs with an invalid XIVE VP structure, only the new guest will abort, not the whole system..
A problem was fixed to prevent a hang on a failed fast reboot request. With the fix, a failed fast reboot will cause the system to do a normal reboot after logging an error message.
A problem was fixed for a failed fast reboot after guarding a failed processor.
|
PNOR OP9_v2.5_4.115 with OpenBMC op940.hmc-5 OP940.00
05/21/2021 | Impact: New Severity: New
New features and functions:
Support for the IBM 7063 Model CR2 HMC appliance that has a HMC minimum level requirement of V9R2M950.
Support of Secure and Trusted boot for the firmware.
Host firmware support for anti-rollback protection. This feature implements firmware rollback protection as described in NIST SP 800-147B “BIOS Protection Guidelines for Servers”. All host firmware is signed with a "secure version". The secure boot verification process will block any firmware secure version that is less than the "minimum secure version" that is maintained in the processor hardware. During the system power on the host firmware will update the "minimum secure version" to match the currently running firmware. |
4.0 Operating System Information
OS levels supported by the 7063-CR2 server:
- HMC V9 R2 M950 or later
The HMC stack runs on an embedded Linux distribution. The HMC on Power version V9 R2 M950 or later is supported on the 7063-CR2.
Use the following steps in the below link to navigate the HMC GUI to determine the HMC level:
https://www.ibm.com/docs/en/power9/9009-42A?topic=code-determining-your-hmc-machine-version-release
From the HMC command line use “lshmc -V”.
https://www.ibm.com/support/knowledgecenter/8247-21L/p8hai/p8hai_viewcodelevel_enh.htm
See Section 3.1 “Firmware Information and Description”.
Follow the instructions on Fix Central. You must read and agree to the license agreement to obtain the firmware packages.
The updating and upgrading of system firmware depends on several factors, such as the current firmware that is installed, and what operating systems is running on the system.
These scenarios and the associated installation instructions are comprehensively outlined in the firmware section of Fix Central, found at the following website:
http://www.ibm.com/support/fixcentral/
Any hardware failures should be resolved before proceeding with the firmware updates to help insure the system will not be running degraded after the updates.
The 7063-CR2 firmware is made up of two separate components
•.BMC - image for the Baseboard Management Controller
•.Server image - PNOR
A boot priority system, allows for the selection of a previous image to be used. This is useful when there is a need to revert to a prior image.
On the OpenBMC UI, the image file that is listed at the top (for each stack, BMC and PNOR), the image with the highest boot priority, is used the next time that the device is booted. You can change the boot order for the image file by clicking the arrow icons.
Image State Definitions
•.Functional: The running image on the device.
•.Active: The image is available to boot from, but is not currently the running image. If the image is the top image in the relevant table, it becomes the functional image the next time the device is rebooted.
•.Activating: The image is in the process of being activated and becomes either Active or Failed.
•.Failed: The image failed to activate.
•.Ready: The image is ready to be activated.
•.Invalid: This image is an invalid image and cannot be activated.
The OpenBMC UI or the openbmctool commands can be used to both view or update the firmware images.
The HMC must be shut down prior to updating firmware.
It is recommended that both BMC and PNOR be updated prior to restarting the HMC.
1. Access Server configuration -> Firmware
2. Scroll down on the page to locate the section to upload the firmware image.
3. Click on Choose a file and browse your local filesystem for the location of the BMC image.
4. Click on Upload firmware
The “Upload in progress...” message is displayed
A confirmation message is displayed when the upload is complete.
5. The new image is now in a Ready state. Click Activate, under the Action column to activate it.
6. The user is presented with a confirmation panel with the options to "ACTIVATE FIRMWARE FILE WITHOUT REBOOTING BMC" and "ACTIVATE FIRMWARE FILE AND AUTOMATICALLY REBOOT BMC".
7. Select ACTIVATE FIRMWARE FILE AND AUTOMATICALLY REBOOT BMC and click Continue
8. The image state will show Activating
9. The image will then transition to Functional state (the running image on the device)
10. If successful, a message is displayed on the upper right of the session, showing Success! BMC is rebooting.
11 After the BMC comes back up and the UI is refreshed, the new image is now listed on the top line (first in boot priority) and the previous image is now listed second. The original second image was removed.
12. This concludes the BMC part of the update.
The next firmware image to update is the PNOR (also known as Server) image.
1. Scroll down on the page to locate the section to upload the firmware image.
2. Click on Choose a file and browse your local filesystem for the location of the PNOR image.
3. Click on Upload firmware
The “Upload in progress... “message is displayed
A confirmation message is displayed when the upload is complete:
4. The new image is now in a Ready state. Click Activate, under the Action column to activate it.
5. The user is presented with a confirmation panel “Confirm server firmware file activation”.with the options to "ACTIVATE FIRMWARE FILE WITHOUT BOOTING SERVER" and "ACTIVATE FIRMWARE FILE AND AUTOMATICALLY BOOT SERVER".
6. Select ACTIVATE FIRMWARE FILE AND AUTOMATICALLY BOOT SERVER and click Continue
7. The image state will show Activating
8. The image will then transition to Functional state (the running image on the device)
9. After a UI refresh, the new image is now listed on the top line (first in boot priority) and the previous image is now listed second. Any original second image is removed (there wasn't one in this example).
10. This concludes the PNOR part of the update.
11. The system is automatically started following PNOR update.
The process of updating firmware on the OpenBMC Power HMC is documented below.
The sequence of events that must happen is the following:
•Power off the Host
•Update and Activate BMC
•Update and Activate PNOR
•Reboot the BMC (applies new BMC image)
•Power on the Host (applies new PNOR image)
The OpenBMC firmware updates (BMC and PNOR) for the Power HMC can be managed via the command line with the openbmctool.
The openbmctool is obtained using the IBM Support Portal.
1.Go to the IBM Support Portal.
2.In the search field, enter your machine type and model. Then click the correct product support entry for your system.
3.From the Downloads list, click the openbmctool for your machine type and model.
4.Follow the instructions to install and run the openbmctool. You will need to provide the file locations of the BMC firmware image tar and PNOR firmware image tar that must be downloaded from Fix Central for the update level needed.
Information on the openbmctool and the firmware update process can be found in the IBM Knowledge Center:
https://www.ibm.com/support/knowledgecenter/POWER9/p9ei8/p9ei8_update_firmware_openbmctool.htm
The service processor, or baseboard management controller (BMC), provides a hypervisor and operating system-independent layer that uses the robust error detection and self-healing functions that are built into the POWER processor and memory buffer modules. OpenPOWER application layer (OPAL) is the system firmware in the stack of POWER processor-based Linux-only servers.
The service processor, or baseboard management controller (BMC), is the primary control for autonomous sensor monitoring and event logging features on the Power HMC.
The BMC supports the Intelligent Platform Management Interface (IPMI) for system monitoring and management. The BMC monitors the operation of the firmware during the boot process and also monitors the OPAL hypervisor for termination.
Various risks that are associated with the Intelligent Platform Management Interface (IPMI) have been identified and documented in the information technology (IT) security community.
Possible risks includes the following three common vulnerabilities and exposures (CVEs):
1) CVE-2013-4037:
The Remote Authenticated Key-Exchange Protocol (RAKP), which is specified by the IPMI standard for authentication, has flaws. Although the system does not allow the use of null passwords, a hacker might reverse engineer the RAKP transactions to determine a password. The authentication process for IPMI requires the management controller to send a hash of the requested password of the user to the client before the client authenticates. This process is a key part of the IPMI specification. The password hash can be broken by using an offline brute force or dictionary attack.
2) CVE-2013-4031:
IBM Power Systems and OpenPower Systems are preconfigured with one IPMI user account, which has the same default login name and password on all affected systems. If a malicious user gains access to the IPMI interface by using this preconfigured account, the user can power off or on, or restart the host server, and create or change user accounts possibly preventing legitimate users from accessing the system. On OpenPower Systems, the default IPMI user name is root. Additionally, if a user fails to change the default user name and password on each of the systems that is deployed, the user has the same login information for each of those systems.
3) CVE-2013-4786:
The IPMI 2.0 specification supports RMCP+ Authenticated Key-Exchange Protocol (RAKP) authentication, which allows remote attackers to obtain password hashes and conduct offline password guessing attacks by obtaining the hash-based message authentication code (HMAC) from a RAKP message 2 response from a BMC.
If a user is not managing a server by using the IPMI, one can configure the system to disallow IPMI network access from the user accounts. This task can be accomplished by using the IPMItool utility or a similar utility for managing and configuring the IPMI management controllers. Use the following IPMItool command to disable the network access for an IPMI user:
ipmitool channel setaccess 1 #user_slot# privilege=15
For more information on the IPMI security vulnerabilities and configuration options and best practices to minimize the risks of this interface, go to the IBM Knowledge Center at the following URL:
https://www.ibm.com/support/knowledgecenter/POWER9/p9eih/p9eih_openbmc_security.htm
The OpenPOWER Abstraction Layer (OPAL) provides hardware abstraction and run time services to the running host Operating System.
For the 7063-CR2 server, only the OPAL bare-metal installs of the Hardware Management Console are supported.
Find out more about OPAL skiboot here:
https://github.com/open-power/skiboot
Petitboot is a kexec based bootloader used by IBM POWER9 systems for doing the bare-metal installs.
After the POWER system powers on, the petitboot bootloader scans local boot devices and network interfaces to find boot options that are available to the system. Petitboot returns a list of boot options that are available to the system. If you are using a static IP or if you did not provide boot arguments in your network boot server, you must provide the details to petitboot. You can configure petitboot to find your boot with the following instructions:
https://www.ibm.com/support/knowledgecenter/linuxonibm/liabw/liabppetitbootadvanced.html
You can edit petitboot configuration options, change the amount of time before Petitboot automatically boots, etc. with these instructions:
https://www.ibm.com/support/knowledgecenter/linuxonibm/liabw/liabppetitbootconfig.html
After you select to boot the ISO media for the Linux distribution of your choice, the installer wizard for that Linux distribution walks you through the steps to set up disk options, your root password, time zones, and so on.
You can read more about the petitboot bootloader program here:
https://www.kernel.org/pub/linux/kernel/people/geoff/petitboot/petitboot.html
Date | Description |
11/17/2023 | OP940.70 release |
10/12/2023 | OP940.60 release |
12/07/2022 | OP940.50 release |
05/20/2022 | OP940.40 release |
12/08/2021 | Added warning for possible BMC boot error in Section 1.0 |
12/07/2021 | OP940.16 release |
09/30/2021 | OP940.11 release |
05/21/2021 | New for Power HMC 7063-CR2 for the OP940.00 release |