IBM Microcode Update Readme File

Firmware Level	Description
28.40.1000 / 002800401000	Impact: Usability Severity: ATT · This fw is required to support MC3208411-T 1G RJ45 SFP transceiver. · Fixed an issue with Selective-Repeat configuration which occasionally caused retransmission to wait for timeout instead of out-of-sequence NACK. · Fixed an issue that kept the adapter cards' quad ports UP when using breakout cables / QSFP-split-4. Now when a 4 alignment loss is noticed, the link in 25G/lane Ethernet is dropped.
The Firmware Levels Below Are No Longer Supported By IBM Once They Have Been Removed From The Microcode Download Website. It is best practices to update to the latest FW level not only for IBM support of these products, but for optimal performance and to ensure that all of the required HW/FW fixes are installed. Once new FW has been released to the field, we will provide a 6 month grace period for customers to update these products to the currently supported FW level. Please Update To The Latest Level At Your Earliest Convenience
28.39.1002 / 002800391002	Impact: Usability Severity: ATT · Fixed an issue that led to packet drops on lossless fabric due to an Rx buffer overflow. · Fixed a code mismatch in the process of handling the cause to the link being down when the remote faults were received. · Fixed a wrong parameter in the cable info MAD that resulted in unnecessary messages in the log. · Fixed an issue that prevent MSI Interrupts from being advertised correctly, resulting in the wrong MSI being sent. · "Get Temperature" OEM command now always returns a unified temperature. · Changed the bar configuration algorithm so that the last update to the bar address will be the one that takes affect when the host configures the same bar address for two different PFs. · Changed the bar configuration algorithm so that the last update to the bar address will be the one that takes affect when the host configures the same bar address for two different PFs. · Fixed a HW bug that resulted in transaction loss that when cache replacement transaction occurs in parallel to code transcoding. · The eeprom module gets stuck in polling in 20% of the times after reset. To resolve the issue, a delay after config module to high power was added. · Fixed a linkup failure issue that occurred when connecting to a 25GbE transceiver by clearing the PSI Aging before trying to open Tx power. · The "Bad Signal Integrity" message seen after power cycle can be safely ignored. The user should monitor BER number. · Added a locking mechanism to protect the firmware from a race condition between insertion and deletion of the same rule in parallel. Such behavior occasionally resulted in firmware accessing a memory that has already been released, thus causing IOMMU / translation error. · Fixed a race condition that led to a firmware assert upon driver removal, or when changing the ETH flow control scheme in case of a stress of larger than MTU ingress packets.
28.38.1500 / 002800381500	Impact: NEW Severity: NEW Original Release for EN26 and EN24 adapter

Document Revision History

Description

V1.0 – 03/20/2024

Original Release

V2.0 – 07/22/2024

Updated instructions for new fw 28.40.1000 / 002800401000 release

Also including bug fix history for 28.39.1002/ 002800391002

2.0 General information

This Readme file is intended to give directions on how to update the microcode found on the PCIe4 x16 4-port 25/10/1 GbE RoCE SFP28 adapter.

1. Non-Concurrent Download (Linux Only)
The microcode installation does NOT support concurrent download in Linux. The device can be used during and after the download, but update will not go into effect until a reboot is performed.

2. It is recommended that the installation be scheduled during a maintenance window or during non-peak production periods.

3. It is best practice to update to latest FW level not only for IBM support of these products, but for optimal performance and to ensure that all the required HW/FW fixes are installed.

4. Once new FW has been released to the field, we will provide a 6 month grace period for customers to update these products to the currently supported FW level.

5. Adapter in PowerVM SRIOV shared mode
- This adapter firmware release notes applies to adapter configured in dedicated mode.
- When adapter is transitioned to SRIOV mode, the system firmware updates the adapter firmware, which may differ from the firmware used in dedicated mode.
- When adapter is moved back to dedicated mode, user will need to update the adapter firmware to the level mentioned in this release notes.

6. Release Notes for adapter firmware in PowerVM SRIOV shared mode.
Please visit fix central (http://www.ibm.com/support/fixcentral/ ) and review the release notes pertaining to your system MTM and installed system firmware.

7. For more information about adapters running in PowerVM SRIOV shared mode visit: SRIOV FAQs and vNIC FAQs

8. To avoid hitting the below mentioned AIX driver issues with the new adapter microcode, please either install that APARs corresponding to your AIX/VIOS version mentioned below,

OR update your AIX/VIOS to latest level available on fix central .

3.0 Installation time

Approximately 20 minutes.

4.0 Machine's Affected

Feature Code: EN26

· 9080-HEX

· 9043-MRX

· 9105-22A/22B/41B/42A

· 9786-22H/42H

Feature Code: EN24

· 9080-HEX

5.0 Linux Requirements Error! Bookmark not defined.

For Linux operating systems, use the following procedure:

1. Find the PCI slot the 4 port ConnectX-7 adapter is plugged into. Issue the following command:

lspci -m -nn -d :1021|grep 0712

For example:

# lspci -m -nn -d :1021|grep 0712

0001:01:00.0 "Ethernet controller [0200]" "Mellanox Technologies [15b3]" "MT2910 Family [ConnectX-7] [1021]" "IBM [1014]" "Device [0712]"

0001:01:00.1 "Ethernet controller [0200]" "Mellanox Technologies [15b3]" "MT2910 Family [ConnectX-7] [1021]" "IBM [1014]" "Device [0712]"

0001:01:00.2 "Ethernet controller [0200]" "Mellanox Technologies [15b3]" "MT2910 Family [ConnectX-7] [1021]" "IBM [1014]" "Device [0712]"

0001:01:00.3 "Ethernet controller [0200]" "Mellanox Technologies [15b3]" "MT2910 Family [ConnectX-7] [1021]" "IBM [1014]" "Device [0712]"

Note:

a. This tells us that the adapter is in the PCIe slot 0001:01:00.0 which will be needed in next steps.

b. Each port is listed as its own PCIe ID

0001:01:00.0 – Port 0 of the adapter

0001:01:00.1 – Port 1 of the adapter

0001:01:00.2 – Port 2 of the adapter

0001:01:00.3 – Port 3 of the adapter.

c. Either of the port PCIe IDs can be used to perform the microcode install.

2. Start MFT tools by running “mst start”

#mst start

Starting MST (Mellanox Software Tools) driver set

Loading MST PCI module - Success

Loading MST PCI configuration module - Success

Create devices

If the system does not have the mst command available, please install the Mellanox Firmware Tools (MFT) available here: http://www.mellanox.com/page/management_tools

3. Verify the current Firmware level by running “flint -d <pci bus ID> -q”. Replace <pci bus ID> with the ID found in the lscpi command from step 2.

# flint -d 0122:60:00.0 q

Image type: FS4

FW Version: 28.40.1000

FW Release Date: 4.2.2024

Product Version: 28.40.1000

Description: UID GuidsNumber

Base GUID: 946dae03005e2882 8

Base MAC: 946dae5e2882 8

Image VSD: N/A

Device VSD: N/A

PSID: IBM0000000061

Security Attributes: secure-fw

This command reveals the current version of the FW; which is 28.40.1000 in this particular case. If the version is less than 28.40.1000, please update.

4. Download the firmware RPM file to the destination machine.

Use this method to download the new microcode to a Linux system:

NOTE: The instructions that follow are specific Linux commands. Linux commands are CASE (lower and upper) SENSITIVE, and must be entered exactly as shown, including filenames.

Transfer the RPM format file to the /tmp directory (using “Save as....”). You will see the filename for the RPM file.
Install rpm on your Linux system by typing:

rpm -ivh /tmp/b315211014101207.002800401000.linux.rpm

The microcode package will install the firmware images in the /lib/firmware directory. If a message is displayed saying the "package <package_name> is already installed", you will need to uninstall the listed rpm package. On the command line type:

rpm -e <package_name>

where <package_name> is the name of the package that was returned in the message. Return to Step 2 and attempt to install the file again.

The microcode file will be installed to the /lib/firmware directory. The file name is：

5. Verify the contents of the image before flashing:

ls -l /lib/firmware/b315211014101207.002800401000 to verify file size:

· b315211014101207.002800401000 = 33554432

sum /lib/firmware/ b315211014101207.002800401000 to verify Checksum:

· b315211014101207.002800401000 = 27220

6. Update the FW with "flint -d <pci bus id> -i <image filename> burn"

# flint -d 0001:01:00.0 -i b315211014101207.002800401000 burn

Current FW version on flash: 28.38.1500

New FW version: 28.40.1000

Burning FW image without signatures - OK

Restoring signature - OK

7. At this point we have burned the new FW on to the Mellanox adapter. However, this FW will not take effect till the system is rebooted.

Reloading the drivers is not enough; it requires the system linux partition to be rebooted.

6.0 AIX Requirements

Adapter is supported on:

· AIX 7.3 with Technology Level 2 and Service Pack 1 and above

· AIX 7.3 with Technology Level 1 and Service Pack 3 and above

· AIX 7.2 with Technology Level 5 and Service Pack 7 and above

· VIOS 4.1.0.10 and above

· VIOS 3.1.4.31 and above

If you are using another release of AIX, ensure that the adapter is supported on that release before you install the adapter. Contact service and support for assistance.

7.0 Determine the Current Microcode Level for AIX

Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the Adapter's VPD.

A. List all PCIe4 x16 4-port 25/10/1 GbE RoCE SFP28 adapter installed in the system by typing:
lsdev |grep b315211014101207

B. To check the current microcode level for the adapter or controller enter the following command:

lsmcode -cd entX

Where "X" is the instance of the adapter. The command will produce output similar to:

# lsmcode -cd ent1

The current microcode level for ent1 is 002800401000

If the ROM Level is less than 002800401000 you should update the microcode.

8.0 Downloading the RPM Format File to the Target Server for AIX

Use this method to download to an AIX system:

NOTE: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, and must be entered exactly as shown, including filenames.

A.    Make two directories on your AIX system to receive the RPM format file.
Enter:      "mkdir /tmp/microcode"
and then create this directory
Enter:      "mkdir /tmp/microcode/RPM"

B. Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as ..."). Change to that directory, "cd /tmp/microcode/RPM".
You'll see the filename for the RPM file.
"rpm -ihv --ignoreos b315211014101207.002800401000.aix.rpm"

C. For AIX: The microcode files will be added to /etc/microcode/.

D. Microcode file will be copied to "/etc/microcode". The file size and checksum of the microcode image will be verified in Section 10.0.
File Names:
b315211014101207.002800401000

NOTE:
- "/etc/microcode" is a symbolic link to "/usr/lib/microcode".
- If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location.

9.0 Discovery Tool Microcode CD-ROM creation and download instructions

To obtain information how to burn a CD-ROM and run the Discovery Tool for an AIX or Linux System please go to:

https://www.ibm.com/support/pages/node/6491185

A. After running the Discovery Tool successfully the "/tmp/microcode/RPM" directory was created and your rpm files are copied from the CD-ROM.

B. Change to that directory, "cd /tmp/microcode/RPM".

C. Unpack the file by executing the instructions below:
Enter the command:
"rpm -ihv --ignoreos b315211014101207.002800401000.aix.rpm "

D. Microcode file will be copied to "/etc/microcode". The file size and checksum of the microcode image will be verified in Section 10.0.
File Names:
b315211014101207.002800401000

NOTE:

- "/etc/microcode" is a symbolic link to "/usr/lib/microcode".

- If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location.

- For customers using the AIX Diagnostics CD, please refer to the IBM System Hardware information Center for instructions.

10.0 Verifying microcode before download and notes for AIX

Please verify the file size and checksum of the raw microcode files matches what is listed below.

ls -l /etc/microcode/b315211014101207.002800401000 to verify file size:

· b315211014101207.002800401000= 33554432

sum /etc/microcode/ b315211014101207.002800401000 to verify Checksum:

· b315211014101207.002800401000= 27220

11.0 Microcode Download Procedure for AIX

11.1 Setting up for Microcode download

A. Stop all applications that use this interface/adapter.

B. Remove the interface/IP address from the all ports identified in section 7.0 for the adapters that will be upgraded.

a. Before detaching the interface, record the IP address and any other pertinent information that was configured on the Adapter. This information may be needed if the microcode update overwrites this section on the Adapter.

C. If the interfaces are members of an SEA, the SEA devices must be moved to a defined state.

a. "rmdev -l enX" - where "X" is the interface number for the Shared Ethernet Adapter.

b. "rmdev -l entX" - where "X" is the interface number for the Shared Ethernet Adapter.

D. If the interfaces are members of an EtherChannel, the EtherChannel device must be moved to a define state.

a. "rmdev -l enX" - where "X” is the interface number for the EtherChannel adapter.

b. "rmdev -l entX" - where "X" is the interface number for the EtherChannel adapter.

E. For every port associated with the adapter, the enX interfaces must be changed to a defined state.

a. "rmdev -l enX" - where "X" is the interface number for the adapter port.

b. This command will be run 2 times, once for each port on the adapter.

11.2 Downloading Microcode to the Adapter

A. At the command line type "diag"

B. Select the "Task Selection" from diagnostics menu.

C. Select "Microcode Tasks” then select “Download Microcode" from the menu.

D. Select all of the entX PCIe4 x16 4-port 25/10/1 GbE RoCE SFP28 adapters that need to be updated from the list of devices by using the arrow keys to highlight the entry and pressing "Enter" to mark it. Press "F7" or "ESC+7" when you are done marking all the adapters you want to flash.

E. If a source selection menu is displayed, Select "/etc/microcode".

F. A dialogue box may be displayed on screen. It will state that the current microcode level on the adapter is not in the /etc/microcode directory. This is acceptable because the adapter will reject any incorrect code. Press "Enter" to continue.

G. Select 002800401000 level and press "Enter" to flash the adapter.

H. The following message will appear on the screen when download is completed: "Microcode download complete successfully. The current microcode level for the device is ... Please run diagnostics on the adapter to ensure that it is functioning properly."

I. If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated.

J. Exit diagnostics.

11.3 Re-configure and Verify adapters

A. Run "cfgmgr" to reconfigure the adapters that were moved to defined before the update.

B. Verify the code level is 002800401000 by typing "lsmcode -cd entX" for each adapter updated, where "X" is the instance of the PCIe4 x16 4-port 25/10/1 GbE RoCE SFP28 adapter.

12.0 Microcode Download Procedure for Systems Without AIX Installed:

Use this procedure if the system has no OS installed or the installed OS does not support microcode download for this device.

1. Boot the system via a Standalone Diagnostic's CD

a. To obtain a Standalone Diagnostic CD-ROM please go to the following web page and follow the instructions: https://www.ibm.com/support/pages/node/6474531

b. After booting to Standalone Diagnostics, the Diagnostic's CD can be removed.

2. The Diagnostic's CD should be replaced by a CD containing the unpacked microcode image.

The CD containing the microcode images should be prepared prior. Create a CD-ROM with the firmware b315211014101207.002800401000 mentioned in section 9.0 and burn the CD with Joliet CD file system.

3. From the Define a System Console menu either type or select vt320.

4. Select the “Task Selection” from diagnostics menu then select “Download Microcode”.

5. Select resource PCIe4 4-port 25/10/1 GbE RoCE SFP28 Adapter x16 (b315211014101207) that microcode will be applied to and PRESS ENTER.

6. Press “F7” or “ESC+7” to coFirmware Release and Management Tracking Toolmmit.(The current microcode level of the resource you selected earlier will be displayed at the top of the screen).

7. Select Input Device (CD-ROM or DVD) then choose the Microcode level to download.

NOTE: A prompt will ask you to insert the microcode CD-ROM into the CD/DVD drive. Insert the CD-ROM which accompanies these instructions. If this level is already installed in the drive you've selected a message will let you know. A message also may come up to let you know that the CD_ROM does not have the previous level microcode file. This is true and is not required to complete the download. PRESS “F7” or “ESC+7” to commit.

8. You will receive a message on the screen. “Current Microcode is....” and “Download has completed successfully” (this may take a few minutes).

9. Return to the Tasks Selection menu and repeat this procedure for each adapter that requires this microcode.

10. Exit diagnostic and reboot system in normal mode.

Table of Contents

1.0 Microcode and Document Revision History