PCIe4 2-port
100GbE RoCE Adapter (FC: EC66, EC67)
******* PLEASE
READ THIS ENTIRE NOTICE *********
DATE: May 4, 2021
1.0 Microcode and Document Revision
History
7.0 Determine the Current Microcode
Level for AIX
8.0 Downloading the RPM Format File to the Target Server for AIX
9.0 Discovery Tool Microcode CD-ROM
creation and download instructions
10.0 Verifying
microcode before download and notes for AIX
11.0 Microcode
Download Procedure for AIX
11.1 Setting
up for Microcode download
11.2 Downloading Microcode to the
Adapter
11.3 Re-configure and Verify adapters
|
Firmware Level |
Description |
|
16.29.1017 /
001600291017 |
Impact: Availability Severity: ATT 1. Changed the default value of DCQCN's NP
parameter min_time_between_cnps to 4 on all devices
to support larger scalability of cluster. 2. Fixed an issue that prevented VXLAN packets
with svlan/cvlan tag from
being matched. 3. Fixed an issue that caused the eth_wqe_too_small counter to count ODP page faults. 4. Fixed an issue that resulted in low performance
after enabling the RoCE Accelerator capability. The
fix is available when all ports are set as Ethernet. 5. On rare cases, a fatal error related to errors
from the PCI transport layer might be reported during FLR. 6. Fixed the chassis manager calculation for
Multi-Host and Socket-Direct adapter cards to allow running NC-SI commands by
the chassis manager BMC. Now the chassis manager is counted as BMC with index
0, regardless of how many BMC there are. 7. Fixed an issue that caused the device to go to
dead IRISC as one of the firmware semaphores could not be released when a
speed change or port state change was triggered. |
|
The Firmware Levels Below Are No Longer Supported By IBM Once They Have Been Removed From The Microcode
Download Website. It is best practices to update to the latest FW level not only
for IBM support of these products, but for optimal performance and to ensure
that all of the required HW/FW fixes are installed. Once new FW has been
released to the field, we will provide a 6 month
grace period for customers to update these products to the currently
supported FW level. Please Update To The Latest Level At
Your Earliest Convenience |
|
|
16.24.8000 (for AIX
and Linux) |
Impact: NEW Severity: NEW Original Release for EC66 and EC67 adapter |
|
Document Revision History |
Description |
|
V1.0 – 08/17/2018 |
Original Release |
|
V1.1 – 08/30/2018 |
Added AIX APARS
section |
|
V1.2 – 04/25/2019 |
Updated instructions for new fw 16.24.8000 / 001600248000
release |
|
V1.3 – 05/04/2021 |
Updated instructions
for new fw 16.29.1017 / 001600291017 release |
This Readme file is intended to give directions on how to update the
microcode found on the PCIe4
2-port 100GbE RoCE Adapter
1. Non-Concurrent Download (Linux
Only)
The microcode installation does NOT support concurrent download in
Linux. The device can be used during and after the download, but update will
not go into effect until a reboot is performed.
2. It is recommended that the
installation be scheduled during a maintenance window or during non-peak
production periods.
3. It is best practice to update
to latest FW level not only for IBM support of these products, but for optimal
performance and to ensure that all the required HW/FW fixes are installed.
4. Once new FW has been released
to the field, we will provide a 6-month grace period for customers to update
these products to the currently supported FW level.
5. Adapter in PowerVM
SRIOV shared mode
- This adapter firmware
release notes applies to adapter configured in dedicated mode.
- When adapter is transitioned
to SRIOV mode, the system firmware updates the adapter firmware, which may
differ from the firmware used in dedicated mode.
- When adapter is moved back to
dedicated mode, user will need to update the adapter firmware to the level
mentioned in this release notes.
6. Release Notes for adapter
firmware in PowerVM SRIOV shared mode.
Please visit fix central
(http://www.ibm.com/support/fixcentral/) and review the release notes
pertaining to your system MTM and installed system firmware.
7. For more information about
adapters running in PowerVM
SRIOV shared mode visit: SRIOV
FAQs and vNIC FAQs
8. To avoid hitting the below
mentioned AIX driver issues with the new adapter microcode, please either
install that APARs corresponding to your AIX/VIOS version mentioned below OR
update your AIX/VIOS to latest level available on fix central.
a). Driver update to support
transition from 4MB to 8 MB microcode file size.
AIX Level APAR
7200-04 (72 TL4) = IJ14175
6100-09 (61 TL9) = IJ14204
7100-05 (71 TL5) =
IJ14334
7200-03 (72 TL3) =
IJ14355
7200-02 (72 TL2) =
IJ14640
7200-01 (72 TL1) =
IJ14691
7100-04 (71 TL4) =
IJ14807
Approximately 20 minutes.
Feature
Code: EC66
· 9040-MR9
· 9009-41A
· 9009-42A
· 9223-42H
Feature
Code: EC67
· 9080-M9S
· 9008-22L
· 9223-22H
For Linux operating systems, use the following
procedure:
1. Find the PCI slot the ConnectX-5 adapter is
plugged into. Issue the following command:
lspci –nn | grep “1019"
For example:
# lspci -nn |grep
1019
0001:01:00.0 Ethernet controller [0200]: Mellanox Technologies MT28800
Family [ConnectX-5 Ex] [15b3:1019]
0001:01:00.1 Ethernet controller [0200]: Mellanox Technologies MT28800
Family [ConnectX-5 Ex] [15b3:1019]
Note:
a. This tells us that the adapter is in the PCIe
slot 0001:01:00.0 which will be needed in next steps.
b.
Each port is listed as its own PCIe ID
0001:01:00.0 – Port 1 of
the adapter
0001:01:00.1 – Port 2 of the adapter.
c. Either of
the port PCIe IDs can be used to perform the microcode install.
2. Start MFT tools by running “mst
start”
#mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
If the system does not have the mst
command available, please install the Mellanox Firmware Tools (MFT) available
here: http://www.mellanox.com/page/management_tools
3. Verify the current Firmware level by running “flint -d <pci bus ID> -q”. Replace <pci bus ID> with the ID found in the lscpi command from step 2.
# flint -d 0001:01:00.0 q
Image type: FS4
FW Version:
16.29.1017
FW Release Date:
11.1.2021
Product Version:
16.29.1017
Description:
UID GuidsNumber
Base GUID:
506b4b0300350142 4
Base MAC:
506b4b350142 4
Image VSD: N/A
Device VSD: N/A
PSID:
IBM0000000020
Security Attributes: N/A
This command reveals the current
version of the FW; which is 16.29.1017
in this particular case. If the
version is less than 16.29.1017,
please update.
4. Download the firmware RPM file to the
destination machine.
Use this method to download the new
microcode to a Linux system:
NOTE: The instructions that follow are specific Linux commands.
Linux commands are CASE (lower and upper) SENSITIVE, and must be
entered exactly as shown, including filenames.
rpm
-ivh /tmp/b315191014103506.001600291017.Linux.rpm
The microcode package will install the firmware
images in the /lib/firmware directory. If a message is displayed saying the
"package <package_name> is already
installed", you will need to uninstall the listed rpm package. On the
command line type:
rpm
-e <package_name>
where <package_name>
is the name of the package that was returned in the message. Return to Step 2
and attempt to install the file again.
5. Verify the contents of the image before flashing:
ls
-l /lib/firmware/b315191014103506.001600291017 to verify file size:
·
b315191014103506.001600291017 = 16777216
sum
/lib/firmware/b315191014103506.001600291017 to verify Checksum:
·
b315191014103506.001600291017 = 02051
6. Update the FW with "flint -d <pci bus id> -i <image
filename> burn"
# flint -d 0001:01:00.0 -i b315191014103506.001600291017
burn
Current FW version on flash: 16.28.8000
New FW version: 16.29.1017
Burning FW image without
signatures - OK
Restoring signature - OK
7. At this point we have burned the new FW on to the
Mellanox adapter. However, this FW will not take effect till the system is
rebooted.
Reloading the drivers is not enough; it requires the system linux partition to be
rebooted.
Adapter is supported starting on
August 2018 fix pack of:
· AIX 7.2 with Technology Level 2 and Service Pack 2
and above
· AIX 7.1 with Technology Level 5 and Service Pack 2
and above
· VIOS 2.2.6.23 and above
If you are using another release of AIX, ensure that the adapter is
supported on that release before you install the adapter. Contact service and
support for assistance.
Prior to
installing the microcode, the following APARs are required to be installed.
Failure to do so may result in failure in aix
advanced diagnostics.
IJ08890
Before you install the microcode, it is important to determine the
microcode level of the Adapter installed in the target system. Use the
following instructions to read the ROM level stored in the Adapter's VPD.
A.
List all PCIe4 2-port 100GbE RoCE
Adapters installed in the system by typing:
lsdev |grep
b315191014103506
B.
To check the current microcode level for the
adapter or controller enter the following command:
lsmcode -cd entX
Where "X" is the instance of the adapter. The command will
produce output similar to:
# lsmcode
-cd ent1
The current
microcode level for ent1 is
001600291017
If the ROM Level is less than 001600291017 you should
update the microcode.
Use
this method to download to an AIX system:
NOTE: The instructions that follow are specific AIX
commands. AIX commands are CASE (lower and upper) SENSITIVE, and must
be entered exactly as shown, including filenames.
A.
Make two directories on your AIX system to receive
the RPM format file.
Enter: "mkdir /tmp/microcode"
and then create this directory
Enter: "mkdir /tmp/microcode/RPM"
B.
Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as
..."). Change to that directory, "cd /tmp/microcode/RPM".
You'll see the filename for the RPM file.
"rpm -ihv --ignoreos b315191014103506.001600291017.aix.rpm"
C.
For AIX: The microcode files will be added to
/etc/microcode/.
D.
Microcode file will be copied to "/etc/microcode".
The file size and checksum of the microcode image will be verified in
Section 10.0.
File Names:
b315191014103506.001600291017
NOTE:
- "/etc/microcode" is a symbolic link
to "/usr/lib/microcode".
- If permission does not allow the copy to the above stated directory or
file then the user will be prompted for a
new location.
To obtain information how to burn a CD-ROM and run the Discovery Tool
for an AIX or Linux System please go to:
http://www-304.ibm.com/webapp/set2/firmware/lgjsn?mode=10&page=cdrom.html
A.
After running the Discovery Tool successfully, the
"/tmp/microcode/RPM" directory was created,
and your rpm files are copied from the CD-ROM.
B.
Change to that directory, "cd /tmp/microcode/RPM".
C.
Unpack the file by executing the instructions
below:
Enter the command:
"rpm -ihv --ignoreos b315191014103506.001600291017.aix.rpm "
D.
Microcode file will be copied to "/etc/microcode".
The file size and checksum of the microcode image will be verified in
Section 10.0.
File Names:
b315191014103506.001600291017
NOTE:
- "/etc/microcode" is a symbolic
link to "/usr/lib/microcode".
- If permission does not allow the copy to the above stated directory or
file then the user will be prompted for a new location.
- For customers using the AIX Diagnostics CD, please refer to the IBM
System Hardware information Center for instructions.
Please verify the file size and checksum of the raw microcode files
matches what is listed below.
ls -l /etc/microcode/b315151014101f06.001400291017
to verify file size:
·
b315151014101f06.001400291017 = 16777216
sum
/etc/microcode/b315151014101f06.001400291017 to verify Checksum:
·
b315151014101f06.001400291017 = 02051
A.
Stop all applications that use this
interface/adapter.
B.
Remove the interface/IP address from the all ports
identified in section 7.0 for the adapters that will be upgraded.
a.
Before detaching the interface, record the IP
address and any other pertinent information that was configured on the
Adapter. This information may be needed
if the microcode update overwrites this section on the Adapter.
C.
If the interfaces are members of an SEA, the SEA
devices must be moved to a defined state.
a.
"rmdev -l enX" - where "X" is the interface number for
the Shared Ethernet Adapter.
b.
"rmdev -l entX" - where "X" is the interface number
for the Shared Ethernet Adapter.
D.
If the interfaces are members of an EtherChannel,
the EtherChannel device must be moved to a define state.
a.
"rmdev -l enX" - where "X” is the interface number for the
EtherChannel adapter.
b.
"rmdev -l entX" - where "X" is the interface number
for the EtherChannel adapter.
E.
For every port associated with the adapter, the enX interfaces must be changed to a defined state.
a.
"rmdev -l enX" - where "X" is the interface number for
the adapter port.
b.
This command will be run 2 times, once for each
port on the adapter.
A.
At the command line type "diag"
B.
Select the "Task Selection" from
diagnostics menu.
C.
Select "Microcode Tasks” then select “Download
Microcode" from the menu.
D.
Select all of the entX
PCIe4 2-port 100GbE RoCE Adapters that need to be
updated from the list of devices by using the arrow keys to highlight the entry
and pressing "Enter" to mark it. Press "F7" or
"ESC+7" when you are done marking all the adapters you want to flash.
E.
If a source selection menu is displayed, Select
"/etc/microcode".
F.
A dialogue box may be displayed on screen. It
will state that the current microcode level on the adapter is not in the /etc/microcode directory. This is acceptable because
the adapter will reject any incorrect code. Press "Enter" to
continue.
G.
Select 001600291017 level and press
"Enter" to flash the adapter.
H.
The following message will appear on the screen
when download is completed: "Microcode download complete
successfully. The current microcode level for the device is ...
Please run diagnostics on the adapter to ensure that it is functioning
properly."
I.
If you selected more than one adapter to update,
then steps 6-9 will repeat until all adapters are updated.
J.
Exit diagnostics.
A.
Run "cfgmgr" to
reconfigure the adapters that were moved to defined before the update.
B.
Verify the code level is 001600291017
by typing "lsmcode
-cd entX" for each adapter updated, where
"X" is the instance of the PCIe4 2-port 100GbE RoCE
Adapter.