IBM Microcode Update Readme File   2 Port 10GbE RoCE SFP+ PCIe 3.0 Adapter (FC:  EC37, EC38 & EL3X, EL53)   2 Port 10GbE RoCE SR PCIe 3.0 Adapter (FC:  EC2M, EC2N & EL40, EL54)       ******* PLEASE READ THIS ENTIRE NOTICE *********  DATE: November 22, 2019 Table of Contents   1.0 Microcode and Document Revision History 2.0 General information 3.0 Installation time 4.0 Machine's Affected 5.0 Linux Requirements 6.0 AIX Requirements 7.0 Determine the Current Microcode Level for AIX 8.0 Downloading the RPM Format File to the Target Server for AIX 9.0 Discovery Tool Microcode CD-ROM creation and download instructions 10.0 Verifying microcode before download and notes for AIX 11.0 Microcode Download Procedure for AIX 11.1 Setting up for Microcode download 11.2 Downloading Microcode to the Adapter 11.3 Verify and Re-configure adapters     1.0     Microcode and Document Revision History   Firmware Level Description   000200425001 / 2.42.5001 Impact: Usability  Severity: ATT - This level of firmware supports the Firmware Secure Boot feature of Power systems. - Fixed an issue which pre-vented the Port Receive counter from being cleared after port counter reset. - Fixed an issue that caused the server to hang and result in NMI when running ?mlxfwtop ?d mt4103_pci_cr0? while restarting the driver in parallel (from a different thread). In this case, the down-stream bridge over the device reported completion timeout error. - Fixed an issue while closing the HCA, where RX packet caused bad access to resources that did not exist, and consequently caused the QPCGW or theirisc to get stuck. - Fixed an issue where the masterSMLID and the LID was either 0 or 0xFFFF when the port was neither active nor armed. - Fixed an issue that pre-vented the link to go up after reboot. - Fixed a rare issue that cause the PCIe configuration cycle that arrived during the time ofsw_reset to generate 2 completions. - Fixed an issue where the two interfaces reported the same MAC address when bonding configuration was used The Firmware Levels Below Are No Longer Supported By IBM It is best practices to update to the latest FW level not only for IBM support of these products, but for optimal performance and to ensure thatall of the required HW/FW fixes are installed. Please Update To The Latest Level At Your Earliest Convenience    2.40.5030 / 000200405030 Impact: Usability  Severity: ATT  - Fixed an issue where outbound completions were blocked during driver restart  - Fixed an issue which caused firmware not to send link down even during close_port command  - Fixed auto sense issue failure of detecting right protocol  - Fixed incorrect of Port Receive and Transmit  Extended counters  - Fixed issue of firmware mishandling packets during TX congestion  - Fixed issue with TX request taking 10ms during link training 2.36.5000 / 000200365000  - Fixed length calculated of UDP packets, which resulted incorrect UPD length causing miscalculation of CRC. - Fixed completion error issue when ECN was enabled. - Fixed a race condition that caused duplicated read request. - Fixed an issue causing packets to get stuck when driver send pause frame withdmac equal to device's mac address. 2.34.5000 / 000200345000 New Release 2.33.5000 / 000200335000 Original Release   Document Revision History Description V5.0 ? 11/22/2019 Version 2.42.5001 release V4.0 ? 2/3/2017 Version 2.40.5030 release V3.0 ? 9/19/2016 Version 2.36.5000 release V2.0 ? 8/5/2015 New Release V1.0 ? 1/26/2015 Original Release         2.0     General information   This Readme file is intended to give directions on how to update the microcode found on the2 Port 10GbE RoCE SFP+ PCIe 3.0 Adapter and 2 Port 10GbE RoCE SR PCIe 3.0 Adapter   Non-Concurrent Download The microcode installation does NOT support concurrent download. The entX devices can be used during and after the download, but update will not go into effect until either the device is reloaded usingrmdev/cfgmgr or a reboot is performed.   NOTE: It is recommended that the installation be scheduled during a maintenance window or during non-peak production periods.   3.0     Installation time   Approximately 20 minutes.   4.0 Machine's Affected   Feature Code EC37 and EC2M: 8284-22A, 9009-22A, 9080-M9S, 9080-MHE, 9080-MME, 9119-MHE, 9119-MME, 9223-22H   Feature Code EL3X and EL40: 5148-21L, 5148-22L, 8247-21L, 8247-22L, 9008-22L   Feature Code EL53 and EL54: 8247-21L, 8247-22L, 8247-42L, 9008-22L   Feature Code EC38 and EC2N: 8284-22A, 8286-41A, 8286-42A, 8408-44E, 8408-E8E, 9009-22A, 9009-41A, 9009-42A, 9040-MR9, 9080-M9S, 9080-MHE, 9080-MME, 9119-MHE, 9119-MME, 9223-22H, 9223-42H     5.0 Linux Requirements Error! Bookmark not defined.   For Linux operating systems, use the following procedure:   1. Find the PCI slot the ConnectX-3 adapter is plugged into. Issue the following command:   lspci | grep "ConnectX-3 Pro"   For example: # lspci | grep "ConnectX-3 Pro" 0000:50:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]   This tells us that the adapter is in the PCIe slot 0000:50:00.0 which will be needed in the following steps.    2. Start MFT tools by running ?mst start?   #mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices   If the system does not have the mst command available, please install the Mellanox Firmware Tools (MFT) available here: http://www.mellanox.com/page/management_tools       3. Verify the current Firmware level by running ?flint -d -q?.  Replace with the ID found in the lscpi command from step 2.   # flint -d 0000:50:00.0 q Image type:      FS2 FW Version:      2.40.5030 FW Release Date: 4.1.2017 Rom Info:        type=FCODE version=0.0.170 devid=4103 proto=ETH Device ID:       4103 Description:     Node             Port1            Port2            Sys image GUIDs:           ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff MACs:                                 f45214751ed0     f45214751ed1 VSD: PSID:            IBM1200111023   # flint -d 0000:50:00.0 q Image type:      FS2 FW Version:      2.40.5030 FW Release Date: 4.1.2017 Rom Info:        type=FCODE version=0.0.170 devid=4103 proto=ETH Device ID:       4103 Description:     Node             Port1            Port2            Sys image GUIDs:           ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff MACs:                                 f45214751ed0     f45214751ed1 VSD: PSID:            IBM1200113023       This command reveals the current version of the FW; which is 2.40.5030 in this particular case.  If the version is less than 2.42.5001, please update. Please note the value for PSID.  This will determine which image is needed in later steps.   4. Download the firmware RPM file to the destination machine.   Use this method to download the new microcode to a Linux system: NOTE: The instructions that follow are specific Linux commands. Linux commands are CASE (lower and upper) SENSITIVE, andmust be entered exactly as shown, including filenames. * Transfer the RPM format file to the /tmp directory (using ?Save as....?). You will see the filename for the RPM file.  * Install rpm on your Linux system by typing: rpm -ivh /tmp /000200425001.Linux.rpm   The microcode package will install the firmware images in the /lib/firmware directory. If a message is displayed saying the "package is already installed", you will need to uninstall the listed rpm package. On the command line type: rpm -e where is the name of the package that was returned in the message. Return to Step 2 and attempt to install the file again.   * The microcode file will be installed to the /lib/firmware directory. The file name is?b31507101410e704.000200425001 and b31507101410eb04.000200425001   5. Verify the contents of the image before flashing: ls -l /lib/firmware/ b31507101410??04.000200425001to verify file size: ˇ       b31507101410e704.000200425001 = 908972 ˇ       b31507101410eb04.000200425001= 908968   sum /lib/firmware/ b31507101410??04.000200425001 to verify Checksum: ˇ       b31507101410e704.000200425001= 38788 ˇ       b31507101410eb04.000200425001= 48331       6. Update the FW with "flint -d -i burn"   NOTE: The PSID value from step 3 will determine which file to use for firmware update. * IBM1200111023 = filename b31507101410e704.000200425001 * IBM1200113023 = filename b31507101410eb04.000200425001     # flint -d 0000:50:00.0 -i b31507101410e704.000200425001 burn       Current FW version on flash:  2.36.5000     New FW version:               2.40.5030   Burning FW image without signatures - OK Restoring signature                  - OK   If you receive the following error message, it means the wrong file was selected for update, and you need to choose the other filename. -E- PSID mismatch. The PSID on flash (IBM1200111023) differs from the PSID in the given image (IBM1200113023).     7. At this point we have burned the new FW on to the Mellanox adapter. However, this FW will not take effect till the system is rebooted. Reloading the drivers is not enough; it requires the system to be rebooted.     6.0 AIX Requirements   Basic card function is supported on: ˇ       AIX Version 6.1 with the 6100-09 Technology Level and service pack 5 and APAR IV68443 or later        ˇ       AIX Version 6.1 with the 6100-08 Technology Level and service Pack 7 or later (planned availability 9/30/2015) ˇ       AIX Version 7.1 with the 7100-03 Technology Level and Service Pack 5 and APAR IV68444 or later ˇ       AIX Version 7.1 with the 7100-02 Technology Level and Service Pack 7  or later (planned availability 9/30/2015)          If you are using another release of AIX, ensure that the adapter is supported on that release before you install the adapter. Contact service and support for assistance.   7.0 Determine the Current Microcode Level for AIX   Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the Adapter's VPD. A.   List the RoCE adapters installed in the system by typing:   lsdev | grep RoCE B.   To check the current microcode level for the adapter or controller enter the following command: lsmcode -cd entX Where "X" is the instance of the adapter. The command will produce output similar to: # lsmcode -cd ent4 The current microcode level for ent4 is 000200425001   If the ROM Level is less than 000200425001 you should update the microcode. 8.0 Downloading the RPM Format File to the Target Server for AIX Use this method to download to an AIX system:   NOTE: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, andmust be entered exactly as shown, including filenames.   A.   Make two directories on your AIX system to receive the RPM format file. Enter:      "mkdir /tmp/microcode" and then create this directory Enter:      "mkdir /tmp/microcode/RPM"   B.   Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as ...").  Change to that directory, "cd /tmp/microcode/RPM". You'll see the filename for the RPM file.        "rpm -ihv --ignoreos 000200425001.aix.rpm" C.   For AIX:  The microcode files will be added to /etc/microcode/.  D.   Two microcodes file will be copied to "/etc/microcode".  The file size and checksum of the microcode image will be verified in Section 10.0. File Names: b31507101410e704.000200425001 b31507101410eb04.000200425001   NOTE:  - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".  - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location.   9.0 Discovery Tool Microcode CD-ROM creation and download instructions To obtain information how to burn a CD-ROM and run the Discovery Tool for an AIX or Linux System please go to:   http://www-304.ibm.com/webapp/set2/firmware/lgjsn?mode=10&page=cdrom.html   A.   After running the Discovery Tool successfully the "/tmp/microcode/RPM" directory was created and your rpm files are copied from the CD-ROM. B.   Change to that directory, "cd /tmp/microcode/RPM". C.   Unpack the file by executing the instructions below: Enter the command: "rpm -ihv --ignoreos 000200425001.aix.rpm" D.   Two microcodes file will be copied to "/etc/microcode".  The file size and checksum of the microcode image will be verified in Section 10.0. File Names: b31507101410e704.000200425001 b31507101410eb04.000200425001     NOTE: - "/etc/microcode" is a symbolic link to "/usr/lib/microcode". - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location. - For customers using the AIX Diagnostics CD, please refer to the IBM System Hardware information Center for instructions.         10.0 Verifying microcode before download and notes for AIX Please verify the file size and checksum of the raw microcode files matches what is listed below. ls -l /etc/microcode/b31507101410??04.000200425001 to verify file size: ˇ       b31507101410e704.000200425001= 908972 ˇ       b31507101410eb04.000200425001= 908968   sum /etc/microcode/b31507101410??04.000200425001 to verify Checksum: ˇ       b31507101410e704.000200425001= 38788 ˇ       b31507101410eb04.000200425001= 48331       11.0 Microcode Download Procedure for AIX 11.1 Setting up for Microcode download   A.   Stop all applications that use this interface/adapter. B.   Remove the interface/IP address from the all ports identified in section 7.0 for the adapters that will be upgraded. a.    Before detaching the interface, record the IP address and any other pertinent information that was configured on the Adapter.  This information may be needed if the microcode update overwrites this section on the Adapter. C.   If the interfaces are members of an SEA, the SEA devices must be moved to a defined state. a.    "rmdev -l enX" - where "X" is the interface number for the Shared Ethernet Adapter. b.    "rmdev -l entX" - where "X" is the interface number for the Shared Ethernet Adapter. D.   If the interfaces are members of an EtherChannel, the EtherChannel device must be moved to a define state. a.    "rmdev -l enX" - where "X? is the interface number for the EtherChannel adapter. b.    "rmdev -l entX" - where "X" is the interface number for the EtherChannel adapter. E.    For every port associated with the adapter, the enX interfaces must be changed to a defined state. a.    "rmdev -l enX" - where "X" is the interface number for the adapter port. b.    This command will be run 2 times, once for each port on the adapter.                11.2 Downloading Microcode to the Adapter A.   At the command line type "diag" B.   Select the "Task Selection" from diagnostics menu. C.   Select "Microcode Tasks? then select ?Download Microcode" from the menu. D.   Select all of the entX RoCE Converged Network Adapters that need to be updated from the list of devices by using the arrow keys to highlight the entry and pressing "Enter" to mark it.  Press "F7" or "ESC+7" when you are done marking all the adapters you want to flash. E.    If a source selection menu is displayed, Select "/etc/microcode". F.    A dialogue box may be displayed on screen.  It will state that the current microcode level on the adapter is not in the /etc/microcode directory.  This is acceptable because the adapter will reject any incorrect code.  Press "Enter" to continue. G.   Select 000200425001 level and press "Enter" to flash the adapter. H.   The following message will appear on the screen when download is completed: "Microcode download complete successfully.  The current microcode level for the device is ...  Please run diagnostics on the adapter to ensure that it is functioning properly."  I.    If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated. J.    Exit diagnostics. 11.3 Re-configure and Verify adapters A.   Run "cfgmgr" to reconfigure the adapters that were moved to defined before the update.  B.   Verify the code level is 000200425001 by typing "lsmcode -cd entX" for each adapter updated, where "X" is the instance of the RoCE adapter.