IBM Microcode Update Readme File   PCIe3 2-port 100 GbE (NIC and RoCE) QSFP28 Adapter  (FC:  EC3M, EC3L)     ******* PLEASE READ THIS ENTIRE NOTICE *********  DATE: May 14th 2021 Table of Contents   1.0 Microcode and Document Revision History 2.0 General information 3.0 Installation time 4.0 Machine's Affected 5.0 Linux Requirements 6.0 AIX Requirements 7.0 Determine the Current Microcode Level for AIX 8.0 Downloading the RPM Format File to the Target Server for AIX 9.0 Discovery Tool Microcode CD-ROM creation and download instructions 10.0 Verifying microcode before download and notes for AIX 11.0 Microcode Download Procedure for AIX 11.1 Setting up for Microcode download 11.2 Downloading Microcode to the Adapter 11.3 Verify and Re-configure adapters   1.0     Microcode and Document Revision History   Firmware Level Description 12.29.1017/ 001200291017 Impact: Availability Severity: ATT 1. Updated Fcode  to fix an issue with TX buffer index not properly wrapping causing iSCSI boot to fail. 2. Increased PHY power consumption limit to 1.5w. 3. Fixed a rare case where the device hanged while running the sw reset flow under heavy stress and with many open resources. 5. Fixed an issue that cause the firmware to hang when an FLR occurred at the same time as the teardown. 6. Fixed an issue to mark PCIe TLP drop packets as ?uncorrectable, non-fatal? Please review comments about SRIOV and AIX APARs in General Information.       12.24.8000 / 001200248000 Impact: Availability Severity: ATT 1. Adds Secure Fcode support 2. Fixes Health Syndrome : HW_FATAL_ERR due to halt in health counter sometimes seen after a NIM install 3. Fixes Fcode probing issue sometimes seen after hotplug/dlpar and EEH scenarios    (The following are from vendor Bug Fixes History from fw release notes that could potentially affect POWER products) 4. Fixes an issue that showed the link as down even when the link was up due to missing asynchronous event. 5. Fixes an issue that occasionally caused the link to drop after running the ?reboot? command due to I2C being stuck. 6. Fixes a reare issue that sometimes caused the firmware to hang 7. Fixes an issue that wrongly reported the maximum temperature in a setup as the current temperature regardless of the actual temperature. The Firmware Levels Below Are No Longer Supported By IBM Once They Have Been Removed From The Microcode Download Website. It is best practices to update to the latest FW level not only for IBM support of these products, but for optimal performance and to ensure that all of the required HW/FW fixes are installed. Once new FW has been released to the field, we will provide a 6 month grace period for customers to update these products to the currently supported FW level. Please Update To The Latest Level At Your Earliest Convenience   12.20.1010 (for AIX and Linux) Impact: Availability  Severity: ATT    1. Fixed an issue that caused device to hang during reboot 2. Enabled firmware resync of internal clocks after getting out of standby mode. 3. Fixed an issue that caused TX to get stuck at link failover 4. Fixed PCIe TX glitch during link training to Gen3. 5. Fixed an issue that prevented physical counters from resetting. Note for AIX users: If microcode level on adapter is 0001200162500 and you plan to update the microcode to a newer level. Please make sure your AIX lpar is updated to latest released version or have APAR mentioned in the link installled prior to updating the microcode. Failure to do so will risk the adapter to be in defined mode. http://www-01.ibm.com/support/docview.wss?uid=isg1fixinfo166023     12.16.2500 / 001200162500 (AIX) 12.17.1502 / 001200171502 (Linux) 1. Original Release - Note: Do not attempt to install Linux firmware 12.17.1502 / 001200171502 on an AIX Operating System. Adapter will fail to configure in AIX with that version.   Document Revision History Description V1.0 ? 11/18/2016 Original Release V1.1 ? 11/5/2017 Updated instructions for 12.20.1010 release V1.1 ? 04/25/2019 Updated instructions for new fw 12.24.8000 / 001200248000 release     2.0     General information   This Readme file is intended to give directions on how to update the microcode found on the PCIe3 2 PORT 10 Gb NIC&ROCE SR/Cu ADAPTER   1.    Non-Concurrent Download (Linux Only) The microcode installation does NOT support concurrent download in Linux. The device can be used during and after the download, but update will not go into effect until a reboot is performed. 2.    It is recommended that the installation be scheduled during a maintenance window or during non-peak production periods. 3.     It is best practice to update to latest FW level not only for IBM support of these products, but for optimal performance and to ensure that all the required HW/FW fixes are installed. 4.    Once new FW has been released to the field, we will provide a 6 month grace period for customers to update these products to the currently supported FW level. 5.    Adapter in PowerVM SRIOV shared mode - This adapter firmware release notes applies to adapter configured in dedicated mode.  - When adapter is transitioned to SRIOV mode, the system firmware updates the adapter firmware, which may differ from the firmware used in dedicated mode. - When adapter is moved back to dedicated mode, user will need to update the adapter firmware to the level mentioned in this release notes. 6.    Release Notes for adapter firmware in PowerVM SRIOV shared mode. Please visit fix central (http://www.ibm.com/support/fixcentral/  ) and review the release notes pertaining  to your system MTM and installed system firmware. 7.    For more information about adapters running in  PowerVM SRIOV shared mode visit:  SRIOV FAQs (https://community.ibm.com/community/user/power/viewdocument/sr-iov-faqs-test?CommunityKey=71e6bb8a-5b34-44da-be8b-277834a183b0&tab=librarydocuments)    and vNIC FAQs (https://community.ibm.com/community/user/power/viewdocument/vnic-and-vnic-failover-faqs?CommunityKey=71e6bb8a-5b34-44da-be8b-277834a183b0&tab=librarydocuments) 8. To avoid hitting the below mentioned AIX driver issues with the new adapter microcode, please either install that APARs corresponding to your AIX/VIOS version mentioned below,       OR update your AIX/VIOS to latest level available on fix central Driver update to support transition from 4MB to 8 MB microcode file size.       AIX Level                  APAR 7200-04  (72 TL4) = IJ14175 6100-09  (61 TL9) = IJ14204 7100-05  (71 TL5) = IJ14334 7200-03  (72 TL3) = IJ14355 7200-02  (72 TL2) = IJ14640 7200-01  (72 TL1) = IJ14691 7100-04  (71 TL4) = IJ14807       3.0     Installation time   Approximately 20 minutes.   4.0 Machine's Affected   Feature Code: EC3L ·         8247-21L ·         8247-22L ·         8284-22A ·         9080-MHE ·         9080-MME ·         9119-MHE ·         9119-MME ·         8335-GTC ·         8335-GTW ·         8335-GTG ·         9008-22L ·         9009-22A ·         9223-22H ·         8335-GTH ·         8335-GTX ·         9080-M9S   Feature Code: EC3M ·         8247-42L ·         8286-41A ·         8286-42A ·         8408-44E ·         8408-E8E (1Q 2017) ·         9009-41A ·         9009-42A ·         9223-42H ·         9040-MR9   5.0 Linux Requirements Error! Bookmark not defined.   For Linux operating systems, use the following procedure:   1. Find the PCI slot the ConnectX-4 adapter is plugged into. Issue the following command:   lspci ?nn | grep ?1013"   For example: # lspci -nn |grep 1013 0001:01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27700 Family [ConnectX-4] [15b3:1013] 0001:01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27700 Family [ConnectX-4] [15b3:1013]   Note:                a. This tells us that the adapter is in the PCIe slot 0001:01:00.0 which will be needed in next steps.                          b. Each port is listed as its own PCIe ID                 0001:01:00.0 ? Port 1 of the adapter                0001:01:00.1 ? Port 2 of the adapter.             c. Either of the port PCIe IDs can be used to perform the microcode install.   2. Start MFT tools by running ?mst start?   #mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices   If the system does not have the mst command available, please install the Mellanox Firmware Tools (MFT) available here: http://www.mellanox.com/page/management_tools       3. Verify the current Firmware level by running ?flint -d -q?.  Replace with the ID found in the lscpi command from step 2.   flint -d 0001:01:00.0 q Image type:          FS3 FW Version:          12.29.1017 FW Release Date:     11.1.2021 Product Version:     rel-12_29_1017 Rom Info:            type=FCODE version=0.0.390 devid=4115 proto=ETH Description:         UID                GuidsNumber Base GUID:           248a070300594a7c        4 Base MAC:            0000248a07594a7c        4 Image VSD: Device VSD: PSID:                IBM2150110033     This command reveals the current version of the FW; which is 12.29.1017 in this particular case.  If the version is less than 12.24.8000, please update.   4. Download the firmware RPM file to the destination machine.   Use this method to download the new microcode to a Linux system: NOTE: The instructions that follow are specific Linux commands. Linux commands are CASE (lower and upper) SENSITIVE, andmust be entered exactly as shown, including filenames. * Transfer the RPM format file to the /tmp directory (using ?Save as....?). You will see the filename for the RPM file.  * Install rpm on your Linux system by typing: rpm -ivh /tmp/ b31513101410f704.001200291017.linux.rpm   The microcode package will install the firmware images in the /lib/firmware directory. If a message is displayed saying the "package is already installed", you will need to uninstall the listed rpm package. On the command line type: rpm -e where is the name of the package that was returned in the message. Return to Step 2 and attempt to install the file again.   * The microcode file will be installed to the /lib/firmware directory. The file name is?   5. Verify the contents of the image before flashing: ls -l /lib/firmware/ b31513101410f704. 001200291017to verify file size: ·         b31513101410f704. 001200291017= 16744768   sum /lib/firmware/ b31513101410f704. 001200291017 to verify Checksum: ·         b31513101410f704. 001200291017= 06690     6. Update the FW with "flint -d -i burn"       # flint -d 0001:01:00.0 -i b31513101410f704. 001200248000 burn       Current FW version on flash:  12.20.1010     New FW version:               12.29.1017   Burning FW image without signatures - OK Restoring signature                  - OK     7. At this point we have burned the new FW on to the Mellanox adapter. However, this FW will not take effect till the system is rebooted. Reloading the drivers is not enough; it requires the system to be rebooted.     6.0 AIX Requirements   Note: If microcode level on the adapter is 0001200162500 and you plan to update the microcode to a newer level. Please make sure you have the latest fixes installed. The APARs mentioned in the link below is required to be installed prior to updating the microcode. http://www-01.ibm.com/support/docview.wss?uid=isg1fixinfo166023     Adapter is supported on: ·         AIX 7.2 with Technology Level 1 and Service Pack 3 and above ·         AIX 7.1 with Technology Level 0 and Service Pack 5 and above ·         AIX 7.1 with Technology Level 4 and Service Pack 4 and above ·         VIOS 2.2.6.0 and above   If you are using another release of AIX, ensure that the adapter is supported on that release before you install the adapter. Contact service and support for assistance.   7.0 Determine the Current Microcode Level for AIX   Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the Adapter's VPD. A.    List the RoCE adapters installed in the system by typing:   lsdev | grep RoCE B.    To check the current microcode level for the adapter or controller enter the following command: lsmcode -cd entX Where "X" is the instance of the adapter. The command will produce output similar to: # lsmcode -cd ent1 The current microcode level for ent4 is 001200291017   If the ROM Level is less than 001200291017 you should update the microcode. Note: Linux firmware 12.17.1502/ 001200171502 should NOT be installed on AIX Operating sytem. That level of firmware is not currently supported on AIX and will put the adapter in ?defined? state and will be unrecoverable.   8.0 Downloading the RPM Format File to the Target Server for AIX Use this method to download to an AIX system:   NOTE: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, andmust be entered exactly as shown, including filenames.   A.    Make two directories on your AIX system to receive the RPM format file. Enter:      "mkdir /tmp/microcode" and then create this directory Enter:      "mkdir /tmp/microcode/RPM"   B.    Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as ...").  Change to that directory, "cd /tmp/microcode/RPM". You'll see the filename for the RPM file.        "rpm -ihv --ignoreos b31513101410f704.001200291017.aix.rpm" C.    For AIX:  The microcode files will be added to /etc/microcode/.  D.   Microcode file will be copied to "/etc/microcode".  The file size and checksum of the microcode image will be verified in Section 10.0. File Names: b31513101410f704.001200291017   NOTE:  - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".  - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location.   9.0 Discovery Tool Microcode CD-ROM creation and download instructions To obtain information how to burn a CD-ROM and run the Discovery Tool for an AIX or Linux System please go to:   http://www-304.ibm.com/webapp/set2/firmware/lgjsn?mode=10&page=cdrom.html   A.    After running the Discovery Tool successfully the "/tmp/microcode/RPM" directory was created and your rpm files are copied from the CD-ROM. B.    Change to that directory, "cd /tmp/microcode/RPM". C.    Unpack the file by executing the instructions below: Enter the command: "rpm -ihv --ignoreos b31513101410f704. 001200291017.aix.rpm " D.   Microcode file will be copied to "/etc/microcode".  The file size and checksum of the microcode image will be verified in Section 10.0. File Names: b31513101410f704. 001200291017     NOTE: - "/etc/microcode" is a symbolic link to "/usr/lib/microcode". - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location. - For customers using the AIX Diagnostics CD, please refer to the IBM System Hardware information Center for instructions.         10.0 Verifying microcode before download and notes for AIX Please verify the file size and checksum of the raw microcode files matches what is listed below. ls -l /etc/microcode/b31513101410f704.01200291017to verify file size: ·         b31513101410f704.001200291017= 16744768   sum /etc/microcode/ b31513101410f704. 001200291017to verify Checksum: ·         b31513101410f704.001200291017= 06690     11.0 Microcode Download Procedure for AIX 11.1 Setting up for Microcode download   A.    Stop all applications that use this interface/adapter. B.    Remove the interface/IP address from the all ports identified in section 7.0 for the adapters that will be upgraded. a.    Before detaching the interface, record the IP address and any other pertinent information that was configured on the Adapter.  This information may be needed if the microcode update overwrites this section on the Adapter. C.    If the interfaces are members of an SEA, the SEA devices must be moved to a defined state. a.    "rmdev -l enX" - where "X" is the interface number for the Shared Ethernet Adapter. b.    "rmdev -l entX" - where "X" is the interface number for the Shared Ethernet Adapter. D.   If the interfaces are members of an EtherChannel, the EtherChannel device must be moved to a define state. a.    "rmdev -l enX" - where "X? is the interface number for the EtherChannel adapter. b.    "rmdev -l entX" - where "X" is the interface number for the EtherChannel adapter. E.    For every port associated with the adapter, the enX interfaces must be changed to a defined state. a.    "rmdev -l enX" - where "X" is the interface number for the adapter port. b.    This command will be run 2 times, once for each port on the adapter.                11.2 Downloading Microcode to the Adapter A.    At the command line type "diag" B.    Select the "Task Selection" from diagnostics menu. C.    Select "Microcode Tasks? then select ?Download Microcode" from the menu. D.   Select all of the entX RoCE Network Adapters that need to be updated from the list of devices by using the arrow keys to highlight the entry and pressing "Enter" to mark it.  Press "F7" or "ESC+7" when you are done marking all the adapters you want to flash. E.    If a source selection menu is displayed, Select "/etc/microcode". F.    A dialogue box may be displayed on screen.  It will state that the current microcode level on the adapter is not in the /etc/microcode directory.  This is acceptable because the adapter will reject any incorrect code.  Press "Enter" to continue. G.   Select 001200248000 level and press "Enter" to flash the adapter. H.   The following message will appear on the screen when download is completed: "Microcode download complete successfully.  The current microcode level for the device is ...  Please run diagnostics on the adapter to ensure that it is functioning properly."  I.     If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated. J.     Exit diagnostics. 11.3 Re-configure and Verify adapters A.    Run "cfgmgr" to reconfigure the adapters that were moved to defined before the update.  B.    Verify the code level is 001200291017 C.      D.   by typing "lsmcode -cd entX" for each adapter updated, where "X" is the instance of the RoCE adapter.