Dual-port QSFP QDR IB/10GbE PCIe 2.0 (FC: 5283, 5285) ******* PLEASE READ THIS ENTIRE NOTICE ********** * *DATE: November 14, 2013 * *Table of Contents* _ _ *Dual-port QSFP QDR IB/10GbE PCIe 2.0** (FC:** **5283, 5285**)* *1.0 Microcode and Document Revision History <#_1.0_Microcode_and>* *2.0 General information <#_2.0_General_information>* *3.0 Installation time <#_3.0_Installation_time>* *4.0 Machine's Affected <#_Toc321724057>* *5.0 Linux Requirements* *6.0 AIX Requirements <#_6.0_AIX_Requirements:>* *7.0 Determine the Current Microcode Level for AIX* <#_7.0_Determine_the> *8.0 Downloading the RPM Format File to the Target Server for AIX <#_8.0_Downloading_the> * *9.0 Discovery Tool Microcode CD-ROM creation and download instructions <#_9.0_Discovery_Tool>* *10.0 Verifying microcode before download and notes for AIX <#_10.0_Verifying_microcode>* *11.0 Microcode Download Procedure for AIX <#_11.0_Microcode_Download>* 11.1 Setting up for Microcode download <#_11.1_Setting_up> 11.2 Downloading Microcode to the Adapter <#_11.2_Downloading_Microcode> 11.3 Verify and Re-configure adapters <#_11.3_Verify_and> *1.0 Microcode and Document Revision History:** * *Firmware Level* *Description* 000200091000 Original Release 000200091316 Multiple bug fixes, updated UAR=7 _ _ *Document Revision History* *Description* 11/14/2013 Updated readme with latest microcode - 000200091316 04/10/2013 Updated the Linux instruction in Section 5.0 03/14/2012 Creating Readme file with latest microcode for this adapter- 000200091000 *_ _**2.0 General information** * This Readme file is intended to give directions on how to update the microcode found on the Dual-port QSFP QDR IB/10GbE PCIe 2.0 (FC: 5283, 5285). *3.0 Installation time* Approximately 20 minutes. *4.0 Machine's Affected:** * Feature Code 5283 : 8202-E4B / 8205-E6B, 9125-F2C, 8231-E1C / 8231-E2C, 8202-E4C / 8205-E6C, 8246-L2D / 8246-L2S Feature Code 5285 : 8202-E4C / 8205-E6C, 9117-MMC / 9179-MHC *5.0 Linux Requirements:* *Error! Bookmark not defined.* To burn FW for Mellanox ConnectX adapters here are the instructions (for Linux): 1. Find the PCI slot the ConnectX-2 adapter is plugged into. Issue the following command: lspci | grep Mellanox For example: # lspci | grep Mellanox *01:00.0* InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) This tells us that the adapter is in the PCI slot 0000:01:00.0 which will be needed in the following steps. 2. Start the MFT software suite by running ?mst start? at the command line: #mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices You can ignore the "unable to initialize libusb: -99" error. This occurs on systems without a USB port present. 3. Verify the current Firmware level by running ?mstflint -d *01:00.0 *-q?. Replace ?01:00.0? with the ID found in the lscpi command from step 1. # mlxburn -d 01:00.0 -q -I- Image type: ConnectX -I- FW Version: 2.9.1000 -I- Device ID: 26428 -I- Description: Node Port1 Port2 Sys image -I- GUIDs: 0002c903000c2d5a 0002c903000c2d5b 0002c903000c2d5c 0002c903000c2d5d -I- MACs: 0002c90c2d5a 0002c90c2d5b -I- VSD: -I- PSID: IBM0FA0000009 This command reveals the current version of the FW; which is 2.9.1000 in this particular case. If the version is less than 2.9.1316, please update. 4. Download the firmware RPM file to the destination machine. Use this method to download the new microcode to a Linux system: *NOTE:* The instructions that follow are specific Linux commands. Linux commands are CASE (lower and upper) SENSITIVE, and */must/* be entered exactly as shown, including filenames. 1. Transfer the RPM format file to the /tmp directory(using ?Save as....?). You will see the filename for the RPM file. 2. Install rpm on your Linux system by typing: *rpm** -ivh /tmp/b3153c6714101504.000200091316.Linux.rpm* 1. The microcode package will install the firmware images in the /lib/firmware directory. If a message is displayed saying the "package is already installed", you will need to uninstall the listed rpm package. On the command line type: *rpm** -e * where is the name of the package that was returned in the message. Return to Step 2 and attempt to install the file again. 2. The microcode file will be installed to the /lib/firmware directory. The file name is?*b3153c6714101504.000200091316*. 5. Verify the contents of the image before flashing: *ls** -l /lib/firmware/b3153c6714101504.000200091316* to verify file size is *671240* *sum** /lib/firmware/b3153c6714101504.000200091316* to verify Checksum is *18690* for both files listed 6. Update the FW with "*flint -d -i burn*" # flint -d 01:00.0 -i b3153c6714101504.000200091316 burn Current FW version on flash: 2.9.1000 New FW version: 2.9.1316 Burning FW image without signatures - OK Restoring signature - OK 7. At this point we have burned the new FW on to the Mellanox adapter. However, this FW will not take effect till the system is rebooted. Reloading the drivers is not enough; it requires the system to be rebooted. *6.0 AIX Requirements:** * Basic card function is supported on : AIX 5L Version V5.3 technology level 5300-12 and later AIX 6L Version V6.1 technology level 6100-06 and later AIX 6L Version V6.1 technology level 6100-07 and later AIX 6L Version V6.1 technology level 6100-08 and later AIX 7L Version V7.1 technology level 7100-00 and later AIX 7L Version V7.1 technology level 7100-01 and later If you are using another release of AIX, ensure that the adapter is supported on that release before you install the adapter. Contact service and support for assistance. *7.0 Determine the Current Microcode Level for AIX:** * Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the Adapter's VPD. 1) List the InfiniBand adapters installed in the system by typing: lscfg -l iba\* Note: All of the InfiniBand adapters that are installed will be displayed. They will be listed as ibaX, where X is 0, 1, etc. 2) To check the current microcode level for the adapter or controller enter the following command lsmcode -cd ibax Where ?x? is the instance of the adapter. The command will produce output similar to: The current microcode level for iba0 is 000200091000 If the *ROM Level* is less than *000200091316* you should update the microcode. *8.0 **Downloading the RPM Format File to the Target Server for AIX* Use this method to download to an AIX system: ** *NOTE:* The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, and */must/* be entered exactly as shown, including filenames. 1) Make two directories on your AIX system to receive the RPM format file. Enter: "mkdir /tmp/microcode" and then create this directory Enter: "mkdir /tmp/microcode/RPM" 2) Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as ..."). Change to that directory, "cd /tmp/microcode/RPM". You'll see the filename for the RPM file. "rpm -ihv --ignoreos b3153c6714101504.000200091316.aix.rpm" 3) For AIX: The microcode files will be added to /etc/microcode/. *NOTE:* - "/etc/microcode" is a symbolic link to "/usr/lib/microcode". - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location. *9.0 Discovery Tool Microcode CD-ROM creation and download instructions** * To obtain information how to burn a CD-ROM and run the Discovery Tool for an AIX or Linux System please go to: http://www14.software.ibm.com/support/customercare/mds/ 1) After running the Discovery Tool successfully the "/tmp/microcode/RPM" directory was created and your rpm files are copied from the CD-ROM. 2) Change to that directory, "cd /tmp/microcode/RPM". 3) Unpack the file by executing the instructions below: Enter the commands: rpm -ihv --ignoreos b3153c6714101504.000200091316.aix.rpm 4) One microcode file will be copied to "/etc/microcode". The file size and checksum of the microcode image will be verified in Section 9.0. File Name: b3153c6714101504.000200091316 *NOTE:* - "/etc/microcode" is a symbolic link to "/usr/lib/microcode". - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location. *10.0 Verifying microcode before download and notes for AIX** * Please verify the file size and checksum of the raw microcode files matches what is listed below. For AIX : "ls -l /etc/microcode/b3153c6714101504.000200091316" to verify file size is *671240*. "sum /etc/microcode/b3153c6714101504.000200091316" to verify Checksum is *18690*. *11.0 Microcode Download Procedure for AIX: * Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's). 11.1 Setting up for Microcode download 1) Stop all applications that use this interface/adapter 2) Detach the Adapter?s interface (not always needed, but it is a good practice to do this step. a) Before detaching the interface, record the ip address and any other pertinent information that was configured on the Adapter. This information may be needed if the microcode update overwrites this section on the Adapter. 11.2 Downloading Microcode to the Adapter 1) At the command line type ?diag? 2) Select the "Task Selection" from diagnostics menu. 3) Select "Microcode Tasks? then select ?Download Microcode" from the menu. 4) Select all of the *PCIE QDR InfiniBand HCA (b3153c67) *that need to be updated from the list of devices by using the arrow keys to highlight the entry and pressing "Enter" to mark it. The devices will be displayed as iba#, where # is the number of the device. Press "F7" when you are done marking all the adapters you want to flash. 5) If a source selection menu is displayed, Select ?/etc/microcode?. 6) A dialogue box may be displayed on screen. It will state that the current microcode level on the adapter is not in the /etc/microcode directory. This is acceptable because the adapter will reject any incorrect code. Press "Enter" to continue. 7) Select *000200091316* level and press "Enter" to flash the adapter. 8) The following message will appear on the screen when download is completed: "Microcode download complete successfully. The current microcode level for the device iba# is ... Please run diagnostics on the adapter to ensure that it is functioning properly." 9) If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated. 10) Exit diagnostics. 11.3 Verify and Re-configure adapters 1) Verify the code level is *000200091316* by typing "lsmcode ?cd iba#" for each InfiniBand adapter updated. 2) Re-configure/Verify the adapter?s information (e.g. interface ip) through the ?smit? menus.