*PCIe2 2-Port 10GbE RoCE SFP+ Adapter (FC: EC27, EL27 & EC28)*

*PCIe2 2-Port 10GbE RoCE SR Adapter (FC: EC29 & EC30)*

* *

 

******* PLEASE READ THIS ENTIRE NOTICE **********
*

 

*DATE: December, 1 2014 *


  Table of Contents

 

*1.0 Microcode and Document Revision History <#_1.0_Microcode_and>***

*2.0 General information <#_2.0_General_information_1>*

*3.0 Installation time <#_3.0_Installation_time>***

*4.0 Machine's Affected <#_Toc321724057>***

*5.0 Linux Requirements <#_Toc319924776>*

*6.0 AIX Requirements <#_6.0_AIX_Requirements:>***

*7.0 Determine the Current Microcode Level for AIX <#_7.0_Determine_the>*

*8.0 Downloading the RPM Format File to the Target Server for AIX
<#_8.0_Downloading_the> ***

*9.0 Discovery Tool Microcode CD-ROM creation and download instructions
<#_9.0_Discovery_Tool>*

*10.0 Verifying microcode before download and notes for AIX
<#_10.0_Verifying_microcode>***

*11.0 Microcode Download Procedure for AIX <#_11.0_Microcode_Download>*

*11.1 Setting up for Microcode download <#_11.1_Setting_up>*

*11.2 Downloading Microcode to the Adapter <#_11.2_Downloading_Microcode>*

*11.3 Verify and Re-configure adapters <#_11.3_Verify_and>*

 

 


    *1.0 Microcode and Document Revision History*

*Firmware Level*

	

*Description*

000200091000

	

Original Release

000200091316

	

Multiple bug fixes, updated UAR=7

000200091326

	

PCIe link training fix for P7 systems

Port link bounce fix when running entstat or netstat

_ _

*Document Revision History*

	

*Description*

11/14/2013

	

Creating readme file with latest microcode - 000200091316

12/1/2014

	

Update with information for new microcode version.

Add notice to reboot when updating from 2.9.1000 due to BAR size change.

Updated supported machine types

 

 


    *2.0 General information*

This Readme file is intended to give directions on how to update the
microcode found on the PCIe2 2-Port 10GbE RoCE* *adapter (FC: EC27,
EL27, EC28, EC29, EC30).

 

*Special Requirements*

If upgrading the adapter firmware from 2.9.1000 (000200091000), reboot
of the OS is required. An rmdev/cfgmgr will not be sufficient.

* *

*Non-Concurrent Download*

The microcode installation does NOT support concurrent download. The
entX devices can be used during and after the download, but update will
not go into effect until either the device is reloaded using
rmdev/cfgmgr or a reboot is performed.

 

*NOTE:* It is recommended that the installation be scheduled during a
maintenance window or during non-peak production periods.

 


    *3.0 Installation time*

Approximately 20 minutes.

 


    *4.0 Machine's Affected*

Feature Code EC27, EC29:

8202-E4B, 8202-E4C, 8202-E4D, 8205-E6B, 8205-E6C, 8205-E6D, 8231-E1C,
8231-E1D, 8231-E2B, 8231-E2C, 8231-E2D, 8246-L1C, 8246-L1D, 8246-L1S,
8246-L1T, 8246-L2C, 8246-L2D, 8246-L2S, 8246-L2T, 8247-21L, 8247-22L,
8268-E1D, 8284-22A, 9119-MHE, 9199-MME

 

Feature Code EC28, EC30:

8202-E4B, 8202-E4C, 8202-E4D, 8205-E6B, 8205-E6C, 8205-E6D, 8231-E2C,
8231-E2D, 8233-E8B, 8236-E8C, 8246-L2S, 8246-L2T, 8248-L4T, 8286-41A,
8286-42A, 8408-E8D, 8412-EAD, 9109-RMD, 9117-MMB, 9117-MMC, 9117-MMD,
9119-FHB, 9119-MHE, 9119-MME, 9179-MHB, 9179-MHC, 9179-MHD

 

Feature Code EL27:

8246-L1C, 8246-L1D, 8246-L1S, 8246-L1T, 8246-L2C, 8246-L2D, 8246-L2S,
8246-L2T

 

 


    *5.0 Linux Requirements* *Error! Bookmark not defined.*

To burn FW for Mellanox ConnectX adapters here are the instructions (for
Linux):

 

1. Find the PCI slot the ConnectX-2 adapter is plugged into. Issue the
following command:

 

lspci | grep Mellanox

 

For example:

# lspci | grep Mellanox

*0003:60:00.0 Ethernet controller: Mellanox Technologies MT26448
[ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)*

 

This tells us that the adapter is in the PCIe slot *0003:60:00.0 *which
will be needed in the following steps.

 

2. Start the MFT software suite by running “mst start” at the command line:

#mst start

Starting MST (Mellanox Software Tools) driver set

Loading MST PCI module - Success

Loading MST PCI configuration module - Success

Create devices

 

You can ignore the "unable to initialize libusb: -99" error.  This
occurs on systems without a USB port present.

 

If the system does not have the mst command available, please install
the Mellanox Firmware Tools (MFT) available here:
http://www.mellanox.com/page/management_tools

 

3. Verify the current Firmware level by running “mstflint -d *01:00.0
*-q”.  Replace “01:00.0” with the ID found in the lscpi command from step 1.

 

[root@io220 ~]# mlxburn -d 0003:60:00.0 -q

-I- Image type:      ConnectX

-I- FW Version:      2.9.1000

-I- Device ID:       26448

-I- Description:     Port1            Port2

-I- MACs:                0002c95a0118     0002c95a0119

-I- VSD:

-I- PSID:            IBM0F30001010

 

This command reveals the current version of the FW; which is 2.9.1000 in
this particular case.  If the version is less than 2.9.1326, please update.

Please note the value for PSID.  This will determine which image is
needed in later steps.

 

4. Download the firmware RPM file to the destination machine.

 

Use this method to download the new microcode to a Linux system:

*NOTE:* The instructions that follow are specific Linux commands. Linux
commands are CASE (lower and upper) SENSITIVE, and */must/* be entered
exactly as shown, including filenames.

   1. Transfer the RPM format file to the /tmp directory(using “Save
      as....”). You will see the filename for the RPM file. 
   2. Install rpm on your Linux system by typing:

*rpm -ivh /tmp/b315506714101604.000200091326.Linux.rpm*

   3. The microcode package will install the firmware images in the
      /lib/firmware directory. If a message is displayed saying the
      "package <package_name> is already installed", you will need to
      uninstall the listed rpm package. On the command line type:

*rpm -e <package_name>*

where <package_name> is the name of the package that was returned in the
message. Return to Step 2 and attempt to install the file again.

   4. The microcode files will be installed to the /lib/firmware
      directory. The file names are?*b315506714101604.000200091326
      *and* b315506714106104.000200091326*.

 

5. Verify the contents of the image before flashing:

*ls -l /lib/firmware/b31550671410??04.000200091326 *to verify file size:

·         b315506714101604.000200091326 = 671264

·         b315506714106104.000200091326 = 671220

*sum /lib/firmware/b31550671410??04.000200091326 *to verify Checksum:

·         b315506714101604.000200091326 = 26105

·         b315506714106104.000200091326 = 46644

 

6. Update the FW with "*flint -d <device id> -i <image filename> burn*"

 

*NOTE:* The PSID value from step 3 will determine which file to use. 

    * IBM0F30000010 = filename b315506714101604.000200091326
    * IBM0F30001010 = filename b315506714106104.000200091326

 

# flint -d 0003:60:00.0 -i b315506714106104.000200091326 burn

 

    Current FW version on flash:  2.9.1000

    New FW version:               2.9.1326

 

Burning FW image without signatures  - OK

Restoring signature                  - OK

 

If you receive the following error message, it means the wrong file was
selected for update, and you need to choose the other filename.

-E- PSID mismatch. The PSID on flash (IBM0F30001010) differs from the
PSID in the given image (IBM0F30000010).

 

7. At this point we have burned the new FW on to the Mellanox adapter.
However, this FW will not take effect till the system is rebooted.

Reloading the drivers is not enough; it requires the system to be rebooted.

 

 


    *6.0 AIX Requirements*

Basic card function is supported on:

            AIX 6L Version V6.1 technology level 6100-06 and later

            AIX 6L Version V6.1 technology level 6100-07 and later

            AIX 6L Version V6.1 technology level 6100-08 and later

            AIX 7L Version V7.1 technology level 7100-00 and later

            AIX 7L Version V7.1 technology level 7100-01 and later

If you are using another release of AIX, ensure that the adapter is
supported on that release before you install the adapter. Contact
service and support for assistance.

 


    *7.0 Determine the Current Microcode Level for AIX*

Before you install the microcode, it is important to determine the
microcode level of the Adapter installed in the target system. Use the
following instructions to read the ROM level stored in the Adapter's VPD.

1)  List the RoCE adapters installed in the system by typing:

# lsdev | grep RoCE

 

*Note: *All of the RoCE adapters that are installed will be displayed.
Depending on how the adapter is configured, it will display as either
entX or roceX, where X is 0, 1, etc.

 

2) To check the current microcode level for the adapter or controller
enter the following command

lsmcode -cd entX

Where "X" is the instance of the adapter. The command will produce
output similar to:

# lsmcode -cd ent4

The current microcode level for ent4

is 000200091000 

 

# lsmcode -cd roce0

The current microcode level for roce0

is 000200091316.

 

If the *ROM Level* is less than *000200091326* you should update the
microcode.


    *8.0 **Downloading the RPM Format File to the Target Server for AIX***


Use this method to download to an AIX system:

 

*NOTE:* The instructions that follow are specific AIX commands. AIX
commands are CASE (lower and upper) SENSITIVE, and */must/* be entered
exactly as shown, including filenames.

 

1)    Make two directories on your AIX system to receive the RPM format
file.
Enter:      "mkdir /tmp/microcode"
and then create this directory
Enter:      "mkdir /tmp/microcode/RPM"

 

2)    Transfer the RPM format file to the /tmp/microcode/RPM directory
(using "Save as ...").  Change to that directory, "cd /tmp/microcode/RPM".
You'll see the filename for the RPM file.
       "rpm -ihv --ignoreos b315506714101604.000200091326.aix.rpm"

3)    For AIX:  The microcode files will be added to /etc/microcode/. 

 

*NOTE:*
 - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".
 - If permission does not allow the copy to the above stated directory
or file then the user will be prompted for a new location.

 


    *9.0 Discovery Tool Microcode CD-ROM creation and download instructions*

To obtain information how to burn a CD-ROM and run the Discovery Tool
for an AIX or Linux System please go to:

 

http://www-304.ibm.com/webapp/set2/firmware/lgjsn?mode=10&page=cdrom.html

 

1)    After running the Discovery Tool successfully the
"/tmp/microcode/RPM" directory was created and your rpm files are copied
from the CD-ROM.

2)    Change to that directory, "cd /tmp/microcode/RPM".

3)    Unpack the file by executing the instructions below:
Enter the command:
"rpm -ihv --ignoreos b315506714101604.000200091326.aix.rpm"

4)    Two microcodes file will be copied to "/etc/microcode".  The file
size and checksum of the microcode image will be verified in Section 10.0.
File Names:
b315506714101604.000200091326
b315506714106104.000200091326

 

*NOTE:*

- "/etc/microcode" is a symbolic link to "/usr/lib/microcode".

- If permission does not allow the copy to the above stated directory or
file then the user will be prompted for a new location.

- For customers using the AIX Diagnostics CD, please refer to the IBM
System Hardware information Center for instructions.

 

 


    *10.0 Verifying microcode before download and notes for AIX*

Please verify the file size and checksum of the raw microcode files
matches what is listed below.


*ls -l /etc/microcode/b31550671410??04.000200091326 *to verify file size:

·         b315506714101604.000200091326 = 671264

·         b315506714106104.000200091326 = 671220

*sum /etc/microcode/b31550671410??04.000200091326 *to verify Checksum:

·         b315506714101604.000200091326 = 26105

·         b315506714106104.000200091326 = 46644

 


    *11.0 Microcode Download Procedure for AIX** *


      11.1 Setting up for Microcode download

 

1)    Stop all applications that use this interface/adapter.

2)    If the device is configured as a roceX device, not entX, skip to
section 11.2.

3)    Remove the interface/IP address from the all ports identified in
section 6.0 for the adapters that will be upgraded.

a.    Before detaching the interface, record the IP address and any
other pertinent information that was configured on the Adapter.  This
information may be needed if the microcode update overwrites this
section on the Adapter.

4)    If the interfaces are members of an SEA, the SEA devices must be
moved to a defined state.

a.    "rmdev -l enX" - where "X" is the interface number for the Shared
Ethernet Adapter.

b.    "rmdev -l entX" - where "X" is the interface number for the Shared
Ethernet Adapter.

5)    If the interfaces are members of an EtherChannel, the EtherChannel
device must be moved to a define state.

a.    "rmdev -l enX" - where "X” is the interface number for the
EtherChannel adapter.

b.    "rmdev -l entX" - where "X" is the interface number for the
EtherChannel adapter.

6)    If the adapter is configured as an Ethernet device (not roceX),
for every port associated with the adapter, the enX interfaces must be
changed to a defined state.

a.    "rmdev -l enX" - where "X" is the interface number for the adapter
port.

b.    This command will be run 2 times, once for each port on the adapter.

 

            


      11.2 Downloading Microcode to the Adapter

1)    At the command line type "diag"

2)    Select the "Task Selection" from diagnostics menu.

3)    Select "Microcode Tasks” then select “Download Microcode" from the
menu.

4)    Select all of the *roceX PCIe2 10GbE RoCE Converged Network
Adapters *or *entX RoCE Converged Network Adapters *that need to be
updated from the list of devices by using the arrow keys to highlight
the entry and pressing "Enter" to mark it.  Press "F7" or "ESC+7" when
you are done marking all the adapters you want to flash.

5)    If a source selection menu is displayed, Select "/etc/microcode".

6)    A dialogue box may be displayed on screen.  It will state that the
current microcode level on the adapter is not in the /etc/microcode
directory.  This is acceptable because the adapter will reject any
incorrect code.  Press "Enter" to continue.

7)    Select *000200091326* level and press "Enter" to flash the adapter.

8)    The following message will appear on the screen when download is
completed: "Microcode download complete successfully.  The current
microcode level for the device is ...  Please run diagnostics on the
adapter to ensure that it is functioning properly." 

9)    If you selected more than one adapter to update, then steps 6-9
will repeat until all adapters are updated.

10) Exit diagnostics.


      11.3 Verify and Re-configure adapters

1)    Verify the code level is *000200091326* by typing "lsmcode -cd
entX" or "lsmcode -cd roceX" for each adapter updated, where "X" is the
instance of the RoCE adapter.

2)    If the adapter is configured as an entX device, run "cfgmgr" to
reconfigure the device.