PCIe2 2-Port 10GbE RoCE
SFP+ Adapter (FC: EC27, EL27 & EC28)
PCIe2 2-Port 10GbE RoCE SR
Adapter (FC: EC29 & EC30)
******* PLEASE
READ THIS ENTIRE NOTICE *********
DATE: December, 1 2014
1.0 Microcode and
Document Revision History
7.0
Determine the Current Microcode Level for AIX
8.0 Downloading the RPM Format File to the
Target Server for AIX
9.0 Discovery Tool
Microcode CD-ROM creation and download instructions
10.0 Verifying
microcode before download and notes for AIX
11.0 Microcode Download Procedure for AIX
11.1 Setting up for Microcode download
11.2 Downloading Microcode to the
Adapter
11.3 Verify and Re-configure adapters
|
Firmware
Level |
Description |
|
000200091000 |
Original
Release |
|
000200091316 |
Multiple
bug fixes, updated UAR=7 |
|
000200091326 |
PCIe
link training fix for P7 systems Port
link bounce fix when running entstat or netstat |
|
Document
Revision History |
Description |
|
11/14/2013 |
Creating
readme file with latest microcode - 000200091316 |
|
12/1/2014 |
Update
with information for new microcode version. Add
notice to reboot when updating from 2.9.1000 due to BAR size change. Updated
supported machine types |
This Readme file is intended to give directions on how to update the
microcode found on the PCIe2 2-Port
10GbE RoCE adapter (FC: EC27, EL27, EC28, EC29, EC30).
Special Requirements
If upgrading the adapter firmware from 2.9.1000 (000200091000), reboot
of the OS is required. An rmdev/cfgmgr will not be sufficient.
Non-Concurrent Download
The microcode installation does NOT support concurrent download. The
entX devices can be used during and after the download, but update will not go
into effect until either the device is reloaded using rmdev/cfgmgr or a reboot
is performed.
NOTE: It is recommended that the installation be scheduled during a
maintenance window or during non-peak production periods.
Approximately 20 minutes.
Feature Code EC27, EC29:
8202-E4B, 8202-E4C, 8202-E4D, 8205-E6B, 8205-E6C,
8205-E6D, 8231-E1C, 8231-E1D, 8231-E2B, 8231-E2C, 8231-E2D, 8246-L1C, 8246-L1D,
8246-L1S, 8246-L1T, 8246-L2C, 8246-L2D, 8246-L2S, 8246-L2T, 8247-21L, 8247-22L,
8268-E1D, 8284-22A, 9119-MHE, 9199-MME
Feature Code EC28, EC30:
8202-E4B, 8202-E4C, 8202-E4D, 8205-E6B, 8205-E6C,
8205-E6D, 8231-E2C, 8231-E2D, 8233-E8B, 8236-E8C, 8246-L2S, 8246-L2T, 8248-L4T,
8286-41A, 8286-42A, 8408-E8D, 8412-EAD, 9109-RMD, 9117-MMB, 9117-MMC, 9117-MMD,
9119-FHB, 9119-MHE, 9119-MME, 9179-MHB, 9179-MHC, 9179-MHD
Feature Code EL27:
8246-L1C, 8246-L1D, 8246-L1S, 8246-L1T, 8246-L2C,
8246-L2D, 8246-L2S, 8246-L2T
To burn FW for Mellanox ConnectX adapters here are
the instructions (for Linux):
1. Find the PCI slot the ConnectX-2 adapter is
plugged into. Issue the following command:
lspci | grep Mellanox
For example:
# lspci | grep Mellanox
0003:60:00.0 Ethernet controller: Mellanox Technologies MT26448
[ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
This tells us that the adapter is in the PCIe slot 0003:60:00.0 which will be needed in the following
steps.
2. Start the MFT software suite by running “mst
start” at the command line:
#mst start
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
You can ignore the "unable to initialize
libusb: -99" error. This occurs on
systems without a USB port present.
If the system does not have the mst command
available, please install the Mellanox Firmware Tools (MFT) available here: http://www.mellanox.com/page/management_tools
3. Verify the current Firmware level by running “mstflint
-d 01:00.0 -q”. Replace “01:00.0”
with the ID found in the lscpi command from step 1.
[root@io220 ~]# mlxburn -d
0003:60:00.0 -q
-I- Image type: ConnectX
-I- FW Version: 2.9.1000
-I- Device ID: 26448
-I- Description: Port1
Port2
-I- MACs: 0002c95a0118 0002c95a0119
-I- VSD:
-I- PSID: IBM0F30001010
This command reveals the current version of the FW;
which is 2.9.1000 in this particular case.
If the version is less than 2.9.1326, please update.
Please note the value for PSID. This will determine which image is needed in
later steps.
4. Download the firmware RPM file to the
destination machine.
Use this method to download the new
microcode to a Linux system:
NOTE: The instructions that follow are specific Linux
commands. Linux commands are CASE (lower and upper) SENSITIVE, and must
be entered exactly as shown, including filenames.
rpm
-ivh /tmp/b315506714101604.000200091326.Linux.rpm
rpm
-e <package_name>
where <package_name> is the name of
the package that was returned in the message. Return to Step 2 and attempt to
install the file again.
5. Verify the contents of the image before
flashing:
ls
-l /lib/firmware/b31550671410??04.000200091326 to verify file size:
·
b315506714101604.000200091326 = 671264
·
b315506714106104.000200091326 = 671220
sum
/lib/firmware/b31550671410??04.000200091326 to verify Checksum:
·
b315506714101604.000200091326 = 26105
·
b315506714106104.000200091326 = 46644
6. Update the FW with "flint -d <device
id> -i <image filename> burn"
NOTE: The PSID value from step 3 will determine which
file to use.
# flint -d 0003:60:00.0
-i b315506714106104.000200091326 burn
Current FW version on flash: 2.9.1000
New FW version: 2.9.1326
Burning FW image without
signatures - OK
Restoring signature - OK
If you receive the
following error message, it means the wrong file was selected for update, and
you need to choose the other filename.
-E- PSID mismatch. The
PSID on flash (IBM0F30001010) differs from the PSID in the given image
(IBM0F30000010).
7. At this point we have burned the new FW on to
the Mellanox adapter. However, this FW will not take effect till the system is
rebooted.
Reloading the drivers is not enough; it requires
the system to be rebooted.
Basic card function is supported on:
AIX 6L Version V6.1 technology level 6100-06 and later
AIX
6L Version V6.1 technology level 6100-07 and later
AIX
6L Version V6.1 technology level 6100-08 and later
AIX 7L
Version V7.1 technology level 7100-00 and later
AIX
7L Version V7.1 technology level 7100-01 and later
If you are using another release of AIX, ensure that the adapter is
supported on that release before you install the adapter. Contact service and
support for assistance.
Before you install the microcode, it is important to determine the
microcode level of the Adapter installed in the target system. Use the
following instructions to read the ROM level stored in the Adapter's VPD.
1) List the RoCE adapters installed in the system by typing:
# lsdev | grep RoCE
Note: All of the RoCE adapters that are installed will be displayed. Depending
on how the adapter is configured, it will display as either entX or roceX,
where X is 0, 1, etc.
2) To check the current microcode level for the adapter or controller
enter the following command
lsmcode -cd entX
Where "X" is the instance of the adapter. The command will
produce output similar to:
# lsmcode -cd ent4
The current microcode
level for ent4
is 000200091000
# lsmcode -cd roce0
The current microcode
level for roce0
is 000200091316.
If the ROM Level is less than 000200091326 you should
update the microcode.
Use this method to
download to an AIX system:
NOTE: The instructions that follow are specific AIX
commands. AIX commands are CASE (lower and upper) SENSITIVE, and must
be entered exactly as shown, including filenames.
1)
Make two directories on your AIX system to receive
the RPM format file.
Enter: "mkdir /tmp/microcode"
and then create this directory
Enter: "mkdir /tmp/microcode/RPM"
2)
Transfer the RPM format file to the
/tmp/microcode/RPM directory (using "Save as ..."). Change to
that directory, "cd /tmp/microcode/RPM".
You'll see the filename for the RPM file.
"rpm -ihv --ignoreos
b315506714101604.000200091326.aix.rpm"
3)
For AIX: The microcode files will be added to
/etc/microcode/.
NOTE:
- "/etc/microcode" is a symbolic link to
"/usr/lib/microcode".
- If permission does not allow the copy to the above stated directory or
file then the user will be prompted for a
new location.
To obtain information how to burn a CD-ROM and run the Discovery Tool
for an AIX or Linux System please go to:
http://www-304.ibm.com/webapp/set2/firmware/lgjsn?mode=10&page=cdrom.html
1)
After running the Discovery Tool successfully the
"/tmp/microcode/RPM" directory was created and your rpm files are
copied from the CD-ROM.
2)
Change to that directory, "cd
/tmp/microcode/RPM".
3)
Unpack the file by executing the instructions below:
Enter the command:
"rpm -ihv --ignoreos b315506714101604.000200091326.aix.rpm"
4)
Two microcodes file will be copied to
"/etc/microcode". The file
size and checksum of the microcode image will be verified in Section 10.0.
File Names:
b315506714101604.000200091326
b315506714106104.000200091326
NOTE:
- "/etc/microcode" is a symbolic link to
"/usr/lib/microcode".
- If permission does not allow the copy to the above stated directory or
file then the user will be prompted for a new location.
- For customers using the AIX Diagnostics CD, please refer to the IBM
System Hardware information Center for instructions.
Please verify the file size and checksum of the raw microcode files matches
what is listed below.
ls -l /etc/microcode/b31550671410??04.000200091326 to verify file size:
·
b315506714101604.000200091326 = 671264
·
b315506714106104.000200091326 = 671220
sum
/etc/microcode/b31550671410??04.000200091326 to verify Checksum:
·
b315506714101604.000200091326 = 26105
·
b315506714106104.000200091326 = 46644
1)
Stop all applications that use this
interface/adapter.
2)
If the device is configured as a roceX device, not
entX, skip to section 11.2.
3)
Remove the interface/IP address from the all ports
identified in section 6.0 for the adapters that will be upgraded.
a.
Before detaching the interface, record the IP
address and any other pertinent information that was configured on the
Adapter. This information may be needed
if the microcode update overwrites this section on the Adapter.
4)
If the interfaces are members of an SEA, the SEA
devices must be moved to a defined state.
a.
"rmdev -l enX" - where "X" is
the interface number for the Shared Ethernet Adapter.
b.
"rmdev -l entX" - where "X" is
the interface number for the Shared Ethernet Adapter.
5)
If the interfaces are members of an EtherChannel,
the EtherChannel device must be moved to a define state.
a.
"rmdev -l enX" - where "X” is the
interface number for the EtherChannel adapter.
b.
"rmdev -l entX" - where "X" is
the interface number for the EtherChannel adapter.
6)
If the adapter is configured as an Ethernet device
(not roceX), for every port associated with the adapter, the enX interfaces
must be changed to a defined state.
a.
"rmdev -l enX" - where "X" is
the interface number for the adapter port.
b.
This command will be run 2 times, once for each
port on the adapter.
1)
At the command line type "diag"
2)
Select the "Task Selection" from
diagnostics menu.
3)
Select "Microcode Tasks” then select “Download
Microcode" from the menu.
4)
Select all of the roceX PCIe2 10GbE RoCE
Converged Network Adapters or
entX RoCE Converged Network Adapters that need to be updated from
the list of devices by using the arrow keys to highlight the entry and pressing
"Enter" to mark it. Press "F7" or "ESC+7"
when you are done marking all the adapters you want to flash.
5)
If a source selection menu is displayed, Select
"/etc/microcode".
6)
A dialogue box may be displayed on screen. It
will state that the current microcode level on the adapter is not in the
/etc/microcode directory. This is acceptable because the adapter will
reject any incorrect code. Press "Enter" to continue.
7)
Select 000200091326 level and press
"Enter" to flash the adapter.
8)
The following message will appear on the screen
when download is completed: "Microcode download complete
successfully. The current microcode level for the device is ...
Please run diagnostics on the adapter to ensure that it is functioning
properly."
9)
If you selected more than one adapter to update,
then steps 6-9 will repeat until all adapters are updated.
10)
Exit diagnostics.
1)
Verify the code level is 000200091326 by
typing "lsmcode -cd entX" or "lsmcode -cd roceX" for each
adapter updated, where "X" is the instance of the RoCE adapter.
2)
If the adapter is configured as an entX device, run
"cfgmgr" to reconfigure the device.