PCIe4 4-Port 32Gb FC Adapter
PCIe4 2-Port 64Gb FC Adapter
******* PLEASE READ THIS ENTIRE NOTICE *********
DATE: April 7, 2023
Table of Contents
1.0 Microcode and Document
Revision History:
5.0 Cautions and
Important Notes
6.0 Determine the Current
Microcode Level
7.0 Downloading the RPM
Format File to the Target Server
7.1 Downloading AIX RPM
Package
7.2 Downloading Linux RPM
Package
8.0 Discovery Tool Microcode CD-ROM
creation and download instructions
9.0 Verifying microcode
before download and notes
9.1 Verifying microcode
before download for AIX
9.2 Verifying microcode
before download for Linux
10.0 Microcode Download
Procedure for AIX:
10.1 Setting up for Microcode
download
10.2 Downloading Microcode to
the Adapter
10.3 Verify and Re-configure
adapters
11.0 Microcode Download
Procedure for Linux using Emulex hbacmd utility
11.1 Download One Command
Manager
|
Microcode Level |
Changes |
|
|
00014000020062400005 |
Impact: Usability Severity:
ATT * 64G Performance
enhancements * Resolved some 64G
congestion issues - Cisco 64G Switch
Interoperability * Added support for
new operating system releases. |
|
|
The Firmware Levels Below Are No Longer Supported by IBM Once They Have Been Removed from The Microcode Download Website. It is required to update to the latest FW level not only for continued IBM support of these products, but for optimal performance and to ensure that all the necessary HW/FW fixes are installed. The latest level can be found at IBM Fix Central · https://www.ibm.com/support/fixcentral/ |
||
|
00014000000057400013 |
Impact: Data Severity:
HIPER *
Firmware incorrectly reports out-of-order error when CCP/CCPE involved *
DUMP (0x17) Type 3 MB command fails with 0x0600 \"SFP not
installed\" status *
Adapter dropping link
on buffer credit recovery (LR/LRR) |
|
|
00014000000057400007 |
Impact: New Severity: New |
|
|
Document Revision History |
Description |
|
3.0 |
10/31: Release adapter firmware 14.2.624.5 |
|
2.0 |
04/23: Release adapter firmware 14.0.574.13 |
|
1.0 |
Initial fw
00014000000057400007 release |
This Readme file is intended to give directions on how to update the microcode found on the following adapters:
PCIe4 4-Port 32G FC Adapter (CCIN: 2CFC with FC: EN1L and EN1M)
· df1000f51410c106
PCIe3 4-Port 64G FC Adapter (CCIN: 2CFD with FC: EN1N and EN1P)
· df1000f51410c206
Approximately 20 minutes.
IBM Power 10 and Following
· Concurrent Download
The microcode installation supports concurrent download while the adapter and attached storage devices are available for use. HOWEVER, it is highly recommended that installation be scheduled during a maintenance window or during non-peak production periods.
· AIX Diag and Standalone Diag DVD
Please also use the latest AIX Standalone
Diagnostics DVD found here
Standalone
Diagnostics CD Overview and Download (ibm.com)
Important Notice: EEH error entries in AIX system errlog are expected when updating adapter FW.
Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the adapter's VPD.
1) List the Fiber Channel adapters in the system by typing:
# lscfg -l fcs\*
Note: All of the Fiber Channel ports that are installed will be displayed. They will be listed as fcsX, where X is 0, 1, etc.
2) To check the current microcode level for the adapter or controller enter the following command
# lsmcode -cd fcsX <Where “X” is the instance of the adapter>
The command will produce output similar to:
The current microcode level for fcsX
is 00014000000057400013.
If the ROM Level is less than 00014000020062400005,
you should update the microcode.
The following steps are for Linux Red Hat or SuSE
systems running 4.18 kernels which support the /sys filesystem. These steps
assume you are logged in with root permissions and one or more Fibre Channel
Adapters are installed.
1) Display the model description for each installed Fibre Channel
adapter. Expect two instances of PCIe4 x-Port xxGb
Fibre Channel Adapters to be displayed: one for each
port on the dual or quad port adapter. Type the following commands to display a
list showing the host{n}: description for each installed adapter.
# cd /sys/class/scsi_host; find -L host* -maxdepth 1 -name "modeldesc"
-printf %h: -exec cat {} \;
The output will be similar to that shown here. The host{n} values will be used
in the next step. If the list is empty, either there are no adapters installed
or there was an error in typing the command. If the list contains other
adapters but no IBM Fibre Channel adapters are listed, none are installed, and
this firmware update should not be applied.
host1:IBM xxxP/Nxxx PCIe4 x-Port xxGb Fibre Channel Adapter
host2:IBM xxxP/Nxxx PCIe4 x-Port xxGb Fibre Channel Adapter
2) Display the firmware version for
each adapter. Type the following command, replacing host{n} with the value
reported for each adapter listed in Step 1.
# cat /sys/class/scsi_host/host{n}/fwrev
The microcode version listed may vary but
output will be similar to that shown here, four numbers separated with “.”,
sli-4:6:e:
14.0.574.13, sli-4:6:e
Repeat this command for each host{n}. If one or more adapters lists a microcode
version that is not 14.2.624.5, then the microcode update should be applied.
3) Display Linux Operating System version. Type the following command to
display the Operating System version. Make a note of Operating System version,
it will be needed later.
For Red Hat systems, type the following command.
# cat /etc/redhat-release
Output will be similar to: Red Hat Enterprise Linux
Server release 8.6
For SuSE systems, type the following command:
# cat /etc/SuSE-release
Output will be similar to:
SUSE Linux Enterprise Server 15 (ppc64)
VERSION = 15
PATCHLEVEL = 4
Use this method to download to an AIX system:
NOTE: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, and must be entered exactly as shown, including filenames.
1) Make two directories on your AIX system to receive the RPM format
file.
Enter: "mkdir /tmp/microcode”
and then create this directory
Enter: "mkdir /tmp/microcode/RPM"
2) Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as ..."). Change to that directory, "cd /tmp/microcode/RPM".
You'll see the filename for the RPM file.
rpm
-ihv --ignoreos /tmp/microcode/RPM/df1000f51410c106.00014000020062400005.aix.rpm
rpm -ihv --ignoreos /tmp/microcode/RPM/df1000f51410c206.00014000020062400005.aix.rpm
3) For AIX: The microcode files will be added to /etc/microcode/.
NOTE:
- "/etc/microcode" is a symbolic link
to "/usr/lib/microcode".
- If permission does not allow the copy to the above stated directory or
file then the user will be prompted for a new location.
Follow these steps to install the Linux RPM package on a Linux system. Note that the Linux commands are case sensitive and must be entered exactly as shown, including file names.
1) Transfer the Linux rpm file to a temporary directory on the target server such as /tmp.
2) Unpack the rpm file by typing:
# rpm -ihv --ignoreos /tmp/microcode/RPM/df1000f51410c106.00014000020062400005.grp.linux.rpm or. # rpm -ihv --ignoreos /tmp/microcode/RPM/df1000f51410c206.00014000020062400005.grp.linux.rpm
3) The microcode package extracts the firmware file to the /lib/firmware directory. If a message is displayed saying the "package <package_name> is already installed", you will need to uninstall the rpm package. On the command line type:
# rpm -e <package_name>
where <package_name> is the name of the package that was returned in the message. Return to Step 2 and attempt to unpack the file again.
4) Verify the microcode image in Section 10.2
To obtain information how to burn a DVD-ROM and run the Discovery Tool for an AIX or Linux System please go to: http://www14.software.ibm.com/support/customercare/mds/
1) After running the Discovery Tool successfully, the "/tmp/microcode/RPM" directory was created, and your rpm files are copied from the DVD-ROM.
2) Change to that directory, "cd /tmp/microcode/RPM".
3) Unpack the file by executing the instructions below:
Enter the commands:
rpm -ihv
--ignoreos /tmp/microcode/RPM/df1000f51410c106.00014000020062400005.aix.rpm
or
rpm -ihv --ignoreos
/tmp/microcode/RPM/df1000f51410c206.00014000020062400005.aix.rpm
4) A microcode file will be copied to "/etc/microcode".
File Name: df1000f51410c106.00014000020062400005
Or
File Name: df1000f51410c206.00014000020062400005
NOTE:
- "/etc/microcode" is a symbolic link
to "/usr/lib/microcode".
- If permission does not allow the copy to the above stated directory or
file then the user will be prompted for a new location.
Please verify the file size and checksum of the raw microcode files matches what is listed below.
For AIX:
Please verify the file size and checksum
# ls -l /usr/lib/microcode/df1000f51410c106.00014000020062400005
-rwxr-xr-x. 1 root root 2245312 Mar 28 12:48
/usr/lib/microcode/df1000f51410c106.00014000020062400005
# sum /usr/lib/microcode/df1000f51410c106.00014000020062400005
45073 2193
Or
# ls -l /usr/lib/microcode/df1000f51410c206.00014000020062400005
-rwxr-xr-x. 1 root root 2245312 Mar 28 12:48
/usr/lib/microcode/df1000f51410c206.00014000020062400005
# sum /usr/lib/microcode/df1000f51410c206.00014000020062400005
45073 2193
Please verify the file location, size and checksum of the raw microcode files matches what is listed below for Linux:
Please verify file size and checksum
# ls -l /lib/firmware/df1000f51410c106.00014000020062400005.grp
-rwxr-xr-x. 1 root root 2245312 Mar 28 12:48
/lib/firmware/df1000f51410c106.00014000020062400005.grp
# sum /lib/firmware/df1000f51410c106.00014000020062400005.grp
45073 2193
or
# ls -l /lib/firmware/df1000f51410c206.00014000020062400005.grp
-rwxr-xr-x. 1 root root 2245312 Mar 28 12:48
/lib/firmware/df1000f51410c206.00014000020062400005.grp
# sum /lib/firmware/df1000f51410c206.00014000020062400005.grp
45073 2193
Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's).
1) Stop all applications that use this interface/adapter
2) You only need to pick one of the fcs#s per adapter for microcode update. Diag will update microcode for all ports.
1) At the command line type “diag”
2) Select the "Task Selection" from diagnostics menu.
3) Select "Microcode Tasks” then select “Download Microcode" from the menu.
4) Select all of the Fibre Channel Adapters that need to be updated from the list of devices
by using the arrow keys to highlight the entry and pressing "Enter" to mark it. The devices will be displayed as fcs#, where # is the number of the device. Press "F7" when you are done marking all the adapters you want to flash.
5) If a source selection menu is displayed, Select “/etc/microcode”.
6) A dialogue box will be displayed on screen. It will state the current microcode level and list the available levels.
7) Select the 00014000020062400005 level and press "Enter" to flash the adapter.
8) The following message will appear on the screen when download is completed: "Installation of the microcode has completed successfully on the following instances of the adapter:… The current microcode level …… Please run diagnostics on these resources to ensure that the adapter is functioning properly."
9) If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated.
10) Exit diagnostics.
1) Verify the code level is 00014000020062400005 by typing "lsmcode -cd fcs#" for each Fiber Channel adapter updated. <Where # is the instance of the adapter>
2) Re-configure/Verify the adapter’s information through the “smit” menus if necessary.
Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's).
Microcode updates on Fibre Channel adapters installed in a Linux system require the use of the Emulex hbacmd utility. The hbacmd utility is included in the Emulex One Command CLI Applications Kit downloads web page located at
https://www.broadcom.com/support/download-search
· Group: Storage Adapters, Controllers, and ICs
· Family: Fibre Channel Host Bus Adapters
· OEM: IBM
· Product: All
· Asset type: All
· Keyword: PowerPC
· Select “Management Software and Tools”
For purpose of updating the microcode, One Command Manager (CLI) will be used. Select the Application Kit {version} (CLI) link.
When prompted, save the Application Kit a directory of your choice on the target server.
Complete the requested user registration information, then select Download Driver button.
When prompted, save the elxcmcore-xxxx-xxxx-x.x.x.x-x.tgz file to a temporary directory, /tmp or you may save to directory of your choice.
Change directory to the directory where you have saved the elxcmcore-xxxx-xxxx-x.x.x.x-x.tar archive.
Unpack the tgz
archive, by typing:
# tar -xvf elxcmcore-xxxx-xxxx-x.x.x.x-x.tar
Change directory to elxocmcore-xxxx-xxxx-x.x.x.x-x
and install the utility, by typing:
# ./install.sh
Answer questions as prompted using descriptive information provided as a guide.
Display the utility version to verify the hbacmd command is functional by typing:
# hbacmd version
Utility component version information should be displayed
Ensure that all I/O activity to storage devices controlled by the adapter is stopped prior to proceeding with firmware update as the adapter will be reset after firmware update.
List installed Emulex adapters by typing: hbacmd listhbas
Output from this command will be similar to the following but specific to
your device type. This information will be repeated one section for each
discovered adapter. Adapters can be identified by looking for e300 in Flags
value. For each adapter, make a note of the Port WWN: value. The Port WWN
values will be required as an argument for commands in next steps.
Manageable HBA List
Port WWN : 10:00:00:10:9b:17:08:cc
Node WWN : 20:00:00:10:9b:17:08:cc
Fabric Name : 10:00:c4:f5:7c:cd:ef:a0
Flags : 8000e300
Host Name : system.ibm.com
Mfg
: Emulex Corporation
Serial
No.
: 1H6CR0Y06D
Port Number : 1
Mode : Initiator
PCI Bus Number : 96
PCI Function : 1
Port Type : FC
Model : xxxP/Nxxx
List hba attributes for each adapter listed in the above step, Type:
# hbacmd hbaattributes {wwpn}, where {wwpn} is one of the port WWPN values.
The output will look similar to the following and will list current versions of FW Version, SLI2 FW, SLI3 FW as well as Boot Code, all of which will be updated by microcode update. Save current operational firmware values for comparison post update.
# hbacmd hbaattributes 10:00:00:10:9b:17:08:cb
HBA Attributes for 10:00:00:10:9b:17:08:cb
Host Name : system.ibm.com
Manufacturer : Emulex Corporation
Serial Number : 1H6CR0Y06D
Model : xxxP/Nxxx
Model Desc : IBM xxxP/Nxxx PCIe4 x-Port xxGb Fibre Channel Adapter
Node WWN : 20 00 00 10 9b 17 08 cb
Node Symname :
HW Version : 0000000c
FW Version : 12.0.574.7
Vendor Spec ID : 10DF
Number of Ports : 1
Driver Name : lpfc
Driver Version : 14.0.574.7
Device ID : F500
HBA Type : xxxP/Nxxx
Operational FW : 14.0.574.7
IEEE Address : 00 10 9b 17 08 cb
Boot Code : Enabled
Boot Version : 14.0.574.7
Board Temperature : Normal
Function Type : FC
Sub Device ID : xxxPCI-IDxxx
PCI Bus Number : 96
PCI Func Number : 0
Sub Vendor ID : 1014
IPL Filename : H62LP1
Service Processor FW Name
:
14.0.574.7
ULP FW Name
: 14.0.574.7
FC Universal BIOS Version
:
14.0.574.7
FC x86 BIOS Version
:
14.0.574.7
FC EFI BIOS Version
:
14.0.574.7
FC FCODE Version
:
14.0.574.7
Flash Firmware Version
:
14.0.574.7
Secure Firmware : Enabled
Update microcode on one port per adapter. This step assumes that microcode
image .rpm package has been installed on system as described in section 8.2.
Warning: Do not interrupt or power off the system while microcode update(s) are
in progress.
Start microcode update on one adapter port, type:
hbacmd download {wwpn} /lib/firmware/df1000f51410c106.00014000020062400005.grp
Where {wwpn} is one of the port WWN values of adapter.
Or
hbacmd download {wwpn} /lib/firmware/df1000f51410c206.00014000020062400005.grp
Completion of download will be indicated by “Download Complete.” message.
System must be rebooted for firmware to be loaded. Reboot system
After system reboot, check firmware version as described in Section 7.2 or using: hbacmd hbaattributes {wwpn} as above for each port.
If, firmware version is not the version you are expecting, carefully retrace steps for omissions and examine console output for errors. Contact IBM support if you need further assistance.