PCIe4 4-Port 32Gb FC Adapter

PCIe4 2-Port 64Gb FC Adapter

 

******* PLEASE READ THIS ENTIRE NOTICE *********

 

DATE: July 29, 2024

Table of Contents

1.0 Microcode and Document Revision History:

2.0 General information

3.0 Installation time

4.0 Machine's Affected:

5.0 Cautions and Important Notes

6.0 Determine the Current Microcode Level

6.1 AIX Systems:

6.2 Linux Systems:

7.0 Downloading the RPM Format File to the Target Server

7.1 Downloading AIX RPM Package

7.2 Downloading Linux RPM Package

8.0 Discovery Tool Microcode CD-ROM creation and download instructions

9.0 Verifying microcode before download and notes

9.1 Verifying microcode before download for AIX

9.2 Verifying microcode before download for Linux

10.0 Microcode Download Procedure for AIX:

10.1 Setting up for Microcode download

10.2 Downloading Microcode to the Adapter

10.3 Verify and Re-configure adapters

11.0 Microcode Download Procedure for Linux using Emulex hbacmd utility

11.1 Download One Command Manager

11.2 Download Code to Adapter

 

 

 

1.0 Microcode and Document Revision History:
 

Microcode Level

Changes

00014000020062400018

Impact: Func   Severity: ATT

·        Adapters are not rejecting class 2 frames

·        WQE Chaining potential for intermittent boot failures

·        Update of FCODE to v320 (adapter boot code)

·        N_Port login command rejected with invalid RPI (NPIV defect)

·        Added support for new operating system releases.

The Firmware Levels Below Are No Longer Supported by IBM Once They Have Been Removed from The Microcode Download Website.

It is required to update to the latest FW level not only for continued IBM support of these products, but for optimal performance and to ensure that all the necessary HW/FW fixes are installed.

The latest level can be found at IBM Fix Central

·       https://www.ibm.com/support/fixcentral/

00014000020062400005

Impact: Func   Severity: ATT

* 64G Performance enhancements

* Resolved some 64G congestion issues

- Cisco 64G Switch Interoperability

* Added support for new operating system releases.

00014000000057400013

Impact: Data   Severity: HIPER
* This firmware update for feature codes EN1L, EN1M, EN1N and EN1P fixes a problem encountered with Linux and attached tape devices. If an operation takes an extended amount of time to complete due to an error or a long running I/O operation, Sequence Level Error Recovery can result in unexpected data being written/read to/from the device resulting in undetected data corruption.  AIX, VIOS and IBM i configurations are not exposed to the problem.

* Firmware incorrectly reports out-of-order error when CCP/CCPE involved

* DUMP (0x17) Type 3 MB command fails with 0x0600 \"SFP not installed\" status

* Adapter dropping link on buffer credit recovery (LR/LRR)

00014000000057400007

Impact: New   Severity: New
This is the initial level of microcode released on the adapters

 

Document Revision History

Description

4.0

07/29/24: Release adapter firmware 14.2.624.18

3.0

10/31/23: Release adapter firmware 14.2.624.5

2.0

04/23/23: Release adapter firmware 14.0.574.13

1.0

Initial fw 00014000000057400007 release


2.0 General information

This Readme file is intended to give directions on how to update the microcode found on the following adapters:

PCIe4 4-Port 32G FC Adapter (CCIN: 2CFC with FC: EN1L and EN1M)

·       df1000f51410c106

 

PCIe3 4-Port 64G FC Adapter (CCIN: 2CFD with FC: EN1N and EN1P)

·       df1000f51410c206

 

3.0 Installation time

Approximately 20 minutes.

 

4.0 Machine's Affected: 

IBM Power 10 and Following

 

5.0 Cautions and Important Notes

 ·    Concurrent Download

The microcode installation supports concurrent download while the adapter and attached storage devices are available for use. HOWEVER, it is highly recommended that installation be scheduled during a maintenance window or during non-peak production periods.

·    AIX Diag and Standalone Diag DVD

Please also use the latest AIX Standalone Diagnostics DVD found here
Standalone Diagnostics CD Overview and Download (ibm.com)    

 Important Notices: 

·       EEH error entries in AIX system errlog are expected when updating adapter FW.

·       Quiesce loads on adapter doing updates or long timeouts may result.  

 

6.0 Determine the Current Microcode Level

 6.1 AIX Systems:

Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the adapter's VPD.

1)  List the Fiber Channel adapters in the system by typing:

lscfg -l fcs*

Note: All of the Fiber Channel ports that are installed will be displayed. They will be listed as fcsX, where X is 0, 1, etc.

2) To check the current microcode level for the adapter or controller enter the following command

lsmcode -cd fcsX <Where “X” is the instance of the adapter>

The command will produce output similar to:

The current microcode level for fcsX

is 00014000020062400005.

If the Microcode Level is less than 00014000020062400018, you should update the microcode.
 

 6.2 Linux Systems:

The following steps are for Linux Red Hat or SuSE systems running 4.18 or greater kernels which support the /sys filesystem. These steps assume you are logged in with root permissions and one or more Fibre Channel Adapters are installed.

1)     Display the model description for each installed Fibre Channel adapter. Expect two instances of PCIe4 x-Port xxGb Fibre Channel Adapters to be displayed: one for each port on the dual or quad port adapter. Type the following commands to display a list showing the host{n}: description for each installed adapter.

cd /sys/class/scsi_host; find -L host* -maxdepth 1 -name "model_desc" |xargs -i -t cat {}

The output will be like that shown here. The host{n} values will be used in the next step. If the list is empty, either there are no adapters installed or there was an error in typing the command. If the list contains other adapters but no IBM Fibre Channel adapters are listed, none are installed, and this firmware update should not be applied.

            cat host0

PCIe4 4-Port 32Gb Fibre Channel Adapter

cat host1

PCIe4 4-Port 32Gb Fibre Channel Adapter

cat host2

PCIe4 4-Port 32Gb Fibre Channel Adapter

cat host3

PCIe4 4-Port 32Gb Fibre Channel Adapter

or

cat host0

PCIe4 2-Port 64Gb Fibre Channel Adapter

cat host1

PCIe4 2-Port 64Gb Fibre Channel Adapter

 

2)     Display the firmware version for each adapter. Type the following command, replacing host{n} with the value reported for each adapter listed in Step 1.

# cat /sys/class/scsi_host/host{n}/fwrev

The microcode version listed may vary but output will be similar to that shown here:

14.2.624.05, sli-4:6:e

Repeat this command for each host{n}. If one or more adapters lists a microcode version that is not 14.2.624.18, then the microcode update should be applied.

3) Display Linux Operating System version. Type the following command to display the Operating System version. Make a note of Operating System version, it will be needed later.

For Red Hat systems, type the following command.

cat /etc/redhat-release

Output will be like: Red Hat Enterprise Linux Server release 9.3

For SuSE systems, type the following command:

# cat /etc/SuSE-release

Output will be similar to:

SUSE Linux Enterprise Server 15 (ppc64)
VERSION = 15
PATCHLEVEL = 4

 

7.0 Unpacking the RPM Format File to the Target Server

7.1 Unpacking the AIX RPM Package

Use this method to unpack to an AIX system:  

NOTE: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, and must be entered exactly as shown, including filenames.

1)     Make the directories on your AIX system to receive the RPM format file:

"mkdir -p /tmp/microcode/RPM”

2)     Transfer the RPM format file to the /tmp/microcode/RPM directory, change to that directory, and unpack the rpm file:

For PCIe4 4-Port 32G FC Adapter (CCIN: 2CFC with FC: EN1L and EN1M):

rpm -ihv --ignoreos df1000f51410c106.00014000020062400018.aix.rpm

For PCIe4 2-Port 64G FC Adapter (CCIN: 2CFD with FC: EN1N and EN1P):

rpm -ihv --ignoreos df1000f51410c206.00014000020062400018.aix.rpm

For AIX, the microcode file will be copied to the /etc/microcode directory which is a symbolic link to the /usr/lib/microcode directory.

File Names:  

For EN1L/EN1M:  df1000f51410c106.00014000020062400018
For EN1N/EN1P:   df1000f51410c206.00014000020062400018

NOTES:
If the permissions do not allow the copy to the above stated directories or files then the user will be prompted for a new location.

If a message is displayed saying the "package <package_name> is already installed", you will need to uninstall the rpm package, using the command line:

# rpm -e <package_name>

where <package_name> is the name of the package that was returned in the message. Return to Step 2 and attempt to unpack the file again.

 

7.2 Unpacking the Linux RPM Package

Follow these steps to install the Linux RPM package on a Linux system. Note that the Linux commands are case sensitive and must be entered exactly as shown, including file names.

1)     Transfer the Linux rpm file to a temporary directory on the target server such as /tmp.

2)     Change the current directory to the temporary directory and unpack the rpm file:                                                                                                      

For PCIe4 4-Port 32G FC Adapter (CCIN: 2CFC with FC: EN1L and EN1M):

rpm -ihv --ignoreos df1000f51410c106.00014000020062400018.grp.linux.rpm                                  

For PCIe4 2-Port 64G FC Adapter (CCIN: 2CFD with FC: EN1N and EN1P):

rpm -ihv --ignoreos df1000f51410c206.00014000020062400018.grp.linux.rpm

For Linux, the microcode file will be copied to the /lib/firmware directory.

File Names:  

For EN1L/EN1M:  df1000f51410c106.00014000020062400018.grp
For EN1N/EN1P:   df1000f51410c206.00014000020062400018.grp

NOTES:
If the permissions do not allow the copy to the above stated directories or files then the user will be prompted for a new location.

If a message is displayed saying the "package <package_name> is already installed", you will need to uninstall the rpm package, using the command line:

# rpm -e <package_name>

where <package_name> is the name of the package that was returned in the message. Return to Step 2 and attempt to unpack the file again.

 

8.0 Discovery Tool Microcode DVD-ROM creation and unpacking instructions

To obtain information how to burn a DVD-ROM and run the Discovery Tool for an AIX or Linux System please go to: http://www14.software.ibm.com/support/customercare/mds/

1)     After running the Discovery Tool successfully, the "/tmp/microcode/RPM" directory was created, and your rpm files are copied from the DVD-ROM.

2)     Change to that directory, "cd /tmp/microcode/RPM".

3)     Unpack the files by executing the instructions in section 7.0 above.

  

9.0 Verifying microcode before download and notes

9.1 Verifying microcode before download for AIX

Please verify the file size and checksum of the raw microcode files matches what is listed below.

For AIX:

Please verify the file size and checksum

For PCIe4 4-Port 32G FC Adapter (CCIN: 2CFC with FC: EN1L and EN1M):

ls -l /usr/lib/microcode/df1000f51410c106.00014000020062400018

sum /usr/lib/microcode/df1000f51410c106.00014000020062400018

For PCIe4 2-Port 64G FC Adapter (CCIN: 2CFD with FC: EN1N and EN1P):

ls -l /usr/lib/microcode/df1000f51410c206.00014000020062400018

sum /usr/lib/microcode/df1000f51410c206.00014000020062400018

 

Expected File Sizes: 2245312

Expected Checksums:  64710 2298

 

9.2 Verifying microcode before download for Linux

 Please verify the file location, size and checksum of the raw microcode files matches what is listed below for Linux:

Please verify file size and checksum

For PCIe4 4-Port 32G FC Adapter (CCIN: 2CFC with FC: EN1L and EN1M):

ls -l /lib/firmware/df1000f51410c106.00014000020062400018.grp

sum /lib/firmware/df1000f51410c106.00014000020062400018.grp

For PCIe4 2-Port 64G FC Adapter (CCIN: 2CFD with FC: EN1N and EN1P):

ls -l /lib/firmware/df1000f51410c206.00014000020062400018.grp

sum /lib/firmware/df1000f51410c206.00014000020062400018.grp

 

Expected File Size: 2245312

Expected Checksum:  64710 2298

 

10.0 Microcode Download Procedure for AIX: 

Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's).

10.1 Setting up for Microcode download

1) Stop all applications that use this interface/adapter

2) You only need to pick one of the fcs#s per adapter for microcode update.

Diag will update microcode for all ports.

 

10.2 Downloading Microcode to the Adapter

1) At the command line type “diag

2) Select the "Task Selection" from diagnostics menu.

3) Select "Microcode Tasks” then select “Download Microcode" from the menu.

4) Select all of the Fibre Channel Adapters that need to be updated from the list of devices

a)     Use the arrow keys to highlight the desired entry and mark it by pressing "Enter". 

b)     Press "F7" when you are done marking all the adapters you want to flash.

5) If a source selection menu is displayed, Select “/etc/microcode”.

6) A dialogue box will be displayed on screen. It will state the current microcode level and list the available levels.

7) Select the 00014000020062400018 level and press "Enter" to flash the adapter.

8) The following message will appear on the screen when download is completed:

"Installation of the microcode has completed successfully on the following instances of the adapter:… 

The current microcode level ……  

Please run diagnostics on these resources to ensure that the adapter is functioning properly." 

9) If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated.

10) Exit diagnostics.

 

10.3 Verify and Re-configure adapters

1) Verify the code level is 00014000020062400018 by typing "lsmcode -cd fcs#" for each Fiber Channel adapter updated. <Where # is the instance of the adapter>

2) Re-configure/Verify the adapter’s information through the “smit” menus if necessary.

  

11.0 Microcode Download Procedure for Linux using Emulex hbacmd utility

Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's).

11.1 Download One Command Manager

Microcode updates on Fibre Channel adapters installed in a Linux system require the use of the Emulex hbacmd utility. The hbacmd utility is included in the Emulex One Command CLI Applications Kit downloads web page located at

https://www.broadcom.com/support/download-search

·       Group: Storage Adapters, Controllers, and ICs

·       Family: Fibre Channel Host Bus Adapters

·       OEM: IBM

·       Product: All

·       Asset type: All

·       Keyword: PowerPC

·       Select “Management Software and Tools”

 

For purpose of updating the microcode, One Command Manager (CLI) will be used.   Select the Application Kit {version} (CLI) link. 

When prompted, save the Application Kit a directory of your choice on the target server.

Complete the requested user registration information, then select Download Driver button.

When prompted, save the elxocmcore-rhel8-rhel9-power-14.2.624.16-1.tgz file to a temporary directory, /tmp or you may save to directory of your choice.

Change directory to the directory where you have saved the elxocmcore-rhel8-rhel9-power-14.2.624.16-1.tgz archive.

Unpack the tgz archive, by typing:

# tar -xvzf elxocmcore-rhel8-rhel9-power-14.2.624.16-1.tgz

Change directory to installation directory and install the utility, by typing:

# ./install.sh

Answer questions as prompted using descriptive information provided as a guide.

Display the utility version to verify the hbacmd command is functional by typing:

# hbacmd version

Utility component version information should be displayed:

Ensure that all I/O activity to storage devices controlled by the adapter is stopped prior to proceeding with firmware update as the adapter will be reset after firmware update.

 

11.2  Download Code to Adapter

List installed Emulex adapters by typing: hbacmd listhbas

Output from this command will be similar to the following but specific to your device type. This information will be repeated one section for each discovered adapter. Adapters can be identified by looking for e300 in Flags value. For each adapter, make a note of the Port WWN: value. The Port WWN values will be required as an argument for commands in next steps.
 

Manageable HBA List

 

Port WWN       : 10:00:00:10:9b:fc:f9:a8

Node WWN       : 20:00:00:10:9b:fc:f9:a8

Fabric Name    : 00:00:00:00:00:00:00:00

Flags          : 8000f500

Host Name      : sanrain1e.aus.stglabs.ibm.com

Mfg            : Emulex Corporation

Serial No.     : Y050HY25R006

Port Number    : 1

Mode           : Initiator

PCI Bus Number : 112

PCI Function   : 0

Port Type      : FC+NVMe

Model          : 03HD016

 

Port WWN       : 10:00:00:10:9b:fc:f9:a9

Node WWN       : 20:00:00:10:9b:fc:f9:a9

Fabric Name    : 00:00:00:00:00:00:00:00

Flags          : 8000f500

Host Name      : sanrain1e.aus.stglabs.ibm.com

Mfg            : Emulex Corporation

Serial No.     : Y050HY25R006

Port Number    : 2

Mode           : Initiator

PCI Bus Number : 112

PCI Function   : 1

Port Type      : FC+NVMe

Model          : 03HD016

 

List hba attributes for each adapter listed in the above step, Type:

# hbacmd hbaattributes {wwpn}, where {wwpn} is one of the port WWPN values.

The output will look similar to the following and will list current versions of FW Version, SLI2 FW, SLI3 FW as well as Boot Code, all of which will be updated by microcode update. Save current operational firmware values for comparison post update.

# hbacmd hbaattributes 10:00:00:10:9b:fc:f9:a9

HBA Attributes for 10:00:00:10:9b:fc:f9:a9

 

Host Name                     : sanrain1e.aus.stglabs.ibm.com

Manufacturer                  : Emulex Corporation

Serial Number                 : Y050HY25R006

Model                         : 03HD016

Model Desc                    : IBM 03HD016 PCIe4 2-Port 64Gb Fibre Channel Adapter

Node WWN                      : 20 00 00 10 9b fc f9 a9

Node Symname                  :

HW Version                    : 0000000e 00000010 00000000

FW Version                    : 14.2.624.05

Vendor Spec ID                : 10DF

Number of Ports               : 1

Driver Name                   : lpfc

Driver Version                : 14.2.0.12; HBAAPI(I) v2.3.d, 07-12-10

Device ID                     : F500

HBA Type                      : 03HD016

Operational FW                : 14.2.624.05

IEEE Address                  : 00 10 9b fc f9 a9

Boot Code                     : Enabled

Boot Version                  : 14.2.579.0

Board Temperature             : Normal

Function Type                 : FC

Sub Device ID                 : 06C2

PCI Bus Number                : 112

PCI Func Number               : 1

Sub Vendor ID                 : 1014

IPL Filename                  : HR2FPN2

Service Processor FW Name     : 14.2.624.05

ULP FW Name                   : 14.2.624.05

FC Universal BIOS Version     : 14.2.579.0

FC x86 BIOS Version           : 14.2.566.0

FC EFI BIOS Version           : 14.2.579.0

FC FCODE Version              : 0x320

Flash Firmware Version        : 14.2.624.05

Secure Firmware               : Enabled

 

Update microcode on one port per adapter. This step assumes that microcode image .rpm package has been installed on system as described in section 8.2.

Warning: Do not interrupt or power off the system while microcode update(s) are in progress.

Start microcode update on one adapter port, type:

For EN1L/EN1M:

hbacmd  download {wwpn} /lib/firmware/df1000f51410c106.00014000020062400018.grp

Where {wwpn} is one of the port WWN values of adapter.

For EN1N/EN1P:

hbacmd  download {wwpn} /lib/firmware/df1000f51410c206.00014000020062400018.grp

 

Completion of download will be indicated by “Download Complete.” message.

System must be rebooted for firmware to be loaded.  Reboot system

After system reboot, check the firmware version as described in Section 7.2 or using: hbacmd  hbaattributes  {wwpn} as above for each port.

If, firmware version is not the version you are expecting, carefully retrace steps for omissions and examine console output for errors.  Contact IBM support if you need further assistance.

 

11.3 Feedback

Contact IBM Hardware Support at https://www.ibm.com/support to provide feedback regarding corrections/updates to the README file.