PCIe3 2-Port 32Gb FC Adapter PCIe3 4-Port 16Gb FC Adapter PCIe3 2-Port 16Gb FC Adapter ******* PLEASE READ THIS ENTIRE NOTICE *********   DATE: July 29, 2024 Table of Contents 1.0 Microcode and Document Revision History: 2.0 General information 3.0 Installation time 4.0 Machine's Affected: 5.0 Cautions and Important Notes 6.0 Determine the Current Microcode Level 6.1 AIX Systems: 6.2 Linux Systems: 7.0 Downloading the RPM Format File to the Target Server 7.1 Downloading AIX RPM Package 7.2 Downloading Linux RPM Package 8.0 Discovery Tool Microcode CD-ROM creation and download instructions 9.0 Verifying microcode before download and notes 9.1 Verifying microcode before download for AIX 9.2 Verifying microcode before download for Linux 10.0 Microcode Download Procedure for AIX: 10.1 Setting up for Microcode download 10.2 Downloading Microcode to the Adapter 10.3 Verify and Re-configure adapters 11.0 Microcode Download Procedure for Linux using Emulex hbacmd utility 11.1 Download One Command Manager 11.2 Download Code to Adapter     1.0 Microcode and Document Revision History:   Microcode Level Changes 00014000020062400010 Impact: Usability   Severity: ATT * Link fails on adapter initialization * Update FCODE to v320 (adapter boot code) * N_Port login command rejected with invalid RPI (NPIV) The Firmware Levels Below Are No Longer Supported by IBM Once They Have Been Removed from The Microcode Download Website. It is best practices to update to the latest FW level not only for IBM support of these products, but for optimal performance and to ensure that all of the required HW/FW fixes are installed. Please Update to The Latest Level At Your Earliest Convenience 00014000020062400006 Impact: Usability   Severity: ATT * Direct attach support for DS8K * Added support for new operating system releases. 00014000000057400007 Impact: Usability   Severity: ATT * Fix double delete under heavy load * Added support for new operating system releases. * Added support for new fibre channel adapters 00012000040025700030 Impact: Usability   Severity: ATT * Open-dev leaving extra item on stack resulting in failed boot * Fcp-targets did not return target ports for IBMi direct attach resulting in failed boot 00012000040025700027 Impact: Usability   Severity: ATT * Add NVMf support in openBoot for CCIN 578F * Fix fcode stack imbalance 00012000040025700015 Impact: Usability   Severity: ATT *Fixed Memory Address Not Aligned issue caused by flogi retry loop. Initial partition boot may fail with one of the following reference codes: BA210001, BA218001, BA210003, or BA218003. *This level of firmware supports the Firmware Secure Boot feature of Power systems   00012000040025700012     Impact: Usability   Severity: ATT Fixes for the following: *FFS READ/WRITE mutex fix to address POST watchdog & Adapter initialization POST Failure *Provide interlock between FLOGI and link up ACQE, error fix. *Release FFS read lock during non-critical sections of garbage collection *This level of firmware supports the Firmware Secure Boot feature of Power systems 00011000040041500005 Impact: Usability   Severity: ATT Fixes for the following: *Scanning adapter in SMS when direct attached to V7000 takes over 2 minutes *Flogi failure seen on the initiator port and change in WWPN after changing ports on Brocade switch *Firmware 32-bit address crossing *Update SGLE HWLOG option to include QR *Periodic EEH errors New Features: *This level of firmware supports the Firmware Secure Boot feature of Power systems *Improved lock performance 00011000020021100010 This is the level of microcode that initially shipped with the adapter when it became generally available.   Document Revision History Description 07/15/2024 Update readme for 14.2.624.10 firmware release 11/27/2023 Release 14.2 firmware. Update readme for 14.2.624.6 firmware release 12/7/2022 Release 14.0 firmware. Updated readme for 14.0.574.7 firmware release 1/21/2021 Updated readme file for 12.4.257.30 firmware release. Consolidated readme files into one for multiple adapters that share firmware image 11/17/2020 Updated readme file for 12.4.257.27 firmware release 2/10/2020 Updated readme file for 12.4.257.15 firmware release 1/15/2020 Updated readme file for 12.4.257.12 firmware release 11/13/2018 Created initial readme file for first microcode  2.0 General information This Readme file is intended to give directions on how to update the microcode found on the following adapters: PCIe3 2-Port 32G FC Adapter (CCIN: 578F with FC: EN1A/EN1B and EL5U/EL5V) * df1000e314101506 PCIe3 4-Port 16G FC Adapter (CCIN: 578E with FC: EN1C/EN1D and EL5W/EL5X) * df1000e314101406 PCIe3 2-Port 16G FC Adapter (CCIN: 579D with FC: EN2A and EN2B) * df1000e31410a006 3.0 Installation time Approximately 20 minutes.   4.0 Machine's Affected:  IBM Power   5.0 Cautions and Important Notes  ·    Concurrent Download The microcode installation supports concurrent download while the adapter and attached storage devices are available for use. HOWEVER, it is highly recommended that installation be scheduled during a maintenance window or during non-peak production periods. ·    AIX Diag and Standalone Diag CD For FW 11.4.415.05 or higher, AIX APAR IJ08893 is required before running AIX Diagnostics. Please also use the latest AIX Standalone Diagnostics CD found here https://www.ibm.com/support/pages/node/6474531 Important Notice:  * EEH error entries in AIX system errlog are expected when updating adapter FW. * Quiesce loads on adapter doing updates or long timeouts may result.    6.0 Determine the Current Microcode Level  6.1 AIX Systems: Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the adapter's VPD. 1)  List the Fiber Channel adapters in the system by typing: lscfg -l fcs* Note: All of the Fiber Channel and Ethernet adapter ports that are installed will be displayed. They will be listed as fcsX and entX, where X is 0, 1, etc. 2) To check the current microcode level for the adapter or controller enter the following command # lsmcode -cd fcsX The command will produce output similar to: The current microcode level for fcsX is 00014000020062400006. If the ROM Level is less than 00014000020062400010, you should update the microcode.    6.2 Linux Systems: The following steps are for Linux Red Hat or SuSE systems which support the /sys filesystem. These steps assume you are logged in with root permissions and one or more Fibre Channel Adapters are installed. 1) Display the model description for each installed Fibre Channel adapter. Expect two instances of PCIe3 x-Port xxGb Fibre Channel Adapters to be displayed: one for each port on the dual or quad port adapters. Type the following commands to display a list showing the host{n}: description for each installed adapter. cd /sys/class/scsi_host; find -L host* -maxdepth 1 -name "modeldesc" -printf %h: -exec cat {} \; The output will be similar to that shown here. The host{n} values will be used in the next step. If the list is empty, either there are no adapters installed or there was an error in typing the command. If the list contains other adapters but no IBM Fibre Channel adapters are listed, none are installed, and this firmware update should not be applied. host1:IBM xxxP/Nxxx PCIe3 x-Port xxGb Fibre Channel Adapter host2:IBM xxxP/Nxxx PCIe3 x-Port xxGb Fibre Channel Adapter 2) Display the firmware version for each adapter. Type the following command, replacing host{n} with the value reported for each adapter listed in Step 1. cat /sys/class/scsi_host/host{n}/fwrev The microcode version listed may vary but output will be similar to that shown here, four numbers separated with “.”, sli-4:6:e: 14.2.624.6, sli-4:6:e Repeat this command for each host{n}. If one or more adapters lists a microcode version that is not 14.2.624.10, then the microcode update should be applied. 3) Display Linux Operating System version. Type the following command to display the Operating System version. Make a note of Operating System version, it will be needed later. For Red Hat systems, type the following command. # cat /etc/redhat-release Output will be like: Red Hat Enterprise Linux release 9.3 For SuSE systems, type the following command: cat /etc/SuSE-release Output will be like: SUSE Linux Enterprise Server 15 (ppc64) VERSION = 15 PATCHLEVEL = 4   7.0 Downloading the RPM Format File to the Target Server 7.1 Downloading AIX RPM Package Use this method to download to an AIX system:   NOTE: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, and must be entered exactly as shown, including filenames. 1) Make the directory on your AIX system to receive the RPM format file: "mkdir -p /tmp/microcode/RPM” 2) Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as ...").  Change to that directory, "cd /tmp/microcode/RPM". For PCIe3 2-Port 32G FC Adapter (CCIN: 578F with FC: EN1A/EN1B and EL5U/EL5V): rpm -ihv --ignoreos df1000e314101506.00014000020062400010.aix.rpm PCIe3 4-Port 16G FC Adapter (CCIN: 578E with FC: EN1C/EN1D and EL5W/EL5X) rpm -ihv --ignoreos df1000e314101406.00014000020062400010.aix.rpm PCIe3 2-Port 16G FC Adapter (CCIN: 579D with FC: EN2A/EN2B) rpm -ihv --ignoreos df1000e31410a006.00014000020062400010.aix.rpm 3) For AIX:  The microcode files will be added to /etc/microcode/.  NOTE:  - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".  - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location.   7.2 Downloading Linux RPM Package Follow these steps to install the Linux RPM package on a Linux system. Note that the Linux commands are case sensitive and must be entered exactly as shown, including file names. 1) Transfer the Linux rpm file to a temporary directory on the target server such as /tmp. 2) Change the current directory to the temporary directory 3) Unpack the rpm file: For PCIe3 2-Port 32G FC Adapter (CCIN: 578F with FC: EN1A/EN1B and EL5U/EL5V): rpm -ihv --ignoreos df1000e314101506.00014000020062400010.aix.rpm PCIe3 4-Port 16G FC Adapter (CCIN: 578E with FC: EN1C/EN1D and EL5W/EL5X) rpm -ihv --ignoreos df1000e314101406.00014000020062400010.aix.rpm PCIe3 2-Port 16G FC Adapter (CCIN: 579D with FC: EN2A/EN2B) rpm -ihv --ignoreos df1000e31410a006.00014000020062400010.aix.rpm The microcode package extracts the firmware file to the /lib/firmware directory. If a message is displayed saying the "package is already installed", you will need to uninstall the rpm package. On the command line type: rpm -e where is the name of the package that was returned in the message. Return to Step 2 and attempt to unpack the file again. 4) Verify the microcode image in Section 10.2   8.0 Discovery Tool Microcode CD-ROM creation and download instructions To obtain information how to burn a CD-ROM and run the Discovery Tool for an AIX or Linux System please go to: http://www14.software.ibm.com/support/customercare/mds/ 1) After running the Discovery Tool successfully, the "/tmp/microcode/RPM" directory was created, and your rpm files are copied from the CD-ROM. 2) Change to that directory, "cd /tmp/microcode/RPM". 3) Unpack the file by executing the instructions below:     Enter the commands: For PCIe3 2-Port 32G FC Adapter (CCIN: 578F with FC: EN1A/EN1B and EL5U/EL5V): rpm -ihv --ignoreos df1000e314101506.00014000020062400010.aix.rpm PCIe3 4-Port 16G FC Adapter (CCIN: 578E with FC: EN1C/EN1D and EL5W/EL5X) rpm -ihv --ignoreos df1000e314101406.00014000020062400010.aix.rpm PCIe3 2-Port 16G FC Adapter (CCIN: 579D with FC: EN2A/EN2B) rpm -ihv --ignoreos df1000e31410a006.00014000020062400010.aix.rpm 5) A microcode file will be copied to "/etc/microcode". For EN1A/EN1B: df1000e314101506.00014000020062400010 For EN1C/EN1D: df1000e314101406.00014000020062400010 For EN2A/EN2B: df1000e31410a006.00014000020062400010 NOTES:  - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".  - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location.    9.0 Verifying microcode before download and notes 9.1 Verifying microcode before download for AIX Please verify the file size and checksum of the raw microcode files matches what is listed below. For AIX: Please verify the file size and checksum For PCIe3 2-Port 32G FC Adapter (CCIN: 578F with FC: EN1A/EN1B and EL5U/EL5V): ls -l /usr/lib/microcode/df1000e314101506.00014000020062400010 sum /usr/lib/microcode/df1000e314101506.00014000020062400010 For PCIe3 4-Port 16G FC Adapter (CCIN: 578E with FC: EN1C/EN1D and EL5W/EL5X) ls -l /usr/lib/microcode/df1000e314101406.00014000020062400010.aix.rpm summ /usr/lib/microcode/df1000e314101406.00014000020062400010.aix.rpm For PCIe3 2-Port 16G FC Adapter (CCIN: 579D with FC: EN2A/EN2B) ls -l /usr/lib/microcode/df1000e31410a006.00014000020062400010 sum -l /usr/lib/microcode/df1000e31410a006.00014000020062400010 Expected file size: 3441124 Expected checksum: 41041 3361   9.2 Verifying microcode before download for Linux  Please verify the file location, size and checksum of the raw microcode files matches what is listed below for Linux: Please verify file size and checksum For PCIe3 2-Port 32G FC Adapter (CCIN: 578F with FC: EN1A/EN1B and EL5U/EL5V): ls -l /lib/firmware/df1000e314101506.00014000020062400010.grp sum /lib/firmware/df1000e314101506.00014000020062400010.grp For PCIe3 4-Port 16G FC Adapter (CCIN: 578E with FC: EN1C/EN1D and EL5W/EL5X) ls -l /lib/firmware/df1000e314101406.00014000020062400010.aix.rpm summ /lib/firmware/df1000e314101406.00014000020062400010.aix.rpm For PCIe3 2-Port 16G FC Adapter (CCIN: 579D with FC: EN2A/EN2B) ls -l /lib/firmware/df1000e31410a006.00014000020062400010 sum -l /lib/firmware/df1000e31410a006.00014000020062400010 Expected file size: 3441124 Expected checksum: 41041 3361 10.0 Microcode Download Procedure for AIX:  Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's). 10.1 Setting up for Microcode download 1) Stop all applications that use this interface/adapter 2) You only need to pick one of the fcs#s per adapter for microcode update. Diag will update microcode for all ports. 10.2 Downloading Microcode to the Adapter 1) At the command line type “diag” 2) Select the "Task Selection" from diagnostics menu. 3) Select "Microcode Tasks” then select “Download Microcode" from the menu. 4) Select all of the Fibre Channel Adapters that need to be updated from the list of devices by using the arrow keys to highlight the entry and pressing "Enter" to mark it.  The devices will be displayed as fcs#, where # is the number of the device.  Press "F7" when you are done marking all the adapters you want to flash. 5) If a source selection menu is displayed, Select “/etc/microcode”. 6) A dialogue box will be displayed on screen. It will state the current microcode level and list the available levels. 7) Select the 00014000020062400010 level and press "Enter" to flash the adapter. 8) The following message will appear on the screen when download is completed: "Installation of the microcode has completed successfully on the following instances of the adapter:…  The current microcode level ……   Please run diagnostics on these resources to ensure that the adapter is functioning properly."  9) If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated. 10) Exit diagnostics. 10.3 Verify and Re-configure adapters 1) Verify the code level is 00014000020062400010 by typing "lsmcode -cd fcs#" for each Fiber Channel adapter updated. 2) Re-configure/Verify the adapter’s information through the “smit” menus if necessary.    11.0 Microcode Download Procedure for Linux using Emulex hbacmd utility Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's). 11.1 Download One Command Manager Microcode updates on Fibre Channel adapters installed in a Linux system require the use of the Emulex hbacmd utility. The hbacmd utility is included in the Emulex One Command CLI Applications Kit downloads web page located at https://www.broadcom.com/support/download-search * Group: Storage Adapters, Controllers, and ICs * Family: Fibre Channel Host Bus Adapters * OEM: IBM * Product: All * Asset type: All * Keyword: PowerPC * Select “Management Software and Tools”   For purpose of updating the microcode, One Command Manager (CLI) will be used.   Select the Application Kit {version} (CLI) link.  When prompted, save the Application Kit a directory of your choice on the target server. Complete the requested user registration information, then select Download Driver button. When prompted, save the elxcmcore-xxxx-xxxx-x.x.x.x-x.tgz file to a temporary directory, /tmp or you may save to directory of your choice. Change directory to the directory where you have saved the elxcmcore-xxxx-xxxx-x.x.x.x-x.tar archive. Unpack the tgz archive, by typing: # tar -xvf elxcmcore-xxxx-xxxx-x.x.x.x-x.tar Change directory to elxocmcore-xxxx-xxxx-x.x.x.x-x and install the utility, by typing: # ./install.sh Answer questions as prompted using descriptive information provided as a guide. Display the utility version to verify the hbacmd command is functional by typing: # /opt/emulex/ocmanager/bin/hbacmd version Utility component version information should be displayed Ensure that all I/O activity to storage devices controlled by the adapter is stopped prior to proceeding with firmware update as the adapter will be reset after firmware update.   11.2 Download Code to Adapter List installed Emulex adapters by typing: /opt/emulex/ocmanager/bin/hbacmd listhbas Output from this command will be similar to the following but specific to your device type. This information will be repeated one section for each discovered adapter. Adapters can be identified by looking for e300 in Flags value. For each adapter, make a note of the Port WWN: value. The Port WWN values will be required as an argument for commands in next steps.   Manageable HBA List Port WWN                  : 10:00:00:10:9b:17:08:5f Node WWN                : 20:00:00:10:9b:17:08:5f Fabric Name               : 10:00:c4:f5:7c:cd:ef:a0 Flags                           : 8000e300 Host Name                  : system.ibm.com Mfg                             : Emulex Corporation Serial No.                    : 1H6CR0Y049 Port Number               : 0 Mode                           : Initiator PCI Bus Number        : 1 PCI Function              : 0 Port Type                   : FC+NVMe Model                         : xxxP/Nxxx   List hba attributes for each adapter listed in the above step, Type: # /opt/emulex/ocmanager/bin/hbacmd hbaattributes {wwpn}, where {wwpn} is one of the port WWPN values. The output will look similar to the following and will list current versions of FW Version, SLI2 FW, SLI3 FW as well as Boot Code, all of which will be updated by microcode update. Save current operational firmware values for comparison post update. # /opt/emulex/ocmanager/bin/hbacmd hbaattributes 10:00:00:10:9b:17:08:cb HBA Attributes for 10:00:00:10:9b:17:08:cb Host Name                              : system.ibm.com Manufacturer                          : Emulex Corporation Serial Number                         : 1H6CR0Y049 Model                                      : xxxP/Nxxx Model Desc                             : IBM xxxP/Nxxx PCIe3 x-Port xxGb Fibre Channel Adapter Node WWN                            : 20 00 00 10 9b 17 08 5f Node Symname                      : HW Version : 0000000c 00000001 00000000 FW Version : 12.4.257.30 Vendor Spec ID : 10DF Number of Ports : 1 Driver Name : lpfc Driver Version : 14.2.0.5; HBAAPI(I) v2.3.d, 07-12-10 Device ID                               : E300 HBA Type                              : xxxP/Nxxx Operational FW : 14.2.264.6 IEEE Address : 00 10 9b 17 08 5f Boot Code : Enabled Boot Version : 14.2.264.6 Board Temperature : Normal Function Type : FC Sub Device ID                        : xxxPCI-IDxxx PCI Bus Number                    : 96 PCI Func Number                   : 0 Sub Vendor ID                        : 1014 IPL Filename                          : H62LP1 PCI Bus Number : 1 PCI Func Number : 0 Sub Vendor ID : 1014 IPL Filename : H62LP1 Service Processor FW Name : 14.2.264.6 ULP FW Name : 14.2.264.6 FC Universal BIOS Version : 14.2.264.6 FC x86 BIOS Version : 14.2.264.6 FC EFI BIOS Version : 14.2.264.6 FC FCODE Version : 14.2.264.6 Flash Firmware Version : 14.2.264.6   Update microcode on one port per adapter. This step assumes that microcode image .rpm package has been installed on system as described in section 8.2. Warning: Do not interrupt or power off the system while microcode update(s) are in progress. Start microcode update on one adapter port, type: For PCIe3 2-Port 32G FC Adapter (CCIN: 578F with FC: EN1A/EN1B and EL5U/EL5V): hbacmd  download {wwpn} /lib/firmware/df1000e314101506.00014000020062400010.grp For PCIe3 4-Port 16G FC Adapter (CCIN: 578E with FC: EN1C/EN1D and EL5W/EL5X) hbacmd  download {wwpn} /lib/firmware/df1000e314101406.00014000020062400010.grp For PCIe3 2-Port 16G FC Adapter (CCIN: 579D with FC: EN2A/EN2B) hbacmd  download {wwpn} /lib/firmware/df1000e31410a006.00014000020062400010.grp Where {wwpn} is one of the port WWN values of adapter. Completion of download will be indicated by “Download Complete.” message. System must be rebooted for firmware to be loaded.  Reboot system After system reboot, check the firmware version as described in Section 7.2 or using: /opt/emulex/ocmanager/bin/hbacmd  hbaattributes  {wwpn} as above for each port. If, firmware version is not the version you are expecting, carefully retrace steps for omissions and examine console output for errors.  Contact IBM support if you need further assistance. 11.3 Feedback Contact IBM Hardware Support at https://www.ibm.com/support to provide feedback regarding corrections/updates to the README file.