IBM Microcode Update Readme File   GX++ 2-port 16Gb Fibre Channel Adapter (FC: EN23)   ******* PLEASE READ THIS ENTIRE NOTICE *********  DATE: May 23, 2018 Table of Contents   1.0 Microcode and Document Revision History: 2.0 General information 3.0 Installation time 4.0 Machine's Affected: 5.0 Linux Requirements:  6.0 AIX Requirements: 7.0 Determine the Current Microcode Level 7.1 AIX Systems: 7.2 Linux Systems: 8.0 Downloading and unpacking the Microcode Package 8.1 Downloading AIX RPM Package 8.2 Downloading Linux RPM Package 9.0 Discovery Tool Microcode CD-ROM creation and download instructions for AIX systems 10.0 Verifying microcode before download and notes for AIX 10.1 Verifying microcode before download for AIX 10.2 Verifying microcode before download for Linux 11.0 Microcode Download Procedure for AIX: 11.1 Setting up for Microcode download 11.2 Downloading Microcode to the Adapter 11.3 Verify and Re-configure adapters 12.0 Microcode Download Procedure for Linux using Emulex hbacmd utility 12.1 Download One Command Manager 12.2 Download Code to Adapter     1.0          Microcode and Document Revision History:     Microcode Level Changes 00011000020021100021   Impact: Usability   Severity: ATT   * adapter has an intermittant issue where reading NIC fcode returns 00 * FCode memory leak in fibre channel driver * adapter with FW 11.1.35.67 failing NIM install * Direct Attach LinkUp Issue with DS8K array. FEC set to off on both sides to work. * Add data scrubbing to adapter FW dump process to remove/obscure user data * Customer reporting spontaneous dump on FCoE adapter running 10.2.252.1921 fw * FC adapter intermittently aborts active I/Os after Accepted REC * VIOS crashed when lpar was booted into Sles The Firmware Levels Below Are No Longer Supported By IBM Once They Have Been Removed From The Microcode Down Load Website. It is best practices to update to the latest FW level not only for IBM support of these products, but for optimal performance and to ensure that all of the required HW/FW fixes are installed. Please Update To The Latest Level At Your Earliest Convenience 00010000020025201919 * Fixed an intermittent issue of Link not coming up after reboot with Cisco switch. * Fixed FC Tape class 3 error recovery * Fixed performance scaling when using IBM Flash system 900 00010000020025201905 *Fix for SRIOV mode, adapter went to error state when config Vnic. *Fix for SRIOV VF TX timeouts when other VFs are FLR ? reseted. *Fix for SRIOV  Vnic ping failure when injecting EEH errors. *Fix for spontaneous livedump with adapter entered error state (9F000013,00001007) *Fix a rogue frame with EoFa and a bad length, seen during switch reboots. *Fix an inbound class 2 FLOGI rejected with non-retryable invalid class of service   00010000020025200016 *Fix for SRIOV race conditions in PCI Function Reset *Fix for Outbound Jammer test (Dropped outbound data frame of single frame sequence) *Fix IO fail to resume after f/w update on FCoE *Fix DMA to 0 in an IBMi system *Fix for SRIOV hard LPAR shutdowns 00010000020025200009 * FC Adapter Port Error on ISL pull * P_RJT response for Fibre Channel Class 2 ELS's * Reboot FCoE adapter path recovery failure - mailbox error and fcs unknown error * FC adapter does not indicate closed exchange when WQE with LNK bit set fails (fw ignores early closure of exchange  for linked scsi commands) * Experiencing NIM install issues with adapters                              * DCBX Get Config Mailbox Cmd timing out. Fix Link bounces   * Unable to set pause Flow control when connected to DCBX switch with PFC off * GRP5 Link COS ACQE needs to be delivered on DCBX/PFC change * AIX issues BA_RJT with reason code "Protocol Error" to target-Initiated ABTS 00001000010005800004   - Exchanges aborted after REC is successfully ACC'd by target device. - COMMON_GET_FLOW_CONTROL issues in dedicated mode - No link after cable plugged in - VIOS disk path does not recover from Initiator port cable pull - Adapter pport stat counter TX_BROADCAST_BYTES incorrect - Logging in and out of fabric   00001000010003400005 1) Adapter livedump following CRC forced errors. 2) fcode fix of SAN boot LPAR stuck at LED  AA00E158 3) No link after cable plugged in. 4) Failed paths to SAN after a switch reboot.   00001000010003400003 This is the level of microcode that initially shipped with the adapter when it became generally available.     Document Revision History Description 01/18/2013 Creating Readme file with latest microcode for this adapter- df1000e214105e04 05/27/2013 Update readme file with level  00001000010003400005 05/28/2014 Update readme file with level  00001000010005800004 12/05/2014 Update readme file with level  00010000020025200009 07/20/2015 Update readme file with level  00010000020025200016 02/22/2016 Update readme file with level  00010000020025201905 09/12/2016 Update readme file with level  00010000020025201919 05/23/2018 Update readme file with level  00011000020021100021   2.0 General information This Readme file is intended to give directions on how to update the microcode found on the GX++ 2-port 16GbFibre Channel Adapter (FC: EN23).   3.0 Installation time   Approximately 20 minutes.   4.0 Machine's Affected:  Feature Code  EN23: 9119-FHB 5.0 Linux Requirements:   Basic card function is supported on: ?  Red Hat Enterprise Linux 6.4 for POWER® , or later ?  SUSE Linux Enterprise Server 11 Service Pack 2, or later.  Download drivers link:   https://www.broadcom.com/support/download       6.0 AIX Requirements: Basic card function is supported on: * AIX Version 7.1 with the 7100-02 Technology Level, or later * AIX Version 7.1 with the 7100-01 Technology Level and Service Pack 6, or later * AIX Version 7.1 with the 7100-00 Technology Level and Service Pack 8, or later * AIX Version 6.1 with the 6100-08 Technology Level, or later * AIX Version 6.1 with the 6100-07 Technology Level and Service Pack 6, or later * AIX Version 6.1 with the 6100-06 Technology Level and Service Pack 10, or later If you are using another release of AIX, ensure that the adapter is supported on that release before you install the adapter. Contact service and support for assistance.       Important Notice:  EEH error entries in AIX system errlog are expected when updating adapter FW.       7.0 Determine the Current Microcode Level  7.1 AIX Systems: Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the Adapter's VPD. 1)  List the Fiber Channel adapters installed in the system by typing:     lscfg -l fcs\* Note: All of the Fiber Channel adapters that are installed will be displayed. They will be listed asfcsX, where X is 0, 1, etc. 2) To check the current microcode level for the adapter or controller enter the following command    lsmcode -cd fcsx Where ?x? is the instance of the adapter. The command will produce output similar to: The current microcode level for fcs0 is  00011000020021100021 If the ROM Level is less than  00011000020021100021 you should update the microcode. 7.2 Linux Systems: The following steps are for Linux Red Hat or SuSE systems running 2.6 kernels which support the /sys filesystem. These steps assume you are logged in with root permissions and one or more IBM 2 port 16GbFibre Channel GX++ adapters, Feature Code EN23 are installed. 1) Display the model description for each installed Fibre Channel adapter. Expect two instances of IBM 2 port 16GbFibre Channel GX++ adapters to be displayed; one for each port on the dual port adapter. Type the following commands to display a list showing thehost{n}:description for each installed adapter.  cd  /sys/class/scsi_host  find -L host* -maxdepth 1 -name "modeldesc" -printf %h: -exec cat {} \; The output will be similar to that shown here. The host{n} values will be used in the next step. If the list is empty, either there are no Feature Code EN23 adapters installed or there was an error in typing the command. If the list contains other adapters but no FC EN23 adapters are listed, none are installed and this firmware update should not be applied. host1:IBM 2 port 16Gb Fibre Channel GX++ adapter with SW optics host2:IBM 2 port 16Gb Fibre Channel GX++ adapter with SW optics 2) Display the firmware version for each Feature Code EN23 adapter. Type the following command, replacing host{n} with the value reported for each FC EN23 adapter listed in Step 1. cat host{n}/fwrev The microcode version listed may vary but output will be similar to that shown here, four numbers separated with ?.?, sli-4:2:b: 11.2.211.21 , sli-4:2:b Repeat this command for each adapter  host{n}. If one or more adapter lists a microcode version that is not11.2.211.21, then the microcode update should be applied. 3) Display Emulex Fibre Channel adapter driver version. Type the following command to display the adapter driver version. modinfo -F version lpfc Only the portion following the ':' is important. Make a note of driver version as it will be needed later. For example: 0:8.3.5.86.1p 4) Display Linux Operating System version. Type the following command to display the Operating System version. Make a note of Operating System version, it will be needed later. For Red Hat systems, type following command. cat  /etc/redhat-release Output will be similar to: Red Hat Enterprise Linux Server release 6.4 (Santiago) For Novell systems, type following command. Output will be similar to: cat  /etc/SuSE-release SUSE Linux Enterprise Server 11 (ppc64) VERSION = 11 PATCHLEVEL = 2 8.0 Downloading and unpacking the Microcode Package 8.1 Downloading AIX RPM Package Use this method to download to an AIX system: NOTE: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, andmust be entered exactly as shown, including filenames. 1) Make two directories on your AIX system to receive the RPM format file.     Enter:      "mkdir /tmp/microcode"    and then create this directory     Enter:      "mkdir /tmp/microcode/RPM" 2) Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as ...").  Change to that directory, "cd /tmp/microcode/RPM". You'll see the filename for the RPM file.        "rpm -ihv --ignoreos df1000e214105e04.00011000020021100021.aix.rpm"                                                                                                 3) For AIX:  The microcode files will be added to /etc/microcode/.    NOTE:  - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".  - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location. 8.2 Downloading Linux RPM Package Follow these steps to install the Linux RPM package on a Linux system. Note that the Linux commands are case sensitive and must be entered exactly as shown, including file names. 1) Transfer the Linux rpm file, df1000e214105e04.00011000020021100021 .grp.Linux.rpm, to a temporary directory on the target server such as /tmp. 2) Unpack the rpm file by typing: rpm -ihv /tmp/df1000e214105e04.00011000020021100021.grp.Linux.rpm 3) The microcode package extract the firmware file to the /lib/firmware directory. If a message is displayed saying the "package is already installed", you will need to uninstall the rpm package. On the command line type: rpm -e where is the name of the package that was returned in the message. Return to Step 2 and attempt to unpack the file again. 4) Verify the microcode image in Section 10.2   9.0 Discovery Tool Microcode CD-ROM creation and download instructions for AIX systems To obtain information how to burn a CD-ROM and run the Discovery Tool for an AIX or Linux System please go to: http://www14.software.ibm.com/support/customercare/mds/ 1) After running the Discovery Tool successfully the "/tmp/microcode/RPM" directory was created and your rpm files are copied from the CD-ROM. 2) Change to that directory, "cd /tmp/microcode/RPM". 3) Unpack the file by executing the instructions below:     Enter the commands:       rpm -ihv --ignoreos df1000e214105e04.00011000020021100021.aix.rpm 4) One microcode file will be copied to "/etc/microcode".  The file size and checksum of the microcode image will be verified in Section 9.0.                        FileName:            df1000e214105e04.00011000020021100021 NOTE:  - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".  - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location.   10.0 Verifying microcode before download and notes   10.1 Verifying microcode before download for AIX Please verify the file size and checksum of the raw microcode files matches what is listed below. For AIX : Please verify the file size and checksum # ls -l df1000e214105e04.00011000020021100021 -rw-r--r--    1 root     system     10846192 Apr 03 22:25 df1000e214105e04.0 0011000020021100021   # sum df1000e214105e04.00011000020021100021 43623 10592  df1000e214105e04.00011000020021100021   10.2 Verifying microcode before download for Linux   Please verify the file location, size and checksum of the raw microcode files matches what is listed below.   Verify file size: ls -l /lib/firmware/df1000e214105e04.00011000020021100021.grp Expected file size: 10846192   Verify checksum: sum  /lib/firmware/df1000e214105e04.00011000020021100021.grp Expected checksum: 43623 10592   11.0 Microcode Download Procedure for AIX:  Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's). 11.1 Setting up for Microcode download 1) Stop all applications that use this interface/adapter 2) You only need to pick one of the fcs# per adapter for microcode update. Diag will update microcode for all ports.                                  11.2 Downloading Microcode to the Adapter 1) At the command line type ?diag? 2) Select the "Task Selection" from diagnostics menu. 3) Select "Microcode Tasks? then select ?Download Microcode" from the menu. 4) Select all of the GX++ 2-Port 16Gb FC Adapter that need to be updated from the list of devices by using the arrow keys to highlight the entry and pressing "Enter" to mark it.  The devices will be displayed as fcs#, where # is the number of the device.  Press "F7" when you are done marking all the adapters you want to flash. 5) If a source selection menu is displayed, Select ?/etc/microcode?. 6) A dialogue box may be displayed on screen.  It will state that the current microcode level on the adapter is not in the /etc/microcode directory.  This is acceptable because the adapter will reject any incorrect code.  Press "Enter" to continue. 7) Select  00011000020021100021 level and press "Enter" to flash the adapter. 8) The following message will appear on the screen when download is completed: "Microcode download complete successfully.  The current microcode level for the device fcs# is ...  Please run diagnostics on the adapter to ensure that it is functioning properly."  9) If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated. 10) Exit diagnostics. 11.3 Verify and Re-configure adapters 1) Verify the code level is 00011000020021100021  by typing "lsmcode-cd fcs#" for each Fiber Channel adapter updated. 2) Re-configure/Verify the adapter?s information through the ?smit? menus if necessary. # lscfg -vl fcs0   fcs0             U78A2.001.DQD0004-P6-C39-T1  GX++ 2-Port 16Gb FC Adapter (df1000e214105e04)           Part Number.................73Y8774         EC Level....................N44581B         Customer Card ID Number.....2B9B         FRU Number..................73Y8776         Device Specific.(ZM)........3         Network Address.............1000005076000164         Device Specific.(Z0)........0000000B         Device Specific.(Z1)........00000010         Device Specific.(Z2)........00000000         Device Specific.(Z3)........08090000         Device Specific.(Z4)........01000001         Device Specific.(Z5)........2E323532         Device Specific.(Z6)........2E323532         Device Specific.(Z7)........C0022B40         Device Specific.(Z8)........2000005076000164         Device Specific.(Z9)........11.2.211.21         Device Specific.(ZA)........11.2.211.21         Device Specific.(ZB)........00000000         Device Specific.(ZC)........00040000         Device Specific.(ZD)........000000FF         Hardware Location Code......U78A2.001.DQD0004-P6-C39-T1         12.0 Microcode Download Procedure for Linux using Emulex hbacmd utility  Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's). 12.1 Download One Command Manager Microcode updates on Fibre Channel adapters installed in a Linux system require the use of the Emulexhbacmd utility. The hbacmd utility is included in the Emulex One Command CLI Applications Kit downloads web page located at :   https://www.broadcom.com/support/download-search (https://urldefense.proofpoint.com/v2/url?u=https-3A__www.broadcom.com_support_download-2Dsearch&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=BjB6y9VuLiMqCivDwI6_PA&m=F5kTsbiiTzptg3YL3m7FEkZAjyYewvkGKsBW_g9wMY4&s=gBXpHsC9RFCU-ZZuDpbPLoJT7sVv8EjdIPDX-sOhunQ&e=)   Group: Storage Adapters, Controllers, and ICs Family: Fibre Channel Host Bus Adapters OEM:  Product: All Asset type: Management Software and Tools  Keyword: PowerPC     For purpose of updating the microcode, One Command Manager (CLI) will be used.   Select the Application Kit {version} (CLI) link.  When prompted, save the Application Kit a directory of your choice on the target server. Complete the requested user registration information, then select Download Driver button.       When prompted, save the elxcmcore-xxxx-xxxx-x.x.x.x-x.tgz file to a temporary directory, /tmp or you may save to directory of your choice. Change directory to the directory where you have saved the elxcmcore-xxxx-xxxx-x.x.x.x-x.tgz archive. Unpack the tgz archive, by typing: tar xzf elxcmcore-xxxx-xxxx-x.x.x.x-x.tgz Change directory to elxocmcore-xxxx-xxxx-x.x.x.x-x and install the utility, by typing: ./install.sh Answer questions as prompted using descriptive information provided as a guide. Display the utility version to verify the hbacmd command is functional by typing: /usr/sbin/ocmanager/hbacmd version Utility component version information should be displayed Ensure that all I/O activity to storage devices controlled by the adapter is stopped prior to proceeding with firmware update as the adapter will be reset after firmware update. 12.2           Download Code to Adapter   List installed Emulex adapters by typing: /usr/sbin/hbacmd  listhbas Output from this command will be similar to the following, repeated one section for each discovered adapter. Feature Code EN23 adapters can be identified by looking fore200 in Flags value. For each adapter, make a note of the Port WWN: value. The Port WWN values will be required as an argument for commands in next steps. Note: model may be shown as 41T8694 in this output.   Port WWN       : 10:00:34:40:b5:da:0c:80 Node WWN     : 20:00:34:40:b5:da:0c:80 Fabric Name    : 10:00:00:05:1e:89:a0:e9 Flags                : 8000e200 Host Name      : mpvhejemp2 Mfg                 : Emulex Corporation Serial No.       : 3440b5da0c80 Port Number  : 0 Mode             : Initiator PCI Bus Number : 1 PCI Function  : 0 Port Type      : FC Model           : 41T8694   List hba attributes for each Feature Code EN23 adapter listed in Step 1, Type /usr/sbin/hbacmd  hbaattributes  {wwpn}, where {wwpn} is one of the port WWPN values listed in Step 1. Output will look similar to the following and will list current versions of FW Version, SLI2 FW, SLI3 FW as well as Boot Code, all of which will be updated by microcode update. Save current operational firmware values for comparison post update. HBA Attributes for 10:00:34:40:b5:da:0c:80   Host Name                     : mpvhejemp2 Manufacturer                  : Emulex Corporation Serial Number                 : 3440b5da0c80 Model                              : 41T8694 Model Desc                    : IBM  2 port 16Gb Fibre Channel GX++ adapter with SW optics Node WWN                   : 20 00 34 40 b5 da 0c 80 Node Symname             : Emulex 41T8694 FV1.1.34.5 DV8.3.7.7.1p HW Version                   : 0000000b FW Version                    : 11.2.211.21 Vendor Spec ID              : 10DF Number of Ports             : 1 Driver Name                   : lpfc Device ID                        : E200 HBA Type                       : 41T8694 Operational FW              : 11.2.211.21 IEEE Address                  : 34 40 b5 da 0c 80 Boot Code                       : Enabled Boot Version                  : KT8.02a7 Driver Version                : 8.3.7.7.1p; HBAAPI(I) v2.3.b, 07-12-10 Board Temperature         : Normal Function Type                 : FC Sub Device ID                 : 045E PCI Bus Number              : 1 PCI Func Number            : 0 Sub Vendor ID                 : 1014 Service Processor FW Name    : 11.2.211.21 ULP FW Name                          : 11.2.211.21 FC Universal BIOS Version  : KT8.02a7 FC x86 BIOS Version           : KA6.01a11 FC EFI BIOS Version           : KD6.01a11 FC FCODE Version              : KN4.02a14 Flash Firmware Version       : 11.2.211.21   Update microcode on one port per adapter. This step assumes that microcode image .rpm package has been installed on system as described in section 8.2. Warning: Do not interrupt or power off the system while microcode update(s) are in progress. Start microcode update on one adapter port, type: /usr/sbin/hbacmd  download {wwpn} /lib/firmware/df1000e214105e04.0 0011000020021100021.grp   , where {wwpn} is one of the port WWN values of adapter.  Completion of download will be indicated by ?Download Complete.? message.  System must be rebooted for firmware to be loaded.  Reboot system After system reboot, check firmware version as described in Section 7.2 or using /usr/sbin/hbacmd  hbaattributes  {wwpn} as above for each port. If, firmware version is not the version you are expecting, carefully retrace steps for omissions and examine console output for errors.  Contact IBM support if you need further assistance.