PCIe3 4-Port 10GbE SFP+ CU Adapter ******* PLEASE READ THIS ENTIRE NOTICE *********   UPDATED DATE: October 20th, 2020 Table of Contents Contents 1.0 Microcode and Document Revision History: 2.0 General information 3.0 Installation time 4.0 Machine's Affected: 5.0 Linux Requirements: 6.0 AIX Requirements: 7.0 Determine the Current Microcode Level for AIX: 7.1 AIX 7.2 Linux 8.0 Downloading the RPM Format File to the Target Server 8.1 AIX 8.2 Linux 9.0 Discovery Tool Microcode CD-ROM creation and download instructions 10.0 Verifying microcode before download and notes 10.1 AIX 10.2 Linux 11.0 Microcode Download Procedure for AIX: 11.1 Setting up for Microcode download 11.2 Downloading Microcode to the Adapter 11.3 Verify and Re-configure adapters 12.0 Microcode Download Procedure for Linux: 12.1 Setting up for Microcode download 12.2 Downloading Microcode to the Adapter 1.0 Microcode and Document Revision History:   Microcode Level Changes 00011000040041500033 Impact: Data Severity: HIPER Fix for the following: * Fix for a condition that may result from frequent resets of adapter Virtual Functions or transmission stalls and could lead to potential undetected data corruption The Firmware Levels Below Are No Longer Supported By IBM Once They Have Been Removed From The Microcode Down Load Website. It is best practices to update to the latest FW level not only for IBM support of these products, but for optimal performance and to ensure that all of the required HW/FW fixes are installed. Once new FW has been released to the field, we will provide a 6 month grace period for customers to update these products to the currently supported FW level. Please update to the latest level at your earliest convenience. 00011000040041500026 Impact: Usability Severity: ATT Fix for the following: * Fix for Power8-exclusive issue where SRIOV enablement causes adapter to become 16Gb generic fiberchannel adapter. 00011000040041500016 Impact: Usability Severity: ATT Fix for the following: * Change mailbox tracking timeout to 25 secs for fast commands doing flash operations * Release FFS read lock during non critical sections of garbage collection * Copyright information showing incorrect for Feature Code EN15/EN16/EN17/EN18 boot code * This level of firmware supports the Firmware Secure Boot feature of Power systems * For open and unused ports experiencing LNC2ENT_HW_TMP_ERRs. This fix is partial and requires IBM APAR IJ10293 for full correction. 00011000020021100021 Impact: Usability Severity: ATT Fix for the following: * adapter has an intermittent issue where reading NIC fcode returns 00 * FCode memory leak in fibre channel driver * adapter with FW 11.1.35.67 failing NIM install * Direct Attach LinkUp Issue with DS8K array. FEC set to off on both sides to work. * Add data scrubbing to adapter FW dump process to remove/obscure user data * Customer reporting spontaneous dump on FCoE adapter running 10.2.252.1921 fw * FC adapter intermittently aborts active I/Os after Accepted REC * VIOS crashed when lpar was booted into Sles 00010000020025201927 Impact: Usability Severity: ATT *TX stall causing back-pressure in the ASIC resulting in mailbox response not being sent to host *System crash when LPAR was booted into SLES *Hardware Ethernet Error caused by a dead lock in adapter firmware. A mutex was taken and not released by mistake. *Adapter enters firmware loop when single bit ECC error is detected. System OS this condition as an adapter command time out. This condition will be reported as a temporary adapter hardware failure. 00010000020025201913 Fixed an intermittent issue Link not coming up after reboot with Cisco switch. 00010000020025201905 *Fix for SRIOV mode, adapter went to error state when config Vnic. *Fix for SRIOV VF TX timeouts when other VFs are FLR – reseted. *Fix for SRIOV Vnic ping failure when injecting EEH errors. *Fix for spontaneous livedump with adapter entered error state (9F000013,00001007) *Fix a rogue frame with EoFa and a bad length, seen during switch reboots. *Fix an inbound class 2 FLOGI rejected with non-retryable invalid class of service 00010000020025200016 This is the level of microcode that initially shipped with the adapter when it became generally available. Document Revision History Description 06/01/2015 Creating Readme file with latest microcode for this adapter - df1020e21410e404 level 00010000020025200016 01/07/2016 Updating Readme for level 00010000020025201905 06/17/2016 Updating Readme for level 00010000020025201913 06/19/2017 Updating Readme for level 00010000020025201927 03/30/2018 Updating Readme for level 00011000020021100021 12/04/2018 Updating Readme for level 00011000040041500016 02/13/2019 Updating Machine Type-Model list in 4.0 Machines Affected 06/17/2019 Updating Readme for level 00011000040041500026 10/20/2020 Updating Readme for level 00011000040041500033 2.0 General information This Readme file is intended to give directions on how to update the microcode found on the PCIe3 4-Port 10GbE CU Adapter (FC: EN17 and EN18). 3.0 Installation time Approximately 20 minutes. NOTE: It is recommended that the installation be scheduled during a maintenance window or during non-peak production periods. 4.0 Machine's Affected:  Feature Code EN17 ( Full Height ), EN18 ( Low Profile ) CCIN 2CE4 EN17: 8247-21L, 8247-22L, 8247-42L, 8284-22A, 8286-41A, 8286-42A, 8408-44E, 8408-E8E, 9008-22L, 9009-22A, 9009-41A, 9040-MR9, 9080-M9S, 9080-MHE, 9080-MME, 9119-MHE, 9119-MME, 9223-22H, 9223-42H EN18: 9080-M9S, 9080-MHE, 9080-MME, 9119-MHE, 9119-MME 5.0 Linux Requirements: * Red Hat Enterprise Linux Version 7.1, or later * Red Hat Enterprise Linux Version 6.6, or later * SUSE Linux Enterprise Server Version 12, or later * Linux Ubuntu 15.04, or later * The latest version of the device driver or iprutils can be downloaded from IBM Service and Productivity Tools website. 6.0 AIX Requirements: Basic card function is supported on : * AIX 6.1 with the 6100-09 Technology Level and Service Pack 5 and APAR IV68443, or later * AIX 7.1 with the 7100-03 Technology Level and Service Pack 5 and APAR IV68444 * AIX 7.1 with the 7100-02 Technology Level and Service Pack 7, or later (planned availability 9/30/2015) * AIX 6.1 with the 6100-08 Technology Level and Service Pack 7, or later (planned availability 9/30/2015) If you are using another release of AIX, ensure that the adapter is supported on that release before you install the adapter. Contact service and support for assistance.   7.0 Determine the Current Microcode Level for AIX: 7.1 AIX Before you install the microcode, it is important to determine the microcode level of the Adapter installed in the target system. Use the following instructions to read the ROM level stored in the Adapter's VPD. 1)  List the Ethernet adapters installed in the system by typing: lscfg -l ent\* Note: All of the Ethernet adapter ports that are installed will be displayed. They will be listed as entX, where X is 0, 1, etc. 2) To check the current microcode level for the adapter or controller enter the following command lsmcode -cd entx Where “x” is the instance of the adapter. The command will produce output similar to: The current microcode level for ent4 is 00011000040041500033. If the ROM Level is less than 00011000040041500033 you should update the microcode. 7.2 Linux NOTE: The instructions that follow are specific Linux commands. Linux commands are CASE (lower and upper) SENSITIVE, and must be entered exactly as shown, including filenames. 1) List the Ethernet interfaces for PCIe3 4-Port 10GbE SFP+ Adapter adapters installed in the system by typing: ifconfig -a | grep eth Note: All of the Ethernet adapters that are installed will be displayed. They will be listed as eth{n}, where {n} is 0, 1, etc. 2) For each adapter listed, use ethtool -i eth{n} to determine which Ethernet interfaces are associated with the PCIe3 4-Port 10GbE SFP+ Adapter. If driver name is “be2net”, that interface is for PCIe3 4-Port 10GbE SFP+ Adapter. For example: ethtool -i eth{n} Where {n} is the instance of the adapter. The command will produce output similar to: ethtool -i eth1 driver: be2net version: 2.102.426r firmware version: 11.4.415.33 bus-info: 0001:01:00.0 If driver name is “be2net”, eth1 is one of PCIe3 4-Port 10GbE SFP+ adapter interfaces. firmware-version is “11.4.415.33”. If firmware-version is not 11.4.415.33 or higher, on any of the adapters listed with a be2net driver; you should update the microcode 8.0 Downloading the RPM Format File to the Target Server 8.1 AIX Use this method to download to an AIX system: NOTE: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE, and must be entered exactly as shown, including filenames. 1) Make two directories on your AIX system to receive the RPM format file.     Enter:      "mkdir /tmp/microcode"    and then create this directory     Enter:      "mkdir /tmp/microcode/RPM" 2) Transfer the RPM format file to the /tmp/microcode/RPM directory (using "Save as ...").  Change to that directory, "cd /tmp/microcode/RPM". You'll see the filename for the RPM file.        "rpm -ihv --ignoreos df1020e21410e404.00011000040041500033.aix.rpm" 3) For AIX:  The microcode files will be added to /etc/microcode/.    NOTE:  - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".  - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location. 8.2 Linux Use this method to download the new microcode to a Linux system: NOTE: The instructions that follow are specific Linux commands. Linux commands are CASE (lower and upper) SENSITIVE, and must be entered exactly as shown, including filenames. 1) Transfer the RPM format file to the /tmp directory(using “Save as....”). You will see the filename for the RPM file. 2) Install rpm on your Linux system by typing: rpm -ivh /tmp/ df1020e21410e404.00011000040041500033.Linux.rpm 3) The microcode package will install the firmware images in the /lib/firmware directory. If a message is displayed saying the "package is already installed", you will need to uninstall the listed rpm package. On the command line type: rpm -e where is the name of the package that was returned in the message. Return to Step 2 and attempt to install the file again. 4) The microcode file will be installed to the /lib/firmware directory. File name?df1020e21410e404.00011000040041500033 9.0 Discovery Tool Microcode CD-ROM creation and download instructions To obtain information how to burn a CD-ROM and run the Discovery Tool for an AIX or Linux System please go to: http://www14.software.ibm.com/support/customercare/mds/ 1) After running the Discovery Tool successfully the "/tmp/microcode/RPM" directory was created and your rpm files are copied from the CD-ROM. 2) Change to that directory, "cd /tmp/microcode/RPM". 3) Unpack the file by executing the instructions below:     Enter the commands:       rpm -ihv --ignoreos df1020e21410e404.00011000040041500033.aix.rpm 4) One microcode file will be copied to "/etc/microcode".  File Name: df1020e21410e404.00011000040041500033 NOTE:  - "/etc/microcode" is a symbolic link to "/usr/lib/microcode".  - If permission does not allow the copy to the above stated directory or file then the user will be prompted for a new location. 10.0 Verifying microcode before download and notes 10.1 AIX Please verify the file size and checksum of the raw microcode files matches what is listed below. For AIX : Please verify the file size and checksum # ls -l df1020e21410e404.00011000040041500033 -rw-r--r-- 1 root root 10970832 Sep 10 16:49 df1020e21410e404.00011000040041500033 # sum df1020e21410e404.00011000040041500033 34419 10714 df1020e21410e404.00011000040041500033 10.2 Linux ls -l /lib/firmware/df1020e21410e404.00011000040041500033 to verify file size is 10970832 sum /lib/firmware/df1020e21410e404.00011000040041500033 to verify Checksum is 34419 11.0 Microcode Download Procedure for AIX:  Microcode download must be performed separately for each Adapter under each Logical Partitions (LPAR's). 11.1 Setting up for Microcode download 1) Stop all applications that use this interface/adapter. 2) Detach the adapter’s interface (not always needed, but it is a good practice to do this step) Before detaching the interface, record the ip address and any other pertinent information that was configured on the Adapter. This information may be needed if the microcode update overwrites this section on the Adapter. Important Note: If NIC ports are used in SEA Shared Ethernet Adapter, or EtherChannel. Detach and rmdev are needed: ifconfig en# detach rmdev -Rl ent# This adapter can concurrently download firmware onto Etherchannel devices, but it is always recommended to take these precautions when updating firmware. 3) You only need to pick one ent# per adapter for microcode update. Diag will update microcode for all ports. Note: in step 3) you might need to perform “rmdev -Rl en# “ to remove child devices to the ent#. Ent# needs to be Available for fw download. 11.2 Downloading Microcode to the Adapter 1) At the command line type “diag” 2) Select the "Task Selection" from diagnostics menu. 3) Select "Microcode Tasks” then select “Download Microcode" from the menu. 4) Select all of the PCIe3 4-Port 10GbE SFP+ Adapters that need to be updated from the list of devices by using the arrow keys to highlight the entry and pressing "Enter" to mark it.  The devices will be displayed as ent#, where # is the number of the device.  Press "F7" when you are done marking all the adapters you want to flash. 5) If a source selection menu is displayed, Select “/etc/microcode”. 6) A dialogue box may be displayed on screen.  It will state that the current microcode level on the adapter is not in the /etc/microcode directory.  This is acceptable because the adapter will reject any incorrect code.  Press "Enter" to continue. 7) Select 00011000040041500033 level and press "Enter" to flash the adapter. 8) The following message will appear on the screen when download is completed: "Microcode download complete successfully.  The current microcode level ...  Please run diagnostics on the adapter to ensure that it is functioning properly."  9) If you selected more than one adapter to update, then steps 6-9 will repeat until all adapters are updated. 10) Exit diagnostics. 11.3 Verify and Re-configure adapters 1) Verify the code level is 00011000040041500033 by typing "lsmcode -cd ent#" or “lsmcode -cd ent#” for each Ethernet adapter updated. 2) Re-configure/Verify the adapter’s information through the “smit” menus if necessary. # lscfg -vl ent4 ent4 U78C9.001.RST0081-P1-C5-T1 PCIe3 4-Port 10GbE SFP+ Adapter (df1020e21410e404) PCIe3 4-port 10GbE SFP+ Adapter, NIC PF: Part Number.................00ND464 Serial Number...............YL10A04450BB EC Level....................N98703 Feature Code/Marketing ID...EN18 Network Address.............0090FA88D5DA ROM Level.(alterable).......00011000040041500033 Hardware Location Code......U78C9.001.RST0081-P1-C5-T1 12.0 Microcode Download Procedure for Linux: 12.1 Setting up for Microcode download 1) Use this series of Linux commands to determine which of any installed Network adapters are PCIe3 4-Port 10GbE SFP+ Adapter: Note: It is advisable to cut and paste these command strings into your system console, to avoid typographical errors. for i in `ls /sys/class/net | grep eth` ; do echo -n "${i}: "; ethtool -i ${i} | grep bus-info; done Output will display the association between eth{n} devices and PCI bus device numbers, nnnn:nn:nn.n eth0: bus-info: eth1: bus-info: 0000:01:00.0 eth2: bus-info: 0000:01:00.1 eth3: bus-info: 0000:01:00.2 eth4: bus-info: 0000:01:00.3 lspci -d 10df:e220 Output will list PCI bus device numbers for the PCIe3 4-Port 10GbE SFP+ Adapter functions, nn:nn.n 01:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30) 01:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30) 01:00.2 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30) 01:00.3 Ethernet controller: Emulex Corporation OneConnect NIC (Lancer) (rev 30) This example shows the four ports of one PCIe3 4-Port 10GbE SFP+ Adapter. Firmware need only be updated on the first port of each adapter, eth1 in this case. If more than 4 ports are shown, it is likely you have more than one adapter, so you would need to also update firmware on first port of second adapter. 2) Stop all applications that use the interface/adapter to be updated 3) Configure the adapter’s interface, eth1 in this example. ifconfig eth1 192.168.1.1/24 4) Check that no errors were reported. If errors were reported verify proper interface was configured and command was entered correctly. 12.2 Downloading Microcode to the Adapter 1) Using the eth{n} configured in previous step use ethtool --flash to update the firmware. ethtool --flash eth{n} df1020e21410e404.00011000040041500033 Note: eth{n} is the interface previously configured with ifconfig (See 12.1 Setting up for Microcode downloading). 2) If flashing is completed successfully, no messages will appear on the screen. You can check messages in system log. dmesg | grep be2net For example, dmesg command will show the following messages: be2net 0000:01:00.0: Flashing firmware file df1020e21410e404.00011000040041500033 be2net 0000:01:00.0: System reboot required for new FW to be active be2net 0000:01:00.0: Firmware flashed successfully These messages show flashing is completed successfully. 3) Reboot the OS instance to activate the updated firmware. 4) Verify firmware has been activated Using same eth{n} interface used for ethtool --flash command, type: ethtool -i eth1 driver: be2net version: 2.102.426r firmware-version: 11.4.415.33 bus-info: 0000:01:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: no supports-priv-flags: no 5) Firmware-version must show 11.4.415.33. If firmware-version is not as expected, carefully retrace steps to confirm no steps were omitted or performed incorrectly. If no errors or omissions were found, contact IBM customer service for assistance.