PCIe3 NVMe Flash_Adapter III (EC5B, EC5C, EC5D, EC5E, EC5F, EC5G) Microcode README IBM POWER Systems Non-Volatile Memory PCIe3 x8 1.6TB SSD NVMe III Adapter (CCIN 58FC) Non-Volatile Memory PCIe3 x8 3.2TB SSD NVMe III Adapter (CCIN 58FD) Non-Volatile Memory PCIe3 x8 6.4TB SSD NVMe III Adapter (CCIN 58FE) AIX & Linux Feature Codes:  EC5B, EC5C, EC5D, EC5E, EC5F, EC5G Microcode Update    DATE:  March 17, 2021   Table of Contents 1.0 Microcode Change History 2.0 General Information 2.1 Installation Time 2.2 Machines Affected 2.3 Operating System Requirements 3.0 Determine the Current Microcode Level for AIX & Linux 4.0 Downloading and Unpacking the Firmware Update Package 4.1 Downloading the RPM Format File to the Target Server 5.0 Verifying microcode before download and notes 5.1 Microcode Download Procedure-NVMe Adapters for AIX 5.2 Microcode Download Procedure-NVMe Adapters for Linux 6.0 Microcode Readme Change History ===================================================================================== 1.0 Microcode Change History:   4d413235 (MA25MA25) Impact:  Serviceability Severity:  ATT     4d413234 (MA24MA24) Impact:  Function Severity:  ATT   Support for IBM i 4d413233 (MA23MA23) Impact:    New Severity:  New   GA Level   2.0 General information This subject microcode is to be used on: PCIe3 1.6TB NVMe Flash Adapter III (CCIN 58FC), FC (Low Profile/High Profile): EC5G / EC5B PCIe3 3.2TB NVMe Flash Adapter III (CCIN 58FD), FC (Low Profile/High Profile): EC5C / EC5D PCIe3 6.4TB NVMe Flash Adapter III (CCIN 58FE), FC (Low Profile/High Profile): EC5E / EC5F 2.1 Installation Time: Approximately 30 minutes   2.2 Machine's Affected: 8335-GTX, 8335-GTH, 9009-22A, 9008-22L, 9009-41G, 9009-42G, 9009-22G, 9040-MR9, 9080-M9S, 9080-M98, 9222-80H See also: Universal Sales Manual or other relevant documents for current listings of supported systems for these adapters   2.3 Operating System Requirements: If you are installing a new feature, ensure that you have the software that is required to support the new feature and that you determine whether any prerequisites must be met for this feature.  Consult the IBM sales manual for your system model to determine the AIX, VIOS, or Linux release level requisites to support the feature in the system.  To check for the feature prerequisites, see the IBM Prerequisite website ( www.ibm.com/support/pages/service-and-productivity-tools).    3.0 Determine the Current Microcode Level for AIX & Linux: These steps assume you are logged in with root permissions and a PCIe3 NVMe Flash III Adapter, Feature Code EC5B, EC5C, EC5D, EC5E, EC5F, or EC5G is installed.   Prerequisites for AIX: There is no additional prerequisite software needed beyond the supported level of AIX.   Prerequisites for Linux: Use the IBM Power Systems service and productivity tools. www.ibm.com/support/pages/service-and-productivity-tools (https://www.ibm.com/support/pages/service-and-productivity-tools)   1. Download the IBM Power Systems service and productivity tools   2. Copy the file to the target system and install the RPM # rpm -ivh ibm-power-repo-latest.noarch.rpm   3. To add the Tools Repository to the package management configuration, run the following command as root user (or with root privilege) and accept the Licensing agreement. # /opt/ibm/lop/configure   Instructions for installing nvme-cli tool using Red Hat Enterprise Linux   To list the currently enabled repositories, enter: # yum repolist enabled   Install the nvme-cli package, enter: # yum install nvme-cli   Checking NVME device firmware for AIX: To list NVMe devices on your system and view firmware level,   1) Type:  ls /dev | grep nvm      nvme0      nvme1   2)  List details of the nvme device(s) by:        Type:  lscfg -vl nvme#                 # lscfg -vl nvme0   nvme0         U78D2.001.WZS000E-P1-C7  PCIe3 x8 NVMe Flash Adapter         PCIe3 1.6TB NVMe Flash Adapter III x8:         Part Number.................02DE947         EC Level....................P46697         Feature Code/Marketing ID...EC5B         FRU Number..................02DE944         Customer Card ID Number.....58FC         Serial Number...............ADYBA01V         Product Specific.(Z0).......1014064A         Product Specific.(Z1).......5         Product Specific.(Z2).......3200         Product Specific.(Z3).......14.6         Product Specific.(Z4).......0         Product Specific.(Z5).......0         Product Specific.(Z6).......A1800106         Product Specific.(Z7).......0YA         Product Specific.(Z8)....... 3020         Product Specific.(Z9).......08         Product Specific.(ZA).......08         Product Specific.(ZB).......40         Manufacture ID..............S46HNY0K900075         Product Specific.(RV).......F         ROM Level.(alterable).......MA24MA24         Hardware Location Code......U78D2.001.WZS000E-P1-C7   The firmware level is listed in the field:  ROM Level (alterable).   If the Firmware Version is less than MA25MA25 then you should update the microcode.     Checking NVMe device firmware on Linux: After nvme-cli is installed on the system, check the current microcode level for the adapter enter the following command: # nvme list   Note: "nvme list" command uses libudev-devel, if it doesn't work, may need to install ?libudev-devel"   If the FW Rev is different from ?MA25MA25? install this update. 1) Type "nvme list" 2) The column ?FW Rev? will list the firmware version.  For example:   Node             SN                  Model                                    Namespace        Usage                    Format             FW Rev  ---------------- ---------------     --------------------------------------   ------------     ----------------------   ----------------   ---------------- /dev/nvme0n1     xxxxxxxxxxxxxx      PCIe3 1.6TB NVMe Flash Adapter III x8     1                1.60  TB /   1.60  TB    4 KiB +  8 B       MA25MA25   If the Firmware Version is less than MA25MA25 then you should update the microcode. 4.0 Downloading and Unpacking the Firmware Update Package NOTE:  For RHEL, the image will be the RPM format file.   1. Download feature code EC5B, EC5C, EC5D, EC5E, EC5F, or EC5G package from FixCentral, and transfer the file, to a temporary directory on the target server such as /tmp.   2. Install the package   (AIX) rpm ?ivh pci.1014064a.MA24MA25-4d413235-1.aix6.1.noarch.rpm   (Linux) rpm ?ivh pci.1014064a.MA24MA25-4d413235-1.Linux.noarch.rpm   3. The microcode package extracts the firmware file to the following directory: ·       Linux:  /lib/firmware ·       AIX:    /etc/microcode   4. Verify the microcode image in Section 5.0 5.0 Verifying microcode before download and notes NOTE: Microcode download must be performed separately for each Adapter bus under distinct Logical Partitions (LPAR's).               Please save current and older versions of the microcode update files in case you need to restore your system. NOTE: Microcode Download CAN be performed concurrently. Installation Time:  Approximately 30 min System Time. Please verify the file size and checksum of the raw microcode files matches the below. Filesize:  3674112 Checksum:  52903   For AIX:    " ls -l /etc/microcode/A1800106.4d413235" to verify Filesize.    "sum /etc/microcode/A1800106.4d413235" to verify Checksum For Linux:    "ls -l /lib/firmware/pci.1014064a.MA25MA25.4d413235" to verify Filesize.    "sum /lib/firmware/pci.1014064a.MA25MA25.4d413235" to verify Checksum. For systems that have LPAR installed, the microcode file must be placed on all partitions that have a NVMe device in order to update the code.  Follow this procedure for each partition: For AIX:     A)  Log into the target partition.     B)  Type: "cd /etc/microcode"     C)  SFTP to the partition that has the media bay assigned. (ie.  "sftp partition1.ibm.com")     D)  Type: "cd /etc/microcode"     E)  Type: "get A1800106.4d413235"     F)  Type: "quit" For Linux:     A)  Log into the target partition.     B)  Type: "cd /lib/firmware"     C)  SFTP to the partition that has the media bay assigned. (ie.  "sftp partition1.ibm.com")     D)  Type: "cd /lib/firmware"     E)  Type: "get pci.1014064a.MA25MA25.4d413235"     F)  Type: "quit" 5.1 Microcode Download Procedure-NVMe Adapters for AIX   Procedure: 1) Stop all applications. 2) Type "ls /dev | grep nvme" and press "Enter".  This will list all of the NVMe adapters in your partition 3) To display details for the nvme device (including current firmware level),      type ?lscfg -vl nvme#?.        For example, for device nvme0, type ?lscfg -vl nvme0?   4) For each NVMe adapter to be updated, issue the following commands:      a)  Type ?diag?      b)  Select ?Task Selection?      c)  Select ?Microcode Tasks?      d)  Select ?Download Microcode?      e)  Select the NVMe device to have microcode updated on.             For example, select:  nvme0  and press F7      f)  Select ?/etc/microcode? as location of updated firmware image.  Press ENTER      g) For level to install, select ?4d413235?.            The system will take several minutes to update the firmware.            Once completed, the screen will display the following message:     INSTALL MICROCODE                                                                                              nvme0    PCIe3 x8 NVMe Flash Adapter   Installation of the microcode has completed successfully on the following instances of the adapter:   nvme0   The current microcode level is 4d413235.   Please run diagnostics on these resources to ensure that the adapter is functioning properly.     Use Enter to continue.     5.2 Microcode Download Procedure-NVMe Adapters for Linux Procedure: 1) Stop all applications. 2) Type "nvme list" and press "Enter".  This will list all of the NVMe adapters in your partition.  In the Node column, the NVMe adapters will be listed as/dev/nvme#n1 where # is the number of the device. 3) For each NVMe adapter to be update, issue the following commands: a)     Type ?nvme fw-log /dev/nvme<#>? and press ?Enter?, where <#> is the number of the device.    For example, for device /dev/nvme0n1, type ?nvme fw-log /dev/nvme0? and press ?Enter?.  This will list the firmware level currently loaded on the adapter as well as the number of slots in the adapter.  These 3 slots are listed as frs1, frs2, and frs3.   b)    Download & activate firmware to the adapter by typing:   ?nvme fw-download /dev/nvme# --fw=/lib/firmware/pci.1014064a.MA25MA25.4d413235? and press ?Enter?, where # is the number of the device.                                i.         nvme fw-download /dev/nvme# --fw=/lib/firmware/pci.1014064a.MA25MA25.4d413235 and press ?Enter?, where # is the number of the device.   For example, for device /dev/nvme0, type ?nvme fw-download /dev/nvme0 ?fw=/lib/firmware/pci.1014064a.MA25MA25.4d413235? and press ?Enter?.                          ii.         ?nvme fw-activate /dev/nvme# -a 0 ?s 1? where # is the number of the device                      iii.         nvme fw-download /dev/nvme# --fw=/lib/firmware/pci.1014064a.MA25MA25.4d413235   (Repeat of step i)                       iv.         ?nvme fw-activate /dev/nvme# -a 0 ?s 2? where # is the number of the device                        v.         nvme fw-download /dev/nvme# --fw=/lib/firmware/pci.1014064a.MA25MA25.4d413235   (Repeat of step i)                       vi.         ?nvme fw-activate /dev/nvme# -a 0 ?s 3? where # is the number of the device                     vii.         ?nvme fw-activate /dev/nvme# -a 3 ?s 3? where # is the number of the device                   viii.          ?nvme reset /dev/nvme#? where # is the number of the device   For example, to download & activate the firmware on NVMe device /dev/nvme0, type:                            i.         nvme fw-download /dev/nvme0 --fw=/lib/firmware/pci.1014064a.MA25MA25.4d413235                        ii.         nvme fw-activate /dev/nvme0 -a 0 ?s 1                      iii.         nvme fw-download /dev/nvme0 --fw=/lib/firmware/pci.1014064a.MA25MA25.4d413235                       iv.         nvme fw-activate /dev/nvme0 -a 0 ?s 2                        v.         nvme fw-download /dev/nvme0 --fw=/lib/firmware/pci.1014064a.MA25MA25.4d413235                       vi.         nvme fw-activate /dev/nvme0 -a 0 ?s 3                     vii.         nvme fw-activate /dev/nvme0 -a 3 ?s 3                   viii.          nvme reset /dev/nvme0               6.0 Microcode Readme Change History 2021-03-17 Latest fixes for low likelihood issues found since MA24MA24 2019-10-02 IBM i Support 2019-05-15 GA microcode level 4d413233 (MA23MA23)