IBM POWER Systems Non-Volatile Memory PCIe4 x8 1.6TB SSD NVMe Adapter (CCIN 594A) Non-Volatile Memory PCIe4 x8 3.2TB SSD NVMe Adapter (CCIN 594B) Non-Volatile Memory PCIe4 x8 6.4TB SSD NVMe Adapter (CCIN 594C) AIX & Linux Feature Codes: EC7A, EC7B, EC7C, EC7D, EC7E, EC7F Microcode Update   DATE: November 17, 2023 Table of Contents 1.0 Microcode Change History 2.0 General Information 2.1 Installation Time 2.2 Machines Affected 2.3 Operating System Requirements 3.0 Determine the Current Microcode Level for AIX & Linux 4.0 Downloading and Unpacking the Firmware Update Package 4.1 Downloading the RPM Format File to the Target Server 5.0 Verifying microcode before download and notes 5.1 Microcode Download Procedure-NVMe Adapters for AIX 5.2 Microcode Download Procedure-NVMe Adapters for Linux 6.0 Microcode Readme Change History ===================================================================================== 1.0 Microcode Change History:   53503439 (SP49) Impact: SVC Severity: ATT Fix timing Issue between Host Command Request and updating temperature in device 53503437 (SP47) Impact: New Severity: New GA Level   2.0 General information This subject microcode is to be used on: PCIe4 1.6TB NVMe Flash Adapter (CCIN 594A), FC (Low Profile/High Profile): EC7A / EC7B PCIe4 3.2TB NVMe Flash Adapter (CCIN 594B), FC (Low Profile/High Profile): EC7C / EC7D PCIe4 6.4TB NVMe Flash Adapter (CCIN 594C), FC (Low Profile/High Profile): EC7E / EC7F 2.1 Installation Time: Approximately 30 minutes   2.2 Machine's Affected: 9009-22A, 9009-22L, 9009-41A, 9009-42A, 9223-22H, 9223-42H, 9009-22G, 9009-41G, 9009-42G, 9040-MR9, 9080-M9S See also: Universal Sales Manual or other relevant documents for current listings of supported systems for these adapters   2.3 Operating System Requirements: If you are installing a new feature, ensure that you have the software that is required to support the new feature and that you determine whether any prerequisites must be met for this feature. Consult the IBM sales manual for your system model to determine the AIX, VIOS, or Linux release level requisites to support the feature in the system. To check for the feature prerequisites, see the IBM Prerequisite website (https://www14.software.ibm.com/support/customercare/iprt/home).  3.0 Determine the Current Microcode Level for AIX & Linux: These steps assume you are logged in with root permissions and a PCIe4 NVMe Flash Adapter, Feature Code EC7A, EC7B, EC7C, EC7D, EC7E, or EC7F is installed.   Prerequisites for AIX: There is no additional prerequisite software needed beyond the supported level of AIX. Prerequisites for Linux: Use the ibm-power-repo to configure the repositories and then use yum to download the nvme-cli package. http://www14.software.ibm.com/support/customercare/sas/f/lopdiags/home.html   1. Download the IBM Linux on Power Tools Repository configuration RPM file available at: http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/yum.html   2. Copy the file to the target system and install the RPM # rpm -ivh ibm-power-repo-latest.noarch.rpm   3. To add the Tools Repository to the package management configuration, run the following command as root user (or with root privilege) and accept the Licensing agreement. # /opt/ibm/lop/configure   Instructions for installing nvme-cli tool using Red Hat Enterprise Linux   To list the currently enabled repositories, enter: # yum repolist enabled   Install the nvme-cli package, enter: # yum install nvme-cli Checking NVME device firmware for AIX: To list NVMe devices on your system and view firmware level, 1) Type: ls /dev | grep nvme nvme0 nvme1 2) List details of the nvme device(s) by: Type: lscfg -vl nvme# # lscfg -vl nvme0 nvme1 U78D4.ND1.2852224-P1-C2 PCIe4 x8 NVMe Flash Adapter PCIe4 1.6TB NVMe Flash Adapter x8 : Part Number.................02DE959 EC Level....................P46698 FRU Number..................02DE956 Product Specific.(AN).......02DE958 Feature Code/Marketing ID...EC7B Customer Card ID Number.....594A Serial Number...............AFYDJ025 Product Specific.(Z0).......10140682 Product Specific.(Z1).......3.0 Product Specific.(Z2).......1600 Product Specific.(Z3).......8.76 Product Specific.(Z4).......1 Product Specific.(Z5).......00 Product Specific.(Z6).......A180010D Product Specific.(Z7).......0YA Product Specific.(Z8).......3023 Product Specific.(Z9).......08 Product Specific.(ZA).......0008 Product Specific.(ZB).......0040 Product Specific.(ZC).......00 Manufacture ID..............S4WANE0N300071 ROM Level.(alterable).......REV.SP49 Product Specific.(RV).......s Hardware Location Code......U78D4.ND1.2852224-P1-C2 The firmware level is listed in the field: ROM Level (alterable). If the Firmware Version is less than REV.SP49 then you should update the microcode. Checking NVMe device firmware on Linux: After nvme-cli is installed on the system, check the current microcode level for the adapter enter the following command: # nvme list   Note: "nvme list" command uses libudev-devel, if it doesn't work, may need to install “libudev-devel"   If the FW Rev is different from “REV.SP49” install this update. 1) Type "nvme list" 2) The column “FW Rev” will list the firmware version. For example: Node SN Model Namespace Usage Format FW Rev ---------------- --------------- -------------------------------------- ------------ ---------------------- ---------------- ---------------- /dev/nvme1n1 S4WANE0N300071 PCIe4 1.6TB NVMe Flash Adapter x8 1 1.60 TB / 1.60 TB 4 KiB + 0 B Rev.SP49 If the Firmware Version is less than REV.SP49 then you should update the microcode. 4.0 Downloading and Unpacking the Firmware Update Package NOTE:  For RHEL, the image will be the RPM format file.   1. Download feature code EC7A, EC7B, EC7C, EC7D, EC7E, or EC7F package from FixCentral, and transfer the file, to a temporary directory on the target server such as /tmp.   2. Install the package (AIX) rpm -ihv --ignoreos --force pci.144dA824-53503439.aix7.2.rpm (Linux) rpm –ivhpci.144dA824.SP49-53503439.linux.rpm   3. The microcode package extracts the firmware file to the following directory: * Linux: /lib/firmware * AIX: /etc/microcode   4. Verify the microcode image in Section 5.0 5.0 Verifying microcode before download and notes NOTE: Microcode download must be performed separately for each Adapter bus under distinct Logical Partitions (LPAR's).               Please save current and older versions of the microcode update files in case you need to restore your system. NOTE: Microcode Download CAN be performed concurrently. Installation Time:  Approximately 30 min System Time. Please verify the file size and checksum of the raw microcode files matches the below. Filesize: 6164480 Checksum: 10344 For AIX: " ls -l /etc/microcode/A180010D.53503439" to verify Filesize.    "sum /etc/microcode/A180010D.53503439" to verify Checksum For Linux: "ls -l /lib/firmware/pci.144dA824.SP49.53503439" to verify Filesize.    "sum /lib/firmware/pci.144dA824.SP49.53503439" to verify Checksum. For systems that have LPAR installed, the microcode file must be placed on all partitions that have a NVMe device in order to update the code.  Follow this procedure for each partition: For AIX:     A)  Log into the target partition.     B)  Type: "cd /etc/microcode"     C)  SFTP to the partition that has the media bay assigned. (ie.  "sftp partition1.ibm.com")     D)  Type: "cd /etc/microcode"     E)  Type: "get A180010D.53503439"     F)  Type: "quit" For Linux:     A)  Log into the target partition.     B)  Type: "cd /lib/firmware"     C)  SFTP to the partition that has the media bay assigned. (ie.  "sftp partition1.ibm.com")     D)  Type: "cd /lib/firmware"     E)  Type: "get pci.144dA824.SP49.53503439"     F)  Type: "quit" 5.1 Microcode Download Procedure-NVMe Adapters for AIX Procedure: 1) Stop all applications. 2) Type "ls /dev | grep nvme" and press "Enter". This will list all of the NVMe adapters in your partition 3) To display details for the nvme device (including current firmware level), type “lscfg -vl nvme#”. For example, for device nvme0, type “lscfg -vl nvme0” 4) For each NVMe adapter to be updated, issue the following commands: a) Type “diag” b) Select “Task Selection” c) Select “Microcode Tasks” d) Select “Download Microcode” e) Select the NVMe device to have microcode updated on. For example, select: nvme0 and press F7 f) Select “/etc/microcode” as location of updated firmware image. Press ENTER g) For level to install, select “53503439”. The system will take several minutes to update the firmware. Once completed, the screen will display the following message: INSTALL MICROCODE nvme0 PCIe4 x8 NVMe Flash Adapter Installation of the microcode has completed successfully on the following instances of the adapter: nvme0 The current microcode level is 53503439. Please run diagnostics on these resources to ensure that the adapter is functioning properly. Use Enter to continue. 5.2 Microcode Download Procedure-NVMe Adapters for Linux Procedure: 1) Stop all applications. 2) Type "nvme list" and press "Enter". This will list all of the NVMe adapters in your partition. In the Node column, the NVMe adapters will be listed as /dev/nvme#n1 where # is the number of the device. 3) For each NVMe adapter to be update, issue the following commands: a) Type “nvme fw-log /dev/nvme<#>” and press “Enter”, where <#> is the number of the device. For example, for device /dev/nvme0n1, type “nvme fw-log /dev/nvme0” and press “Enter”. This will list the firmware level currently loaded on the adapter as well as the number of slots in the adapter. These 3 slots are listed as frs1, frs2, and frs3. b) Download & activate firmware to the adapter by typing: “nvme fw-download /dev/nvme# --fw=/lib/firmware/pci.1014064a.REV.SP49.4d413234” and press “Enter”, where # is the number of the device. i. nvme fw-download /dev/nvme# --fw=/lib/firmware/pci.144dA824.SP49.53503439 and press “Enter”, where # is the number of the device. For example, for device /dev/nvme0, type “nvme fw-download /dev/nvme0 –fw=/lib/firmware/pci.144dA824.SP49.53503439” and press “Enter”. ii. “nvme fw-activate /dev/nvme# -a 0 –s 1” where # is the number of the device iii. nvme fw-download /dev/nvme# --fw=/lib/firmware/pci.144dA824.SP49.53503439 (Repeat of step i) iv. “nvme fw-activate /dev/nvme# -a 0 –s 2” where # is the number of the device v. nvme fw-download /dev/nvme# --fw=/lib/firmware/pci.144dA824.SP49.53503439 (Repeat of step i) vi. “nvme fw-activate /dev/nvme# -a 0 –s 3” where # is the number of the device vii. “nvme fw-activate /dev/nvme# -a 3 –s 3” where # is the number of the device viii. “nvme reset /dev/nvme#” where # is the number of the device For example, to download & activate the firmware on NVMe device /dev/nvme0, type: i. nvme fw-download /dev/nvme0 --fw=/lib/firmware/pci.144dA824.SP49.53503439 ii. nvme fw-activate /dev/nvme0 -a 0 –s 1 iii. nvme fw-download /dev/nvme0 --fw=/lib/firmware/pci.144dA824.SP49.53503439 iv. nvme fw-activate /dev/nvme0 -a 0 –s 2 v. nvme fw-download /dev/nvme0 --fw=/lib/firmware/pci.144dA824.SP49.53503439 vi. nvme fw-activate /dev/nvme0 -a 0 –s 3 vii. nvme fw-activate /dev/nvme0 -a 3 –s 3 viii. nvme reset /dev/nvme0 6.0 Microcode Readme Change History 2023-11-17 Fix timing Issue between Host Command Request and updating temperature in device - 53503439 (SP49) 2021-5-17 GA microcode level 53503437 (SP47)