IMPORTANT NOTICE: This microcode install is NOT concurrent, a system power cycle OR a partition reboot is required PRIOR to installing this microcode. Microcode install must be performed within 24 hours after system power cycle or partition reboot.
In a case that a system power cycle or a partition reboot was not done before this microcode install, the NVMe device may become unresponsive/hung after microcode install, BUT the NVMe device will fully recover following a system power cycle or partition reboot. You must then do another microcode install to NVMe after power cycle (or partition reboot) to complete microcode install.

Note that the partition referred to in the above statement may be a VIOS, and a redundant VIOS configuration will avoid disruptive impact of this microcode install to the VIOS client partitions.

1. General Information

The Microcode files provided in this package are for NVMe drives used in IBM Power Systems Servers with AIX or Linux operating systems

1.1 Disk Drives Affected

Below is the list of microcode files in this package:

NOTE: To insure the integrity of microcode files, please make sure the checksum value is correct after extraction

Size

Disk Drive Model

FRU#

P/N (VPD)

Microcode File Name

Check

sum

400GB

XP400HE

00LY537

00LY530

A1800100.49423044

33702

1.2 Microcode Change History

Release date	Code level	Severity/ Impact	Change Description
October 2019	IB0D	Function/ HIPER	-Fixed a critical FW defect that if not installed can cause NVMe drives to fail permanently rendering the drive unreadable. This fix is required to be installed on NVMe drives after a fresh boot of OS using the NVMe drive.
July 2019	IB0C	Function/ HIPER	- With some unique workload AND a system reboot/power cycle drive could go into read/write protect mode. This FW has a workaround that recovers the drive. This workaround requires a system power cycle to recover the drive. - Fixed Arm Data Abort: If a Controller Level Reset occurs to activate the firmware during a write to NOR, the NOR will become corrupted
Feb 2019	IB0A	Function/ ATT	-Fixed a mis-calculation bug that was causing drive to incorrectly report 100% life used when in fact drive had less than 1% life used -Drive may go offline after a system reboot, or after OS unconfigure/configure operation resulting in drive reporting error: 2/C0 (Table Rebuild/Potential Data Loss)
May 2018	IB07	Function/ ATT	- Boot drive may not show up after fresh AIX install - Uncorrectable LB 0 - nvme write-unc causes IO 0 QID 7 timeout, reset controller
March 201	IB06	New/New	Initial Release

===============================================================

2. AIX and VIOS(PowerVM Virtual I/O Server) Operating System

Note: For VIOS systems, microcode of shared physical I/O resources can only be updated from VIOS partition.

To update disk drive microcode, from VIOS partition, enter the following command:

>aix

Then follow the instruction in section 2.3 to transfer/extract the microcode file, and section 2.4.1 to update the disk drive code.

2.1 Installation Time

Approximately 20 minutes if using Diagnostics CD; Approximately 5 minutes/drive using AIX concurrent diagnostics.

2.2 Determine Drive Model, FRU&PN and Current Microcode Level
To see the drive type and current microcode level, Execute the following command:
lscfg –vl nvme*

Vital Product Data (VPD)

# lscfg -vl nvme0

nvme0 U78D2.001.WZS0025-P1-C50-C1 PCIe3 x4 NVMe Flash Adapter

400GB NVMe Gen3 M.2 Flash Module :

Part Number.................01CM500

EC Level....................P02084

FRU Number..................00LY537

Product Specific.(AN).......00LY530

Feature Code/Marketing ID...ES14

Customer Card ID Number.....58CE

Serial Number...............ZEM000P0

Product Specific.(Z0).......IBM10140620

Product Specific.(Z1).......1.5

Product Specific.(Z2).......400

Product Specific.(Z3).......1.09

Product Specific.(Z4).......0

Product Specific.(Z5).......0

Product Specific.(Z6).......A1800100

Product Specific.(Z7).......0NM

Product Specific.(Z8).......2635

Product Specific.(Z9).......0A

Product Specific.(ZA).......1E

Product Specific.(ZB).......10

Manufacture ID..............ZEM000P0

ROM Level.(alterable).......Rev.IB0D ß--- Code Level IB0D

Product Specific.(RV)........

        Note: NVMe device code level can also be checked using the following AIX command
        >lsmcode -rd nvme0
        > A1800100.49423044

===============================================================

2.3 Transferring and Unpacking the Microcode Update Package

2.3.1 Internet AIX self extracting file (.bin)

a) Make a directory on an AIX system to receive the AIX format file.
Enter: mkdir /tmp/fwupdate
b) Transfer the AIX format file to /tmp/fwupdate directory (using "Save as ...").
You'll see the filename is NVMeN5000.bin
c) Unpack the file by executing the instructions below.
Enter the commands:
cd /tmp/fwupdate
chmod +x NVMeN5000.bin
./NVMeN5000.bin
[Don't overlook the periods (.) in the above command.]

The directory NVMeN5000 with microcode files listed in section 1.1 will be added to /tmp/fwupdate.
d-Enter the following command to copy the microcode files to " /etc/microcode " directory :
   cp /tmp/fwupdate/ NVMeN5000/*   /etc/microcode
e- If you are using stand alone AIX diag (bootable USB flash or CD/DVD) to update drive’s microcode, transfer microcode files to a USB drive and place it in “/etc/microcode” directory.
    Please refer to this page for USB flash drive usage
   https://www.ibm.com/support/pages/using-and-taking-advantage-usb-devices-and-aix

2.3.2 Internet RPM package

Note: The instructions that follow are specific AIX commands. AIX commands are CASE (lower and upper) SENSITIVE,

and must be entered exactly as shown, including filenames.

a) Make a directory on an AIX system to receive the RPM format file.
Enter: mkdir /tmp/microcode

b) Transfer the RPM format file to the /tmp/microcode directory.
You'll see the filename is NVMeN5000-IB0D-AIX.rpm

c) Unpack the file by executing the instructions below:
Enter the commands:
rpm -ihv --ignoreos --force NVMeN5000-IB0D-AIX.rpm ;(please note, there are two dashes)

The file size and checksum will be verified.

The microcode files will be added to /etc/microcode/
d) If you are using stand alone AIX diag (bootable USB flash or CD/DVD) to update drive’s microcode, transfer microcode files to a USB drive and place it in “/etc/microcode” directory.
Please refer to this page for USB flash drive usage
https://www.ibm.com/support/pages/using-and-taking-advantage-usb-devices-and-aix

========================================================

2.4 Microcode Download Procedure

Warning: Do not power off the system or the drive during microcode download as this may permanently damage the disk drive

NOTE: Please save current and older versions of the microcode update files in case you need to restore your system.

Caution: The microcode download process is nondestructive to the nvme drives; however, as a precaution, ensure that a backup operation has been performed on the affected systems/partitions PRIOR to the microcode installation.

Can microcode be installed Concurrently? NO
A system power cycle OR a partition reboot is required prior to microcode download to nvme device.

Installation Time: Approximately 30 minutes

2.4.1 Concurrent Download

1. A system power cycle OR a partition reboot is required prior to microcode download to nvme device after which the microcode may be downloaded concurrently

2. When system comes up, within 24 hours of reboot, Type diag at the AIX prompt.
3. Select the "Task Selection" from diagnostics menu.
4. Select "Microcode Download" from "Task Selection" menu.
5. Select resource that microcode will be applied to(hdisk? For physical disk, pdisk for RAID configured disks) and press ENTER, then press "F7" to commit..
6- A small screen displays the message " *** NOTICE*** The microcode installation occurs while the attached drives are available for use..... "
    Press enter to continue
7. Select the source for microcode, use arrow key to go up/down, Enter to continue.
                      /etc/microcode
                      usb mass storage

- Use /etc/microcode if microcode file is in the system
- Use “usb mass storage” if using usb flash drive

8- Display shows a list of microcodes available for the selected drive, highlight option "M 49423044", the latest microcode for the drive and press ENTER.
A message may warn you that current level of microcode is not available on the media, press ENTER to continue.

9. You will receive a msg. "Current Microcode is...." and "Download has completed successfully"
(this will take a several minutes)
10. Return to the Tasks Selection menu and repeat this procedure for each nvme (in the same partition) that requires this microcode.
11. Exit diagnostic

2.4.2 Standalone (Service Mode) Download

This procedure is used for all drives including rootvg or paging space drives
1. Properly shutdown the system.
2. Boot up the system in standalone diagnostics mode with the AIX diag CD/DVD or USB Thumb drive
3. Select the "Task Selection" from diagnostics menu.
4. Select "Microcode Download" from "Task Selection" menu.
5. Select resource that microcode will be applied to and PRESS ENTER.
6. Press "F7" to commit.
      (The current microcode level of the resource you selected earlier will be displayed at the top of the screen)
7. Make the following selections on the next screen:
    file system
      /etc/microcode
      usb mass storage    (ISO 9660 file system format)

8. You will receive a msg. "Current Microcode is...." and "Download has completed successfully"
(this may take a few minutes)
9. Return to the Tasks Selection menu and repeat this procedure for each hard disk that requires this microcode.
10. Exit diagnostic and reboot system in normal mode.

======================================================

3. Linux Operating System

Note: The following procedure is for Red Hat Linux, please use microcode file NVMeN5000-IB0D-LINUX.tgz for Ubuntu

tar -C-xzvf NVMeN5000-IB0D-LINUX.tgz

Red Hat Enterprise Linux for IBM Power LE

3.1 Determine the Current Microcode Level for Linux:

These steps assume you are logged in with root permissions and a PCIe3 NVMe Flash Adapter, Feature Code ES14 is installed.

Prerequisites for Red Hat Linux:

Use the ibm-power-repo to configure the repositories and then use yum to download the nvme-cli package.

http://www14.software.ibm.com/support/customercare/sas/f/lopdiags/home.html

1. Download the IBM Linux on Power Tools Repository configuration RPM file available at:

http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/yum.html

2. Copy the file to the target system and install the RPM

# rpm -ivh ibm-power-repo-latest.noarch.rpm

3. To add the Tools Repository to the package management configuration, run the following command as root user (or with root privilege) and accept the Licensing agreement.

# /opt/ibm/lop/configure

Instructions for installing nvme-cli tool using Red Hat Enterprise Linux

To list the currently enabled repositories, enter:

# yum repolist enabled

Install the nvme-cli package, enter:

# yum install nvme-cli

After nvme-cli is installed on the system, to check the current microcode level for the adapter enter the following command:

# nvme list

Note: "nvme list" command uses libudev-devel, if it doesn't work, may need to install “libudev-devel"

If the FW Rev is different from “IB0D” install this update.

1) Type "nvme list"
2) The column “FW Rev” will list the firmware version. For example:

Node SN Model Namespace Usage Format FW Rev

---------------- ------------ ----------------------------------- ------------ ---------------------- ---------------- -------

/dev/nvme0n1 xxxx PCIe3 400GB NVMe Flash 1 400 GB / 400GB 4 KiB + 0 B IB0D

If the Firmware Version is less than IB0D then you should update the microcode.

3.2 Downloading and Unpacking the Firmware Update Package

1. Download feature code ES14 package from FixCentral, and transfer the file, to a temporary directory on the target server such as /tmp.

2. Unpack the package

rpm -ivh --ignoreos NVMeN5000-IB0D-LINUX.rpm

3. The microcode package extracts the firmware file to the /lib/firmware/ directory

3.3 Verifying microcode before download and notes

NOTE: Microcode download must be performed separately for each nvme under distinct Logical Partitions (LPAR's).
Please save current and older versions of the microcode update files in case you need to restore your system.

Please verify the file size and checksum of the raw microcode files matches the below.
Filesize: 1310208
Checksum: 33702
For Linux: "ls -l /lib/firmware/A1800100.49423044 " to verify Filesize.
"sum /lib/firmware/A1800100.49423044 " to verify Checksum.

For systems that have LPAR installed, the microcode file must be placed on all partitions that have(owns) an NVMe device in order to update the code.
Follow this procedure for each partition:

1. Log into the target partition.

2. Type: "cd /lib/firmware"

3. SFTP to the partition that has the media bay assigned. (ie. "sftp partition1.ibm.com")

4. Type: "cd /lib/firmware"

5. Type: "get A1800100.49423044"

6. Type: "quit"

3.4 Microcode Download Procedure-NVMe Adapters for Linux

1) Reboot the partition OR power cycle the system before next step

2) Type "nvme list" and press "Enter". This will list all of the NVMe adapters in your partition. In the Node column, the NVMe adapters will be listed as /dev/nvme#n1 where # is the number of the device.
3) For each NVMe adapter to be update, issue the following commands:

a) Type “nvme fw-log /dev/nvme<#>” and press “Enter”, where <#> is the number of the device.

For example, for device /dev/nvme0n1, type

“nvme fw-log /dev/nvme0” and press “Enter”. This will list the firmware level currently loaded on the adapter as well as the number of slots in the adapter.
These 4 slots are listed as frs1, frs2, frs3 and frs4.

b) Download and activate firmware to the adapter by typing:

Note that there are 4 FW “slots” in this nvme, each slot must be updated AND activated, example below is for nvme0

i. nvme fw-download /dev/nvme0 --fw=/lib/firmware/A1800100.49423044

ii. nvme fw-activate /dev/nvme0 -a 0 –s 1

iii. nvme fw-download /dev/nvme0 --fw=/lib/firmware/A1800100.49423044

iv. nvme fw-activate /dev/nvme0 -a 0 –s 2

v. nvme fw-download /dev/nvme0 --fw=/lib/firmware/A1800100.49423044

vi. nvme fw-activate /dev/nvme0 -a 0 –s 3

vii. nvme fw-download /dev/nvme0 --fw=/lib/firmware/A1800100.49423044

viii. nvme fw-activate /dev/nvme0 -a 0 –s 4

ix. nvme fw-activate /dev/nvme0 -a 3 –s 3 ß this step may be omitted if the nvme device does not support the function

x. nvme reset /dev/nvme0

3.5 Revisions

Dec 12, 2019: Added instruction for USB thumb drive