IBM Storage Protect

IBM Storage Protect
HSM for Windows Client 8.1.19


Networking

The HSM for Windows webserver uses secure HTTPS connections to its clients, TLS 1.2 or higher.
If no valid certificate exists, the HSM network service, hsmnet, will not start.

Prerequisites
Certificates
Create Corporate Certificates
Create Self-signed Certificates
Install HSM Certificates
Install Chain of Trust Certificates
Network Service
HSM Web Client
Connect an HSM Node
Configure the HSM Proxy Server
Prometheus
Scraping a Single HSM Installation
Scraping Multiple HSM Installations
Grafana
Sample Dashboard
Metric Interpretation

Prerequisites


HSM certificate creation and handling requires a Windows OpenSSL 1.1.1 LTS installation. The OpenSSL version is important, other branches will not work.

We recommend NOT to install OpenSSL files in the Windows system32 directory, but in the OpenSSL installation directory.

HSM searches required OpenSSL binaries at runtime using the PATH environment variable. Add the OpenSSL bin-directory (NOT the installation directory) to the PATH variable on the HSM file server. Make sure, that the two files libcrypto-1_1-x64.dll and libssl-1_1-x64.dll are located in the bin-directory. Check the version using

> openssl version

Certificates


Certificate creation with the HSM for Windows tool dsmcertificate.exe must be done on the HSM file server, where the certificate will be used. Reason: The certificate name is stored in the registry for later use, and your password is kept savely in a GsKit store for automated use by the HSM webserver.

In order to use secure TLS connections between the HSM webserver and the Prometheus TSDB webclient, you may create corporate certificates, or self-signed certificates.

Remarks:

Create Corporate Certificates

If corporate certificates are available in your company, they are provided by the Certificate Authority (CA) of your company. You need to create certificate request files <csr-files> for the HSM file servers and send them to the CA, they will return certificate files <crt-files> for your HSM file servers.

The following OpenSSL command creates a certificate request file. OpenSSL will ask you for a password, and other descriptive parameters for the certificate. Make sure, that you enter the fully qualified hostname (FQDN) of your HSM server for the subject name CN-part (common name). Remember your password.

> openssl req -new -keyout <key-file> -out <csr-file>

OpenSSL creates 2 files: The certificate request file, <csr-file>, and a password-protected private key file, <key-file>. The private key file and its password is your secret and must never be provided to anyone. Send the certificate request file to the certificate authority together with your certificate order. The certificate authority will return you the desired HSM server certificate file <crt-file>.

Additionally, you may receive certificates of your company, that the new certificate refers to (root and intermediate certificates of the chain of trust of the HSM server certificate). In this README we assume, that you obtained 2 certificates, <crt-root-file>, and <crt-intermediate-file>. See section Install Chain of Trust Certificates.

To utilize the certificate with HSM, issue the command

> dsmcertificate cert -x509 <crt-file> -rsa <key-file> [-t <validity>]

in a command shell in the HSM installation root.

Specify the certificate file for the HSM server, <crt-file>, with option -x509, and your private key file, <key-file> with option -rsa.

The certificates validity period in years can be specified with option -t. If omitted, the validity period defaults to 10 years.

You will be asked for the password, that was used when the certificate request was created.

Corporate certificates are created for the HSM server as well as for the client computer, that is running Prometheus. Certificate file are named based on the fully qualified domain name (FQDN) of the HSM server, e.g. HSM-server-<host>.crt. The <host> is the FQDN, that was extracted from the subject name CN-part (common name) of the certificate input file, <crt-file>.

Remark:

Create Self-signed Certificates

Self-signed certificates refer to themselves. You can create them easily by issuing

> dsmcertificate self-cert [-cn <CN>] [-t <validity>] [-p[s]]

in a command shell in the HSM installation root. You will be asked twice for the password.

The computer name can be specified with option -cn. TLS certificates require the fully specified hostname in the CN (Common Name) entry of the issuer. If option -cn is omitted the certificate is created for the computer using the FQDN (fully qualified domain name).

The certificates validity period in years can be specified with option -t. If omitted, the validity period defaults to 10 years.

A password can optionally be created from random data using option -p[s]. -p creates certificates without showing the password, -ps also prints the password to the command line.

Remark:

Install HSM Certificates

Whether you created coporate certificates, or self-signed certificate, in both cases you will find 4 files in the cert-subdirectory of the HSM installation:

<installation-dir>\cert\HSM-server-<host>.pem
<installation-dir>\cert\client\HSM-server-<host>.crt
<installation-dir>\cert\client\HSM-client-<host>.crt
<installation-dir>\cert\client\HSM-client-<host>.key

<host> is the fully qualified domain name (FQDN) of the HSM file server machine, or the name specified in option -cn.

Use the 4 files in the following manner:

<installation-dir>\cert\HSM-server-<host>.pem
contains the encrypted, private RSA key, as well as the X.509 server certificate.

The file is used by the HSM webserver. The path to this file is stored in the registry, so do not move the file away from its place.

<installation-dir>\cert\client\HSM-server-<host>.crt
contains only the X.509 server certificate, i.e. the public part of HSM-server-<host>.pem.

To use the certificate, you need to install the certificate in the Windows certificate store for your webclient. Your webclient is the web browser running the HSM Web Client, the HSM Proxy Server (connecting to remote HSM Proxy Nodes), or Prometheus, respectively. Follow these steps to install the certificate on Windows:

<installation-dir>\cert\client\HSM-client-<host>.crt
<installation-dir>\cert\client\HSM-client-<host>.key
contain a client certificate, and client key, respectively. These are required for Prometheus only. They are NOT required for the HSM Web Client.

Copy both files to the Prometheus installation and update the Prometheus configuration file as described in the Prometheus section of this README.

Install Chain of Trust Certificates

If your certificates are corporate cerificates, and your company provided root and intermediate certificates, install them on the computer that is running the webclient, i.e. Prometheus.

Self-signed certificates do not rely on chain of trust certificates, so no further certificates need to be installed.

<crt-root-file>
Trusted root certificate.

The root certificate file must be installed in the Windows "Trusted Root Certificate Authorities" store of the client computer in the exactly same way as the HSM server certificate. Follow the instructions for <installation-dir>\cert\client\HSM-server-<host>.crt.

<crt-intermediate-file>
Intermediate certificate.

The intermediate certificate file must be installed in the Windows trust store of the client computer in the same way as the HSM server certificate. Follow the instructions for <installation-dir>\cert\client\HSM-server-<host>.crt.

However, as it is not a root certificate, we recommend installing it in the "Intermediate Certificate Authorities" store instead of the "Trusted Root Certificate Authorities".

Remarks:

Network Service


The HSM network service, hsmnet, is configured NOT to start automatically, by default. To start and stop the service manually enter

> net start hsmnet
> net stop hsmnet

The HSM network service listens on port 8888, per default. The port can be changed using regedit in the Windows registry at

HKLM\Software\IBM\ADSM\CurrentVersion\HsmClient\Net\ListeningPorts 8888s

Note, that the trailing character s must be appended for HTTPS ports.

Remarks:

HSM Web Client


The HSM Web Client monitors several HSM installations in a single web application, and it can be used to maintain HSM installations. To connect to HSM, where the HSM network service, hsmnet, is running, enter the URL

https://<hsm-server>:8888

in your browser, where hsm-server is the name or IP-address of the file server, that hosts the HSM installation.

Before connecting with your browser to the HSM Web Client, execute the following 3 steps:

Connect an HSM Node

Copy the public X.509 certificate

<installation-dir>\cert\client\HSM-server-<host>.crt

to the computer, where you want to use the web browser to monitor HSM. Double-click the crt-file, and install the certificate to the "Trusted Root Certificate Authorities" certificate store following the instructions listed in chapter Install HSM Certificates.

Add one local HSM Web user on each HSM installation using the dsmadmin tool

> dsmadmin add-user <web-user> [password]

Remarks:

Verify, that the web-user was stored correctly, e.g.

> dsmadmin list-users

  --------------------------------------------
   User       HSM Host  Port  Security  Proxy
  --------------------------------------------
   name              -     -         -  node
  --------------------------------------------

Next, choose one of your HSM installations to test the connection.

Remark:

Make sure, that the HSM network service, hsmnet, is running, and open the URL of the HSM installation server in your browser.

Log in to the HSM Web Client using the credential, that you installed with dsmadmin.

Configure the HSM Proxy Server

The HSM Proxy Server can be any of your HSM installations. The HSM Proxy Server acts as a client to other HSM installations, that you register as HSM Proxy Nodes. In other words, the HSM Proxy Server acts as client to HSM Proxy Nodes just as the web browser acts as client to the HSM Proxy server.

Copy public X.509 certificates

<installation-dir>\cert\client\HSM-server-<host1>.crt   (on host1)
<installation-dir>\cert\client\HSM-server-<host2>.crt   (on host2)
...

from all HSM Proxy Nodes to the HSM Proxy Server. Double-click crt-files on the HSM Proxy Server, and install certificates to the "Trusted Root Certificate Authorities" certificate store following the instructions listed in chapter Install HSM Certificates.

Register all HSM Proxy Node users on the HSM Proxy Server

> dsmadmin add-user -h host1[:port] <web-user> [password]
> dsmadmin add-user -h host2[:port] <web-user> [password]
> ...

Remarks:

Verify, that the web-user was stored correctly, e.g.

> dsmadmin list-users

  ----------------------------------------
   User  HSM Host  Port  Security  Proxy
  ----------------------------------------
   name1    proxy     -         -  server
   name2    host1  8888   TLS 1.2  node
   name3    host2  8888   TLS 1.2  node
   ...
  ----------------------------------------

Remark:

Verify, that connections from the HSM Proxy Server to each HSM Proxy Node work

> dsmadmin ping-node <host1[:port]> [<web-user> [password]]
> dsmadmin ping-node <host2[:port]> [<web-user> [password]]
> ...

Remarks:

Make sure, that HSM network services, hsmnet, are running, on the HSM Proxy Server and on all HSM Proxy Nodes, i.e. all HSM installations. Open the URL of the HSM installation server in your browser.

Log in to the HSM Web Client using the credential, that you installed with dsmadmin on the HSM Proxy Server.

Prometheus


Prometheus maintains a time series database. For HSM data, it queries HSM installations.

Before Prometheus can connect to your HSM installations, prepare for a secure TLS connection between HSM servers and Prometheus:

Remark:

Scraping a Single HSM Installation

Prometheus comes with a configuration file, that is named prometheus.yml by default. Add an HSM job to the configuration file, e.g.


  # HSM for Windows Clients
  - job_name: 'hsm'
  
    scrape_interval: 60s
    scrape_timeout: 30s
    static_configs:
      - targets: ['<host>:8888']
  
    scheme: https
  
    tls_config:
      cert_file: C:\Prometheus\HSM-client-<host>.crt
      key_file:  C:\Prometheus\HSM-client-<host>.key
  
    metrics_path: /prometheus
   

Then, start prometheus.exe and wait until Prometheus is up and running.

Trouble Shooting:

Scraping Multiple HSM Installations

To monitor more than 1 HSM installation, you may want to add multiple jobs to the Prometheus configuration file, one more for each separate installation.

Another way is to add multiple HSM installations to the same Prometheus configuration job. Please refer to to Prometheus documentation.

Example with 2 HSM installations:

The client certificate files HSM-client-*.crt from the 2 HSM installations are stored in HSM-client.crt, and client key files HSM-client-*.key are stored in HSM-client.key, i.e.

> type HSM-client-<host1>.crt > HSM-client.crt
> type HSM-client-<host2>.crt >> HSM-client.crt

> type HSM-client-<host1>.key > HSM-client.key
> type HSM-client-<host2>.key >> HSM-client.key

You may also use

> merge-certs.bat

to automatically create the merged files. Copy all certificate and key files, as well as merge-certs.bat, from the HSM directory <installation-dir>\cert\client\ to the Prometheus installation, then run merge-certs.bat in the current directory.

merge-certs.bat relies on the client file names, that were created using dsmcertificate.exe. It merges all certificate files, that match HSM-client-*.crt into a file called HSM-client.crt, and all private key files, that match HSM-client-*.key into HSM-client.key, respectively.

Prometheus configuration file for our sample with the 2 HSM servers <host1>, <host2>:


  # HSM for Windows Clients
  - job_name: 'hsm'
 
    scrape_interval: 60s
    scrape_timeout: 30s
    static_configs:
      - targets: ['<host1>:8888','<host2>:8888']
 
    scheme: https
 
    tls_config:
      cert_file: C:\Prometheus\HSM-client.crt
      key_file:  C:\Prometheus\HSM-client.key
 
    metrics_path: /prometheus
   

Grafana


The HSM metrics can be visualized with Grafana. Therefore the Prometheus data source needs to be added to the Grafana configuration.

Sample Dashboard

HSM comes with a preconfigured Grafana dashboard template, that can be used to monitor HSM installations:

<installation-dir>\networking\HSM-grafana-dashboard.json


Overview
hsm_recalled_files,  hsm_migrated_files_details
Graph Originator Result Description
Recall Activity user success, failure HSM alerts and analysis.
See chapter Metric Interpretation for details.
Denied Recall Alerts user denied
Graph Originator Reason
Migration Candidate Analysis job, list, threshold migration, content freed, content update
Recall
hsm_recalled_files,  hsm_recalled_bytes
Graph Originator Result Description
Recalled User Files user success, denied, failure Recalls triggered by end-users.
Recalls may be denied due to user recall quotas.
Recalled User Bytes user -
Recalled Admin Files admin, list success, failure Recalls initiated by the HSM administrator (admin: dsmclc recall, list: dsmclc recalllist)
Recalled Admin Bytes admin, list -
Migrate
hsm_migrated_files,  hsm_migrated_bytes,  hsm_migrated_files_details,  hsm_migrated_bytes_details
Graph Originator Result Description
Migrated Files job, list, threshold success, failure Migrations initiated by the HSM administrator (job: dsmclc migrate / dsmgui, list: dsmclc migratelist, threshold: hsmmonitor)

Detailed graphs split successful migrations into:
migration: first migration of a file
content freed: file was recalled, not modified, migrated again
content update: file was recalled, modified, migrated again with a higher version number
acl update: file security was updated
attr update: file attributes were updated
Migrated Bytes job, list, threshold -
Graph Originator Reason
Migrated Files Details job, list, threshold migration, content freed, content update, acl update, attr update
Migrated Bytes Details job, list, threshold migration, content freed, content update
Backup Before Migrate
hsm_backup_files,  hsm_backup_bytes
Graph Originator Result Description
Backup Before Migrate Files migration success, failure Backup before HSM migration (optional).

Remark: Normal daily backup operations performed by using the B/A Client are not counted.
Backup Before Migrate Bytes migration -
Retrieve
hsm_retrieved_files,  hsm_retrieved_bytes
Graph Originator Result Description
Retrieved Files admin success, failure Retrievals initiated by the HSM administrator (admin: dsmclc retrieve / dsmgui)
Retrieved Bytes admin -
Move
hsm_moved_stubs,  hsm_moved_backend_files,  hsm_moved_backend_bytes
Graph Originator Result Description
Moved Stubs admin success, failure Stub move operations initiated by the HSM administrator (admin: dsmmove, tasks: hsmtasks)
Moved Backend Files tasks success, failure
Moved Backend Bytes tasks -
Network Usage
hsm_network_bytes
Graph Originator Action Description
Network Bytes job, list, threshold migration Network traffic between the file server running HSM and the Storage Protect backend server, caused by the HSM client actions migration, backup, recall, retrieve, move.

Remark: Meta data transferred to or from the backend server is neglected.
migration backup
admin, list, user recall
admin retrieve
tasks move
Volume Usage
hsm_volume_used_bytes,  hsm_volume_free_bytes,  hsm_volume_used_percent
Graph Volume Description
Volume Used Percent Volume Label and Drive Letter Windows file server volume information.
Volume Free Bytes Volume Label and Drive Letter
Volume Used Bytes Volume Label and Drive Letter


Metric Interpretation

Metric interpretation is explained here for Grafana curves, but valid for any visualization of the HSM metric curves.

Recall Activity
The HSM administrator can supervise HSM recall activity in this graph. The Recall Activity graph live-monitors the number of files, that users are recalling on HSM file servers.

If the number of recalls is higher than expected, revise your migration policies and migration jobs. Also see the Migration Candidate Analysis graph below.

Curve peaks indicate mass recalls, especially if they occur outside business hours. Mass recalls is a frequently reoccurring HSM problem caused by misconfigured scanners, like Anti-Virus software or indexing software, accessing HSM stub file content unintendedly.
Denied Recall Alerts
The Denied Recall Alerts graph alerts the HSM Administrator to intervent, if quota limits apply. When recall limits exceed, users do not have access to their stub files.

HSM allows to limit the number of recalls per time for users and/or user groups. If the user quota feature is not activated, no curves will be shown.

Remark: Review alerts in the HSM Web Client, or check HSM listing files listing\hsmservice-*.log to find out which users are affected.
Migration Candidate Analysis
The Migration Candidate Analysis graph analyses whether files are frequently re-migrated. Many frequently re-migrated files indicate that job and/or migration policies include files, that should not be migrated yet, because they are still in use.

Curves with result content freed show files that have been recalled and read by users. The result content update indicates, that recalled files have also been modified and new content versions have been stored in the backend.