Copyright International Business Machines Corp. 2020.
US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
The HSM for Windows webserver uses secure HTTPS connections to its clients,
TLS 1.2 or higher.
If no valid certificate exists, the HSM network service, hsmnet
,
will not start.
Prerequisites
Certificates
Create Corporate Certificates
Create Self-signed Certificates
Install HSM Certificates
Install Chain of Trust Certificates
Network Service
Prometheus
Scraping a Single HSM Installation
Scraping Multiple HSM Installations
Grafana
Sample Dashboard
Metric Interpretation
HSM certificate creation and handling requires a Windows OpenSSL 1.1.1 LTS installation. The OpenSSL version is important, other branches will not work.
We recommend NOT to install OpenSSL files in the Windows system32
directory, but in the OpenSSL installation directory.
HSM searches required OpenSSL binaries at runtime using the PATH
environment variable. Add the OpenSSL bin-directory (NOT the installation
directory) to the PATH
variable on the HSM file server. Make
sure, that the two files libcrypto-1_1-x64.dll
and
libssl-1_1-x64.dll
are located in the bin-directory. Check the
version using
> openssl version
Certificate creation with the HSM for Windows tool
dsmcertificate.exe
must be done on the HSM file server,
where the certificate will be used. Reason: The certificate name is stored
in the registry for later use, and your password is kept savely in a
GsKit store for automated use by the HSM webserver.
In order to use secure TLS connections between the HSM webserver and the Prometheus TSDB webclient, you may create corporate certificates, or self-signed certificates.
If corporate certificates are available in your company, they are
provided by the Certificate Authority (CA) of your company. You need to create
certificate request files <csr-files>
for the HSM file
servers and send them to the CA, they will return certificate files
<crt-files>
for your HSM file servers.
The following OpenSSL command creates a certificate request file. OpenSSL will ask you for a password, and other descriptive parameters for the certificate. Make sure, that you enter the fully qualified hostname (FQDN) of your HSM server for the subject name CN-part (common name). Remember your password.
> openssl req -new -keyout <key-file> -out <csr-file>
OpenSSL creates 2 files: The certificate request file,
<csr-file>
, and a password-protected private key
file, <key-file>
. The private key file and its
password is your secret and must never be provided to anyone. Send the
certificate request file to the certificate authority
together with your certificate order. The certificate authority will
return you the desired HSM server certificate file
<crt-file>
.
Additionally, you may receive certificates of your company, that the
new certificate refers to (root and intermediate certificates of the
chain of trust of the HSM server certificate). In this README we assume,
that you obtained 2 certificates, <crt-root-file>
,
and <crt-intermediate-file>
. See section
Install Chain of Trust
Certificates.
To utilize the certificate with HSM, issue the command
> dsmcertificate cert -x509 <crt-file> -rsa <key-file>
in a command shell in the HSM installation root.
Specify the certificate file for the HSM server,<crt-file>
, with option -x509
, and your
private key file, <key-file>
with option -rsa
.
You will be asked for the password, that was used when the certificate request was created.
Corporate certificates are created for the HSM server as well as for the
client computer, that is running Prometheus. Certificate file are named
based on the fully qualified domain name (FQDN) of the HSM server, e.g.
HSM-server-<hostname>.crt
. The <hostname>
is the FQDN, that was extracted from the subject name CN-part (common name)
of the certificate input file, <crt-file>
.
Self-signed certificates refer to themselves. You can create them easily by issuing
> dsmcertificate self-cert [-cn <CN>] [-p[s]]
in a command shell in the HSM installation root.
The computer name can be specified with option -cn
. TLS
certificates require the fully specified hostname in the CN
(Common Name) entry of the issuer. If option -cn
is omitted
the certificate is created for the computer using the FQDN (fully qualified
domain name). You will be asked twice for the password.
A password can optionally be created from random data using option -p[s]
.
-p
creates certificates without showing the password, -ps
also prints the password to the command line.
Whether you created coporate certificates, or self-signed certificate, in both cases you will find 4 files in the cert-subdirectory of the HSM installation:
<installation-dir>\cert\HSM-server-<hostname>.pem
<installation-dir>\cert\client\HSM-server-<hostname>.crt
<installation-dir>\cert\client\HSM-client-<hostname>.crt
<installation-dir>\cert\client\HSM-client-<hostname>.key
<hostname>
is the fully qualified domain name (FQDN) of
the HSM file server machine, or the name specified in option -cn
.
Use the 4 files in the following manner:
<installation-dir>\cert\HSM-server-<hostname>.pem
The file is used by the HSM webserver. The path to this file is stored in the registry, so do not move the file away from its place.
<installation-dir>\cert\client\HSM-server-<hostname>.crt
HSM-server-<hostname>.pem
.
To use the certificate, you need to install the certificate in the certificate store of your webclient, which is Prometheus. Follow these steps to install the certificate on Windows:
HSM-server-<hostname>.crt
to the
client machine"Yes"
"Local Machine"
and click "Next"
"Place all certificates in the following store"
and select "Trusted Root Certificate Authorities"
"Ok"
to install<installation-dir>\cert\client\HSM-client-<hostname>.crt
<installation-dir>\cert\client\HSM-client-<hostname>.key
Copy both files to the Prometheus installation and update the Prometheus configuration file as described in the Prometheus section of this README.
If your certificates are corporate cerificates, and your company provided root and intermediate certificates, install them on the computer that is running the webclient, i.e. Prometheus.
Self-signed certificates do not rely on chain of trust certificates, so no further certificates need to be installed.
<crt-root-file>
The root certificate file must be installed in the Windows
"Trusted Root Certificate Authorities"
store of
the client computer in the exactly same way as the HSM server
certificate. Follow the instructions for
<installation-dir>\cert\client\HSM-server-<hostname>.crt
.
<crt-intermediate-file>
The intermediate certificate file must be installed in the Windows trust
store of the client computer in the same way as the HSM server certificate.
Follow the instructions for
<installation-dir>\cert\client\HSM-server-<hostname>.crt
.
However, as it is not a root certificate, we recommend installing
it in the "Intermediate Certificate Authorities"
store
instead of the "Trusted Root Certificate Authorities"
.
Remarks:
Certificate creation with dsmcertificate.exe
does not
overwrite any files, as certificate files may already be in use on
HTTPS-client side. However, if you would like to cleanup existing files,
use
> dsmcertificate delete-cert [-cn <CN>]
Certificate information can be obtained using the OpenSSL command
> openssl x509 -text -noout -in <file.crt|file.pem>
Private key access requires a password. Private key information can be obtained using the OpenSSL command
> openssl rsa -text -noout -in <file.key|file.pem>
The HSM network service, hsmnet
, is configured NOT to start
automatically, by default. To start and stop the service manually enter
> net start hsmnet
> net stop hsmnet
The HSM network service listens on port 8888, per default. The port can be
changed using regedit
in the Windows registry at
HKLM\Software\IBM\ADSM\CurrentVersion\HsmClient\Net\ListeningPorts 8888s
Note, that the trailing character s
must be appended for HTTPS ports.
Remarks:
The HSM network service, hsmnet
, runs as the well-known
user Network Service
. Running the service with that
user protects HSM and the fileserver, HSM is running on, against
network attacks.
The hsmnet
service running with restricted authorization
cannot perform administrative tasks, these tasks are performed by the
hsmtasks
service.
As hsmnet
relies on hsmtasks
to work
properly, hsmtasks
must be running, if hsmnet
becomes operative. The service dependency is configured in Windows
by the HSM setup. If you start hsmnet
, hsmtasks
will start automatically. And, you cannot shut down hsmtasks
before hsmnet
has been shut down.
Prometheus comes with a configuration file, that is named prometheus.yml by default. Add an HSM job to the configuration file, e.g.
# HSM for Windows Clients
- job_name: 'hsm'
scrape_interval: 30s
scrape_timeout: 30s
static_configs:
- targets: ['<hostname>:8888']
scheme: https
tls_config:
cert_file: C:\Prometheus\HSM-client-<hostname>.crt
key_file: C:\Prometheus\HSM-client-<hostname>.key
metrics_path: /prometheus
Then, start prometheus.exe and wait until Prometheus is up and running.
Trouble Shooting:
If Prometheus refuses to start, make sure that the YML indention structure is correct. The syntax of the Prometheus YML configuration file is sensible regarding indention.
If Prometheus does start, but the connection does not work, try to switch off security in the Prometheus configuration file:
tls_config:
insecure_skip_verify: true
# cert_file: C:\Prometheus\HSM-client-<hostname>.crt
# key_file: C:\Prometheus\HSM-client-<hostname>.key
If Prometheus does start, but does not connect, check your
certificates and certificates installed in the Windows trust store
(client side). The Prometheus web page displays some additional
details for the reason, check the Prometheus menu entry at
Status
→ Targets
.
Use OpenSSL on client side to connect to the HSM network service
> openssl s_client -connect <hostname>:8888
To monitor more than 1 HSM installation, you may add multiple jobs to the Prometheus configuration file, one for each separate installation.
Another way is to add multiple HSM installations to the same Prometheus configuration job.
Example with 2 HSM installations:
The client certificate files HSM-client-*.crt
from the
2 HSM installations are stored in HSM-client.crt
,
and client key files HSM-client-*.key
are stored in
HSM-client.key
, i.e.
> type HSM-client-<hostname1>.crt > HSM-client.crt
> type HSM-client-<hostname2>.crt >> HSM-client.crt
> type HSM-client-<hostname1>.key > HSM-client.key
> type HSM-client-<hostname2>.key >> HSM-client.key
You may also use
> merge-certs.bat
to automatically create the merged files. Copy all certificate and key
files, as well as merge-certs.bat
, from the HSM directory
<installation-dir>\cert\client\
to the Prometheus
installation, then run merge-certs.bat
in the current directory.
merge-certs.bat
relies on the client file names, that were
created using dsmcertificate.exe
. It merges all certificate files,
that match HSM-client-*.crt
into a file called
HSM-client.crt
, and all private key files, that match
HSM-client-*.key
into HSM-client.key
, respectively.
Prometheus configuration file for our sample with the 2 HSM servers
<hostname1>
, <hostname2>
:
# HSM for Windows Clients
- job_name: 'hsm'
scrape_interval: 30s
scrape_timeout: 30s
static_configs:
- targets: ['<hostname1>:8888','<hostname2>:8888']
scheme: https
tls_config:
cert_file: C:\Prometheus\HSM-client.crt
key_file: C:\Prometheus\HSM-client.key
metrics_path: /prometheus
The HSM metrics can be visualized with Grafana. Therefore the Prometheus data source needs to be added to the Grafana configuration.
HSM comes with a preconfigured Grafana dashboard template, that can be used to monitor HSM installations:
<installation-dir>\networking\HSM-grafana-dashboard.json
Import the template file into Grafana: Open Grafana in your browser, and navigate to the Grafana "Home". Open the "+"-menu and click "Import". Next click "Upload .json File" and select the HSM template file.
The template is structured in multiple sections. Each section contains a number of graphs. Graphs are so-called range vectors (a set of time series containing a range of data points over time for each time series, refer to Grafana documentation for further explanations). Range vectors in the template collect data points over the last 180 seconds.
File rate curves distinguish at least the result values success and
failure. An exception is the metric hsm_migrated_files_details, where
only success is split into details for readablitiy reasons.
Curve amplitudes of file rate curves are defined in [files/second].
Byte rate curves relate to the number of bytes on disk for
successful HSM operations.
Curve amplitudes of byte rate curves are defined in [bytes/second].
Overview | |||
---|---|---|---|
hsm_recalled_files, hsm_migrated_files_details | |||
Graph | Originator | Result | Description |
Recall Activity | user | success, failure | HSM alerts and analysis. See chapter Metric Interpretation for details. |
Denied Recall Alerts | user | denied | |
Graph | Originator | Reason | |
Migration Candidate Analysis | job, list, threshold | migration, content freed, content update | |
Recall | |||
hsm_recalled_files, hsm_recalled_bytes | |||
Graph | Originator | Result | Description |
Recalled User Files | user | success, denied, failure | Recalls triggered by end-users. Recalls may be denied due to user recall quotas. |
Recalled User Bytes | user | - | |
Recalled Admin Files | admin, list | success, failure | Recalls initiated by the HSM administrator
(dsmclc recall ,
dsmclc recalllist ) |
Recalled Admin Bytes | admin, list | - | |
Migrate | |||
hsm_migrated_files, hsm_migrated_bytes, hsm_migrated_files_details, hsm_migrated_bytes_details | |||
Graph | Originator | Result | Description |
Migrated Files | job, list, threshold | success, failure | Migrations initiated by the HSM administrator
(dsmclc migrate / dsmgui ,
dsmclc migratelist ,
hsmmonitor ) Detailed graphs split successful migrations into: |
Migrated Bytes | job, list, threshold | - | |
Graph | Originator | Reason | |
Migrated Files Details | job, list, threshold | migration, content freed, content update, acl update, attr update | |
Migrated Bytes Details | job, list, threshold | migration, content freed, content update | |
Backup Before Migrate | |||
hsm_backup_files, hsm_backup_bytes | |||
Graph | Originator | Result | Description |
Backup Before Migrate Files | migration | success, failure | Backup before HSM migration (optional). Remark: Normal daily backup operations performed by using the B/A Client are not counted. |
Backup Before Migrate Bytes | migration | - | |
Retrieve | |||
hsm_retrieved_files, hsm_retrieved_bytes | |||
Graph | Originator | Result | Description |
Retrieved Files | admin | success, failure | Retrievals initiated by the HSM administrator
(dsmclc retrieve / dsmgui ) |
Retrieved Bytes | admin | - | |
Move | |||
hsm_moved_stubs, hsm_moved_backend_files, hsm_moved_backend_bytes | |||
Graph | Originator | Result | Description |
Moved Stubs | admin | success, failure | Stub move operations initiated by the HSM administrator
(dsmmove , hsmtasks )
|
Moved Backend Files | tasks | success, failure | |
Moved Backend Bytes | tasks | - | |
Network Usage | |||
hsm_network_bytes | |||
Graph | Originator | Action | Description |
Network Bytes | job, list, threshold | migration | Network traffic between the file server running HSM and the
Spectrum Protect backend server, caused by the HSM client actions
Remark: Meta data transferred to or from the backend server is neglected. |
migration | backup | ||
admin, list, user | recall | ||
admin | retrieve | ||
tasks | move | ||
Volume Usage | |||
hsm_volume_used_bytes, hsm_volume_free_bytes, hsm_volume_used_percent | |||
Graph | Volume | Description | |
Volume Used Percent | Windows file server volume information. | ||
Volume Free Bytes | |||
Volume Used Bytes |
Metric interpretation is explained here for Grafana curves, but valid for any visualization of the HSM metric curves.
Recall Activity | |
---|---|
![]() |
The HSM administrator can supervise HSM recall activity in this
graph. The Recall Activity graph live-monitors the number of files,
that users are recalling on HSM file servers. If the number of recalls is higher than expected, revise your migration policies and migration jobs. Also see the Migration Candidate Analysis graph below. Curve peaks indicate mass recalls, especially if they occur outside business hours. Mass recalls is a frequently reoccurring HSM problem caused by misconfigured scanners, like Anti-Virus software or indexing software, accessing HSM stub file content unintendedly. |
Denied Recall Alerts | |
![]() |
The Denied Recall Alerts graph alerts the HSM Administrator to intervent,
if quota limits apply. When recall limits exceed, users do not have
access to their stub files. HSM allows to limit the number of recalls per time for users and/or user groups. If the user quota feature is not activated, no curves will be shown. Remark: Check HSM listing files listing\hsmservice-*.log to find out which users are
affected. |
Migration Candidate Analysis | |
![]() |
The Migration Candidate Analysis graph analyses whether files are
frequently re-migrated. Many frequently re-migrated files indicate
that job and/or migration policies include files, that should not
be migrated yet, because they are still in use. Curves with result |