IBM HCI Manual Node Power Operations

This guide explains how to perform node power operations on an IBM HCI (Hardware Converged Infrastructure) cluster manually — without running ocs-ci — using SSH, IPMI, and Redfish directly.

Use this when:

• ocs-ci is unavailable or you want to verify a node outside of a test run

• The cluster API is unreachable and you need to recover nodes

• You want to debug power management issues independently

---

Architecture Overview

Test runner / you
      │
      │  SSH (rack_ssh_username / rack_ssh_password)
      ▼
  MGem / Rack host  (rackIP from rack-details JSON)
      │
      │  IPMI (Lenovo)  or  Redfish/curl (Dell)
      ▼
  Node BMC  (ipv4 / ipv6 from rack-details JSON)

All IPMI and Redfish commands are executed on the MGem, not from your local machine. The node BMC IPs are only routable from the MGem.

---

Step 1 — Find Your Rack Details

The rack-details JSON is saved at:

<DATA_DIR>/rack_details/<cluster_name>.json

DATA_DIR defaults to <ocs-ci-root>/data but can be overridden with the OCSCI_DATA_DIR environment variable.

Backups are also kept at:

• <DATA_DIR>/rack_details/<cluster_name>_backup_<timestamp>.json

• ~/<cluster_name>_rack_backup_<timestamp>.json

If the file does not exist, see Step 1a — Generating the Rack Details File before continuing.

---

Step 1a — Generating the Rack Details File

The file is generated automatically when ocs-ci first initialises IBMHCI. If it is missing entirely (first run, or the file was deleted), build it manually using the steps below.

Build manually (cluster API required)

Use this when the cluster is accessible.

1. Collect BMC IPs from the kickstart ConfigMaps

If the cluster is still partially accessible, extract the data directly:

# List all kickstart configmaps
oc get configmap -n ibm-spectrum-fusion-ns | grep kickstart-

# Dump one configmap (replace kickstart-<rack_serial> with your rack serial)
oc get configmap kickstart-<rack_serial> -n ibm-spectrum-fusion-ns -o jsonpath='{.data}' | python3 -m json.tool

The JSON value inside the configmap contains a computeNodeIntegratedManagementModules array with each node’s OCPRole, ipv4, ipv6, manufacturer, and secretName.

2. Fetch BMC credentials from node secrets

# Get the secret for a node (secretName comes from the configmap above)
oc get secret <secretName> -n ibm-spectrum-fusion-ns \
  -o jsonpath='{.data.isfmgmtUserName}' | base64 -d && echo
oc get secret <secretName> -n ibm-spectrum-fusion-ns \
  -o jsonpath='{.data.isfmgmtUserPasswrd}' | base64 -d && echo

3. Find the rackIP (MGem IP)

The rackIP is not stored in the cluster — it lives in an external GitHub rack-config file. Ask your team/infrastructure owner for the MGem IP for each rack serial. It is the IP address you SSH into in Step 2.

Alternatively, check any existing backup file:

ls -lt ~/<cluster_name>_rack_backup_*.json 2>/dev/null | head -3
cat ~/<cluster_name>_rack_backup_<timestamp>.json | python3 -m json.tool \
  | grep -A2 rackIP

4. Write the JSON file

Create the directory and write the file manually:

mkdir -p <DATA_DIR>/rack_details

cat > <DATA_DIR>/rack_details/<cluster_name>.json <<'EOF'
{
  "<rack_serial>": {
    "nodes": {
      "<node_role>": {
        "ipv4": "<bmc_ipv4>",
        "ipv6": null,
        "manufacturer": "Lenovo",
        "role": "master",
        "username": "<bmc_username>",
        "password": "<bmc_password>"
      }
    },
    "rackInfo": {
      "ibmSerialNumber": "<rack_serial_upper>",
      "ibmMTM": "9155-R42",
      "rackGen": 2,
      "storageType": "fdf",
      "isfModel": "9155",
      "ipStack": "v4",
      "rackIP": "<mgem_ip>"
    }
  }
}
EOF

# Restrict permissions (file contains BMC passwords)
chmod 600 <DATA_DIR>/rack_details/<cluster_name>.json

5. Verify the file is valid JSON

python3 -m json.tool <DATA_DIR>/rack_details/<cluster_name>.json > /dev/null \
  && echo "JSON OK" || echo "JSON INVALID"

6. Verify BMC reachability from the MGem

SSH into the MGem and ping each BMC IP:

ssh <rack_ssh_username>@<rackIP>
ping -c 1 -W 5 <bmc_ipv4>

Once the file exists and the pings succeed, proceed from Step 2.

---

Example config (multi-AZ cluster):

{
  "<rack_serial_1>": {
    "nodes": {
      "<node_role_1>": {
        "ipv4": "<bmc_ipv4_1>",
        "ipv6": null,
        "manufacturer": "Lenovo",
        "role": "master",
        "username": "<bmc_username>",
        "password": "<bmc_password>"
      },
      "<node_role_2>": {
        "ipv4": "<bmc_ipv4_2>",
        "ipv6": null,
        "manufacturer": "Lenovo",
        "role": "worker",
        "username": "<bmc_username>",
        "password": "<bmc_password>"
      }
    },
    "rackInfo": {
      "rackIP": "<mgem_ip_1>"
    }
  },
  "<rack_serial_2>": {
    "nodes": {
      "<node_role_1>": {
        "ipv4": "<bmc_ipv4_3>",
        "ipv6": null,
        "manufacturer": "Lenovo",
        "role": "master",
        "username": "<bmc_username>",
        "password": "<bmc_password>"
      }
    },
    "rackInfo": {
      "rackIP": "<mgem_ip_2>"
    }
  }
}

Key fields used in every operation:

Field	Description
rackIP	IP of the MGem — the SSH hop host
ipv4 / ipv6	BMC IP of the node (reachable only from MGem)
username / password	BMC credentials (IPMI or Redfish)
manufacturer	Lenovo → use IPMI; Dell → use Redfish

---

Step 2 — SSH into the MGem

ssh <rack_ssh_username>@<mgem_ip>

All commands in Steps 3–6 are run inside this SSH session.

---

Step 3 — Verify BMC Reachability (Ping)

Before running any power command, confirm the node BMC is reachable from the MGem.

IPv4:

ping -c 1 -W 5 <bmc_ipv4>

IPv6:

ping6 -c 1 -W 5 <bmc_ipv6>

A successful ping (0% packet loss) confirms the network path is healthy. If ping fails, try another rack’s MGem (see Multi-AZ fallback).

---

Step 4 — Lenovo Nodes (IPMI)

All Lenovo operations use ipmitool via lanplus.

Install ipmitool (if not present)

which ipmitool || yum install -y ipmitool || dnf install -y ipmitool

Power status

ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' power status
# Expected output: "Chassis Power is on"  or  "Chassis Power is off"

IPv6:

ipmitool -I lanplus -H "<bmc_ipv6>" -U <bmc_username> -P '<bmc_password>' power status

Note: IPv6 addresses must be quoted for ipmitool.

Power on

ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' power on

Power off (graceful)

ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' power soft

Power off (force)

ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' power off

Power cycle

ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' power cycle

Power reset

ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' power reset

---

Step 5 — Dell Nodes (Redfish)

All Dell operations use curl against the Redfish REST API.

IPv6 addresses must be wrapped in [brackets] in URLs.

Discover the Systems URI

curl -k -sS -f \
  -u <bmc_username>:'<bmc_password>' \
  https://<bmc_ipv4>/redfish/v1/Systems

From the JSON response, note the @odata.id of the first member, e.g. /redfish/v1/Systems/System.Embedded.1. Use this as <SYSTEM_URI> below.

Power status

curl -k -sS -f \
  -u <bmc_username>:'<bmc_password>' \
  https://<bmc_ipv4><SYSTEM_URI> | grep -i powerstate
# Expected: "PowerState": "On"  or  "PowerState": "Off"

Power on

curl -k -sS -f \
  -u <bmc_username>:'<bmc_password>' -X POST \
  https://<bmc_ipv4><SYSTEM_URI>/Actions/ComputerSystem.Reset \
  -H 'Content-Type: application/json' \
  -d '{"ResetType": "On"}'

Power off (graceful)

curl -k -sS -f \
  -u <bmc_username>:'<bmc_password>' -X POST \
  https://<bmc_ipv4><SYSTEM_URI>/Actions/ComputerSystem.Reset \
  -H 'Content-Type: application/json' \
  -d '{"ResetType": "GracefulShutdown"}'

Power off (force)

curl -k -sS -f \
  -u <bmc_username>:'<bmc_password>' -X POST \
  https://<bmc_ipv4><SYSTEM_URI>/Actions/ComputerSystem.Reset \
  -H 'Content-Type: application/json' \
  -d '{"ResetType": "ForceOff"}'

Power cycle

curl -k -sS -f \
  -u <bmc_username>:'<bmc_password>' -X POST \
  https://<bmc_ipv4><SYSTEM_URI>/Actions/ComputerSystem.Reset \
  -H 'Content-Type: application/json' \
  -d '{"ResetType": "PowerCycle"}'

Power reset (graceful)

curl -k -sS -f \
  -u <bmc_username>:'<bmc_password>' -X POST \
  https://<bmc_ipv4><SYSTEM_URI>/Actions/ComputerSystem.Reset \
  -H 'Content-Type: application/json' \
  -d '{"ResetType": "GracefulRestart"}'

Power reset (force)

curl -k -sS -f \
  -u <bmc_username>:'<bmc_password>' -X POST \
  https://<bmc_ipv4><SYSTEM_URI>/Actions/ComputerSystem.Reset \
  -H 'Content-Type: application/json' \
  -d '{"ResetType": "ForceRestart"}'

---

Step 6 — Power on All Nodes (Recovery)

When the cluster API is unreachable and all nodes are off, power them all on from the MGem without any ocs-ci involvement.

Repeat the following for each rack in your cluster, substituting the rack’s MGem IP and each node’s BMC IP from the rack-details JSON:

ssh <rack_ssh_username>@<mgem_ip>

# Power on each node in this rack (one BMC IP per node)
for BMC_IP in <bmc_ipv4_1> <bmc_ipv4_2> <bmc_ipv4_3>; do
  echo "Powering on $BMC_IP"
  ipmitool -I lanplus -H $BMC_IP -U <bmc_username> -P '<bmc_password>' power on
done

Wait ~5 minutes then verify each node booted:

ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' power status

---

Multi-AZ MGem Fallback

In a multi-AZ cluster each rack has its own MGem (rackIP). If a node’s BMC is not reachable from its own rack’s MGem, try another rack’s MGem.

Look up the rackIP values for all racks from the rack-details JSON, then:

# BMC not reachable from its own rack's MGem?
# SSH into another rack's MGem and try from there:
ssh <rack_ssh_username>@<other_mgem_ip>
ping -c 1 -W 5 <bmc_ipv4>
ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' power status

---

Quick Reference

Operation	Lenovo (IPMI)	Dell (Redfish ResetType)
Status	power status	GET <SYSTEM_URI> → PowerState
Power on	power on	On
Power off graceful	power soft	GracefulShutdown
Power off force	power off	ForceOff
Power cycle	power cycle	PowerCycle
Reset graceful	power reset	GracefulRestart
Reset force	power reset (with force)	ForceRestart

---

Troubleshooting

ipmitool: command not found

yum install -y ipmitool   # RHEL/CentOS
dnf install -y ipmitool   # Fedora/newer RHEL

Error in open session response message : insufficient resources for session Too many open IPMI sessions on the BMC. Close them:

ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' session info all
ipmitool -I lanplus -H <bmc_ipv4> -U <bmc_username> -P '<bmc_password>' session deactivate <session-id>

curl: (22) The requested URL returned error: 401 Wrong Redfish credentials. Verify username/password from the rack-details JSON.

Ping succeeds but IPMI/Redfish fails

• Check credentials in the rack-details JSON (username, password fields).

• Verify ipmitool is installed on the MGem.

• For Dell nodes, ensure the Systems URI is correct by re-running the discovery curl command.

Node BMC not pingable from own MGem

• Try another rack’s MGem (see Multi-AZ MGem Fallback).

• If no MGem can reach the BMC, there is a network-level issue that must be resolved before power operations are possible.