From 111e5da1b5c69a577b1992e15210f0568f81af6f Mon Sep 17 00:00:00 2001 From: Srivatsa Sangli <58572624+sangli-pensando@users.noreply.github.com> Date: Thu, 25 Jun 2026 15:28:22 -0700 Subject: [PATCH] feat: add bmc-einj-enable and ras-inject-test skills for RAS error injection testing (#1557) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: add bmc-einj-enable and ras-inject-test skills for RAS error injection testing Add two new Claude Code skills for hardware RAS error injection testing: 1. /bmc-einj-enable — enables EINJ on AMD GPU OAM slots via BMC Redfish API, discovers OAM slots, POSTs enable action, issues FullPowerCycle, and verifies activation. Supports SMCI BMCs. Tested on MI350X. 2. /ras-inject-test — runs end-to-end RAS error injection tests using amdgpuras, cross-validates ECC counters between amd-smi (ground truth) and the device metrics exporter, collects AFID data, generates structured test reports, and publishes results to Confluence under a release-based page hierarchy. Both skills were validated on smci350-rck-g03-b19-03 (MI350X, 8 GPUs): - GFX and MMHUB blocks: PASS (amd-smi and DME counters match) - PCIe BIF: SKIP (hardware doesn't expose ECC counters) - XGMI/WAFL: FAIL (injection accepted but counters don't increment) - UMC/SDMA: excluded (cause fatal GPU resets) Co-Authored-By: Claude * fix: address Copilot review — placeholder creds, plan alignment, timeout - Replace literal IPs/passwords with placeholders in all examples - Add sshpass security note recommending SSHPASS env var for production - Use discovered Redfish reset target URI instead of hardcoded path - Add timeout to amd-smi metric --ecc-block example command - Align plans with implementation: FullPowerCycle (not GracefulRestart), both OEM paths, Confluence publishing is in-scope (optional) Co-Authored-By: Claude --------- Co-authored-by: Claude (cherry picked from commit 42b51df8c88fde01787cc6961f2c5dae60b5511f) --- .claude/commands/bmc-einj-enable.md | 1 + .claude/commands/ras-inject-test.md | 1 + .claude/skills/README.md | 578 ++++++++++++++++++ .claude/skills/bmc-einj-enable/SKILL.md | 224 +++++++ .claude/skills/ras-inject-test/SKILL.md | 275 +++++++++ .../plans/2026-06-24-bmc-einj-enable-skill.md | 72 +++ .../plans/2026-06-24-ras-inject-test-skill.md | 78 +++ 7 files changed, 1229 insertions(+) create mode 120000 .claude/commands/bmc-einj-enable.md create mode 120000 .claude/commands/ras-inject-test.md create mode 100644 .claude/skills/README.md create mode 100644 .claude/skills/bmc-einj-enable/SKILL.md create mode 100644 .claude/skills/ras-inject-test/SKILL.md create mode 100644 docs-internal/knowledge/plans/2026-06-24-bmc-einj-enable-skill.md create mode 100644 docs-internal/knowledge/plans/2026-06-24-ras-inject-test-skill.md diff --git a/.claude/commands/bmc-einj-enable.md b/.claude/commands/bmc-einj-enable.md new file mode 120000 index 00000000..c8763b6b --- /dev/null +++ b/.claude/commands/bmc-einj-enable.md @@ -0,0 +1 @@ +../skills/bmc-einj-enable/SKILL.md \ No newline at end of file diff --git a/.claude/commands/ras-inject-test.md b/.claude/commands/ras-inject-test.md new file mode 120000 index 00000000..421bbb44 --- /dev/null +++ b/.claude/commands/ras-inject-test.md @@ -0,0 +1 @@ +../skills/ras-inject-test/SKILL.md \ No newline at end of file diff --git a/.claude/skills/README.md b/.claude/skills/README.md new file mode 100644 index 00000000..af5bc128 --- /dev/null +++ b/.claude/skills/README.md @@ -0,0 +1,578 @@ +# GPU Operator Skills + +Project-specific Claude Code skills for GPU Operator development and testing. + +## Available Skills by Operand + +### Testing & Development (by Component) + +#### `/pytest-dev` + +**File**: `pytest-dev.md` | **Component**: Generic + +Generic pytest development skill for AMD GPU testing infrastructure. + +**Use cases**: + +- Implement pytest testcases from approved test plans +- Write new test functions following project patterns +- Debug failing tests from CI job logs +- Understand test infrastructure and fixture relationships +- Navigate the test codebase +- Ensure cross-platform compatibility (K8s + OpenShift) + +**Example**: + +```bash +/pytest-dev Implement test scenarios from approved test plan at path/to/test-plan.md +/pytest-dev Debug the failures in job log 30158283 +``` + +--- + +#### `/pytest-dcm-dev` + +**File**: `pytest-dcm-dev.md` | **Component**: Device Config Manager + +Device Config Manager (DCM) GPU partition testing. + +**Use cases**: + +- DCM partition testing (SPX, DPX, QPX, CPX with NPS1/NPS2/NPS4) +- Label verification across multiple nodes +- Driver reload timing validation +- Cleanup patterns for failed tests +- Partition profile file validation + +**Test files**: + +- `tests/pytests/k8/gpu-operator/test_config_manager.py` +- `tests/pytests/k8/gpu-operator/test_multi_deviceconfig.py` + +**Example**: + +```bash +/pytest-dcm-dev Add partition tests for SPX mode +/pytest-dcm-dev Debug label verification failure +``` + +--- + +#### `/pytest-dme-dev` + +**File**: `pytest-dme-dev.md` | **Component**: Device Metrics Exporter + +Device Metrics Exporter (DME) testing - Prometheus metrics collection and validation. + +**Use cases**: + +- Metrics exporter deployment testing (Helm and standalone) +- Validate metric values against AMD-SMI ground truth +- Troubleshoot metric endpoint connectivity (ClusterIP vs NodePort) +- Debug metric collection and comparison logic +- Prometheus metric format and labeling validation + +**Test files**: + +- `tests/pytests/k8/gpu-operator/test_metrics_exporter.py` +- `tests/pytests/k8/exporter/test_metrics_exporter.py` +- `tests/pytests/k8/gpu-operator/test_metrics_values.py` + +**Example**: + +```bash +/pytest-dme-dev Implement metric validation tests from test plan +/pytest-dme-dev Debug metric endpoint connectivity issue +``` + +--- + +#### `/pytest-npd-dev` + +**File**: `pytest-npd-dev.md` | **Component**: Node Problem Detector + +Node Problem Detector (NPD) integration testing with amdgpuhealth plugin. + +**Use cases**: + +- NPD amdgpuhealth plugin testing +- ConfigMap/DaemonSet volume mounting +- Metric endpoint validation +- NPD condition monitoring +- Platform coverage (K8s + OpenShift) + +**Test files**: + +- `tests/pytests/k8/gpu-operator/test_node_problem_detector.py` +- `tests/pytests/k8/exporter/test_node_problem_detector.py` + +**Example**: + +```bash +/pytest-npd-dev Implement NPD condition monitoring tests +/pytest-npd-dev Debug ConfigMap mounting issue +``` + +--- + +#### `/pytest-anr-dev` + +**File**: `pytest-anr-dev.md` | **Component**: Auto Node Remediation + +Auto Node Remediation (ANR) testing - Argo Workflow-based node remediation. + +**Use cases**: + +- ANR deployment and workflow testing +- Validate remediation workflows (driver reload, node reboot, drain) +- Troubleshoot Argo Workflow execution +- Debug NHC (Node Health Check) integration +- Custom ConfigMap remediation templates +- K8s only (not supported on OpenShift) + +**Test files**: + +- `tests/pytests/k8/gpu-operator/test_node_remediation.py` +- `tests/pytests/k8/gpu-operator/test_anr_deployment.py` + +**Example**: + +```bash +/pytest-anr-dev Implement workflow execution tests from test plan +/pytest-anr-dev Debug workflow stuck in Running phase +``` + +**Note**: ANR is K8s-only, not supported on OpenShift + +--- + +#### `/pytest-driver-dev` + +**File**: `pytest-driver-dev.md` | **Component**: ROCm/AMDGPU Driver + +ROCm/AMDGPU driver version management and testing. + +**Use cases**: + +- Driver deployment testing (deviceconfig vs inbox) +- Driver version upgrades and downgrades +- Validate driver blacklist configuration +- Test driver-deviceplugin integration +- Debug KMM (Kernel Module Management) workflows +- Manage driver spec files for different ROCm versions + +**Test files**: + +- `tests/pytests/k8/gpu-operator/test_driver_deviceplugin.py` + +**Example**: + +```bash +/pytest-driver-dev Debug KMM worker failure +/pytest-driver-dev Update driver spec for new ROCm 6.4 +``` + +--- + +#### `/pytest-upgrade-dev` + +**File**: `pytest-upgrade-dev.md` | **Component**: GPU Operator & Operand Upgrades + +GPU Operator and operand upgrade testing from released versions to RC builds. + +**Use cases**: + +- GPU operator helm upgrade testing (base → RC) +- Operand upgrade workflows (RollingUpdate/OnDelete) +- Multi-version upgrade matrix testing +- Upgrade hook and CRD patching validation +- K8s only (Helm-based, not OpenShift OLM) + +**Test files**: + +- `tests/pytests/k8/gpu-operator/upgrade/*.py` + +**Job config**: `tests/jobs/upgrade/.job.yml` + +**Example**: + +```bash +/pytest-upgrade-dev Implement upgrade from v1.5.0 to RC +/pytest-upgrade-dev Debug helm upgrade hook failure +``` + +**Note**: K8s-only, not applicable to OpenShift OLM + +--- + +### Test Execution + +#### `/run-tests` + +**File**: `run-tests.md` | **Component**: Test Launcher + +Run GPU Operator pytest test suites against a cluster using `k8_test_launcher.sh`. + +**Use cases**: + +- Execute full test suites against K8s/OpenShift clusters +- Run specific test modules or individual test cases +- Generate test reports (HTML/XML) for CI/CD integration +- Collect tech-support on test failures +- Test with specific driver versions or workloads + +**Example**: + +```bash +# Run all GPU operator tests +/run-tests gpu-operator --manifest tests/pytests/image-manifest/mi350_collab_images.yaml + +# Run specific module +/run-tests gpu-operator --manifest --module installation + +# Run with debug mode +/run-tests gpu-operator --manifest --module metrics --testcase values --debug +``` + +**Output**: Test reports in `logs/` directory (HTML, XML, test logs) + +--- + +### Test Analysis & Triage + +Component-focused analysis skills that perform deep correlation and root cause analysis: + +#### `/analyze-dme` + +**File**: `analyze-dme.md` | **Component**: Device Metrics Exporter + +Deep analysis of Device Metrics Exporter issues by correlating AMD-SMI output, gpuagent connectivity, and exporter endpoint metrics. + +**Use cases**: + +- Diagnose missing metrics (temperature, activity, power, etc.) +- Correlate AMD-SMI vs exporter output to identify data flow breaks +- Check gpuagent socket connectivity +- Identify product bugs vs configuration issues vs test bugs +- Validate metrics against metrics-support.json + +**Analysis Flow**: AMD-SMI → GPUAgent → Exporter → Prometheus Endpoint + +**Example**: + +```bash +# Analyze CI run +/analyze-dme --target-id 30883642 + +# Analyze local test +/analyze-dme --logs-path tests/pytests/logs/ + +# With verbose output +/analyze-dme --verbose --output dme-report.json +``` + +**Output**: Root cause categorization (PRODUCT_BUG | CONFIG_ISSUE | TEST_BUG | HEALTHY) + +--- + +#### `/analyze-npd` *(Planned)* + +**Component**: Node Problem Detector + +Analyze NPD failures: +- ConfigMap validation +- Problem detection rules +- Kubernetes event reporting +- Metric endpoint health + +--- + +#### `/analyze-anr` *(Planned)* + +**Component**: Auto Node Remediation + +Analyze ANR workflow failures: +- Argo Workflow execution +- Remediation action success/failure +- NHC integration +- Custom remediation templates + +--- + +#### `/analyze-driver` *(Planned)* + +**Component**: ROCm/AMDGPU Driver + +Analyze driver failures: +- KMM worker issues +- Driver loading/unloading +- Version compatibility +- Firmware issues + +--- + +#### `/triage` + +**File**: `triage.md` + +Categorize test failures as product bugs, infrastructure issues, or test code bugs. + +**Use cases**: + +- Triage all failures from a test run +- Categorize: PRODUCT_BUG | INFRASTRUCTURE_ISSUE | TEST_BUG +- Generate bug reports for product issues +- Identify test improvements needed + +**Input**: Results from component analysis skills (`/analyze-dme`, etc.) + +**Example**: + +```bash +# Triage all failures from CI run +/triage --target-id 30883642 + +# Triage with component analysis +/triage --logs-path tests/pytests/logs/ --deep-analysis +``` + +**Output**: Categorized failures with root causes and next steps + +--- + +### Release Engineering + +#### `/gen-image-manifest` + +**File**: `gen-image-manifest.md` | **Component**: Image Manifests + +Generate or update image manifest YAML files with latest builds from assets-hq.pensando.io. + +**Use cases**: + +- Update existing manifest with latest RC builds for a branch +- Generate new manifest for a new release branch +- Query build servers for latest gpu-operator, device-metrics-exporter, device-config-manager, kernel-module-management builds + +**Manifest files**: `tests/pytests/image-manifest/*.yaml` + +**Example**: + +```bash +# Update existing manifest +/gen-image-manifest tests/pytests/image-manifest/mi350_collab_images.yaml --branch collab-7.12 + +# Generate new manifest +/gen-image-manifest tests/pytests/image-manifest/v1.6.0_rc_images.yaml --branch v1.6.0 +``` + +--- + +### Planning + +#### `/test-plan-dev` + +**File**: `test-plan-dev.md` + +Generates comprehensive test plans from Product Requirement Documents (PRDs). + +**Use cases**: + +- Analyze PRDs and extract testable requirements +- Create structured test plans with scenario mapping +- Define test coverage matrix and priorities +- Identify test data and environment requirements +- Generate reviewable test plan documents + +**Workflow**: Use this FIRST before implementing tests + +**Example**: + +```bash +/test-plan-dev Generate test plan from PRD at prds/2026/Q2/PRD-GPU-20260406-01.md +``` + +--- + +### Hardware / BMC + +#### `/bmc-einj-enable` + +**File**: `bmc-einj-enable/SKILL.md` | **Component**: BMC / RAS + +Enable EINJ (Error Injection) on AMD GPU OAM slots via BMC Redfish API. Pre-requisite for `amdgpuras` RAS error injection testing. + +**Use cases**: + +- Enable EINJ on SMCI BMC before running `amdgpuras` RAS tests +- Discover which OAM slot(s) support EINJ +- Power cycle the host via Redfish to activate EINJ +- Verify EINJ state after reboot + +**Supported BMC vendors**: SMCI (Supermicro) — other vendors may not expose the OEM endpoint. + +**Example**: + +```bash +/bmc-einj-enable +``` + +--- + +#### `/ras-inject-test` + +**File**: `ras-inject-test/SKILL.md` | **Component**: RAS / ECC Testing + +Run end-to-end RAS error injection tests on AMD GPUs. Injects errors via `amdgpuras`, verifies ECC counters in `amd-smi` (ground truth) and device-metrics-exporter, collects AFID data, and generates a test report. + +**Use cases**: + +- Validate DME ECC metric accuracy against amd-smi after hardware error injection +- Test all injectable blocks (UMC, SDMA, GFX, MMHUB, PCIe, XGMI) across all GPUs +- Collect AFID data and correlate with `gpu_afid_errors` metric +- Generate structured test reports for Confluence upload + +**Prerequisites**: EINJ enabled (`/bmc-einj-enable`), `amdgpuras` installed, DME running on host. + +**Example**: + +```bash +/ras-inject-test +/ras-inject-test --release v1.5.2 +/ras-inject-test --release v1.5.1 --blocks gfx,mmhub,pcie_bif +``` + +--- + +## Component-to-Skill Mapping + +| GPU Operator Component | Development Skill | Analysis Skill | Test Location | +|---------------------------------|-----------------------|--------------------|---------------------------------------------| +| Device Config Manager (DCM) | `/pytest-dcm-dev` | *(planned)* | `k8/gpu-operator/test_config_manager.py` | +| Device Metrics Exporter (DME) | `/pytest-dme-dev` | `/analyze-dme` | `k8/gpu-operator/test_metrics_exporter.py` | +| Node Problem Detector (NPD) | `/pytest-npd-dev` | *(planned)* | `k8/gpu-operator/test_node_problem_detector.py` | +| Auto Node Remediation (ANR) | `/pytest-anr-dev` | *(planned)* | `k8/gpu-operator/test_node_remediation.py` | +| ROCm/AMDGPU Driver | `/pytest-driver-dev` | *(planned)* | `k8/gpu-operator/test_driver_deviceplugin.py` | +| **Operator & Operand Upgrades** | `/pytest-upgrade-dev` | - | `k8/gpu-operator/upgrade/*.py` | +| Generic / Multi-component | `/pytest-dev` | `/analyze-results` | Any test file | +| **Test Execution** | `/run-tests` | - | Runs `k8_test_launcher.sh` | +| **Image Manifests** | `/gen-image-manifest` | - | `image-manifest/*.yaml` | +| **Failure Triage** | - | `/triage` | All failures | + +## Workflow + +### Development Workflow + +The typical test development workflow: + +1. **Planning** → `/test-plan-dev` from PRD +2. **Review** → Human approval of test plan +3. **Select Component Skill** → Choose based on component being tested: + - DCM partition tests → `/pytest-dcm-dev` + - Metrics exporter tests → `/pytest-dme-dev` + - NPD integration tests → `/pytest-npd-dev` + - ANR workflow tests → `/pytest-anr-dev` + - Generic/multi-component → `/pytest-dev` +4. **Implementation** → Use selected skill to implement tests from approved plan +5. **Execution** → `/run-tests` to run tests against cluster +6. **Debugging** → Use same component skill with job log reference + +**Example Development Workflow**: + +```bash +# Step 1: Generate test plan +/test-plan-dev Generate test plan from PRD at .claude/knowledge/prds/2026/Q2/PRD-GPU-20260406-01.md + +# Step 2: Human reviews and approves test plan + +# Step 3: Implement tests for specific component +/pytest-dme-dev Implement metric validation tests from approved plan at test-plan.md + +# Step 4: Debug failures if needed +/pytest-dme-dev Debug metric comparison failures in job log 30158283 +``` + +--- + +### Analysis Workflow + +When tests fail in CI, use component-focused analysis skills: + +1. **Generic Analysis** → `/analyze-results --target-id ` + - Parses test reports, extracts failures + - Identifies which components failed + - Routes to component-specific analysis + +2. **Component Deep-Dive** → Use focused skill based on failing component: + - DME failures → `/analyze-dme --target-id ` + - NPD failures → `/analyze-npd --target-id ` *(planned)* + - ANR failures → `/analyze-anr --target-id ` *(planned)* + - Driver failures → `/analyze-driver --target-id ` *(planned)* + +3. **Triage** → `/triage --target-id ` + - Uses component analysis results + - Categorizes as: PRODUCT_BUG | INFRASTRUCTURE_ISSUE | TEST_BUG + - Generates bug reports and next steps + +**Example Analysis Workflow**: + +```bash +# Step 1: Download CI logs and quick overview +/analyze-results 30883642 + +# Output shows: 21 DME failures, 12 NPD failures, 8 test-runner failures + +# Step 2: Deep-dive into DME issues +/analyze-dme --target-id 30883642 + +# Output: ROOT CAUSE - GPUAgent socket not accessible (PRODUCT_BUG) + +# Step 3: Triage all failures +/triage --target-id 30883642 + +# Output: Categorized failures with bug report templates +``` + +**Why Component-Focused Analysis?** + +Generic `/analyze-results` finds "21 metrics tests failed" but doesn't know WHY. + +Component-focused `/analyze-dme`: +- ✅ Checks AMD-SMI output (GPU layer) +- ✅ Checks gpuagent connectivity (collection layer) +- ✅ Checks exporter endpoint (exposition layer) +- ✅ Correlates with metrics-support.json (expected vs actual) +- ✅ Identifies exact root cause: "gpuagent socket not available" +- ✅ Categorizes: PRODUCT_BUG vs CONFIG_ISSUE vs TEST_BUG + +**Each component skill is an expert** with deep domain knowledge of that subsystem! + +## Creating New Skills + +To add a new skill: + +1. Create a new `.md` file with frontmatter: + +```markdown +--- +name: skill-name +description: Brief description +--- + +# Skill Name + +Instructions for Claude... +``` + +1. Add to this README +2. Commit to make available project-wide + +## Invoking Skills + +Skills are invoked with the `/` prefix: + +```bash +/skill-name [arguments] +``` + +Claude Code will load the skill and execute according to its instructions. diff --git a/.claude/skills/bmc-einj-enable/SKILL.md b/.claude/skills/bmc-einj-enable/SKILL.md new file mode 100644 index 00000000..e297aae4 --- /dev/null +++ b/.claude/skills/bmc-einj-enable/SKILL.md @@ -0,0 +1,224 @@ +--- +name: bmc-einj-enable +description: Enable EINJ (Error Injection) on AMD GPU OAM slots via BMC Redfish API. Pre-requisite for amdgpuras RAS error injection testing. Currently supports SMCI BMC only. +--- + +You are a skill that enables hardware Error Injection (EINJ) on AMD GPU OAM slots via the BMC Redfish API. This is a prerequisite before `amdgpuras` can inject RAS errors on the host GPU. + +# Invocation + +``` +/bmc-einj-enable +``` + +Example: +``` +/bmc-einj-enable +``` + +All three arguments are required. The BMC must be network-reachable from the machine running this skill. + +# Supported BMC Vendors + +- **SMCI (Supermicro)** — confirmed working +- Other vendors may not expose the `EINJState` / `Chassis.ErrInjection` Redfish OEM endpoint + +# Procedure + +Work through the steps below in order. Use `curl -k` (insecure TLS) for all Redfish calls since BMC certificates are typically self-signed. + +## Step 1 — Discover which OAM slot supports EINJ + +Iterate through OAM_0 to OAM_7 and check which slot(s) expose an `EINJState` field. Not all OAM slots on every server support EINJ. + +The `EINJState` field location varies by BMC firmware. Check both known paths: +- `Oem.EINJState` (seen on newer SMCI firmware) +- `Oem.Ami.AMD.EINJState` (seen on older SMCI firmware) + +For each slot X in [0..7]: +```bash +curl -sk -u ":" \ + "https:///redfish/v1/Chassis/OAM_${X}" \ + | python3 -c " +import sys, json +try: + data = json.load(sys.stdin) + if 'error' in data: + print(f'OAM_{sys.argv[1]}: not found') + sys.exit(0) + oem = data.get('Oem', {}) + # Try direct path first (newer firmware), then nested path (older firmware) + einj = oem.get('EINJState') + if einj is None: + einj = oem.get('Ami', {}).get('AMD', {}).get('EINJState') + if einj is not None: + boot_einj = oem.get('BootEINJState', 'N/A') + print(f'OAM_{sys.argv[1]}: EINJState={einj}, BootEINJState={boot_einj}') + else: + print(f'OAM_{sys.argv[1]}: no EINJState field') +except Exception as e: + print(f'OAM_{sys.argv[1]}: error - {e}') +" "${X}" +``` + +Collect all OAM slots that return an `EINJState` value. If none are found, report failure: +``` +ERROR: No OAM slots with EINJState found on BMC . +This BMC may not support EINJ via Redfish (only SMCI BMCs are known to work). +``` + +## Step 2 — Check current EINJ state + +For each OAM slot that has `EINJState`: + +- If `EINJState` is `"Enabled"` or `"Enable"` — already enabled, report and skip to summary. +- If `EINJState` is `"Disabled"` or `"Disable"` — proceed to Step 3 to enable it. +- If `EINJState` contains `"pending"` (e.g. `"pending to enable"`) — a power cycle is pending. Report this and skip to Step 4. + +Note: state string format varies by firmware — match case-insensitively. + +## Step 3 — Enable EINJ + +For each OAM slot where `EINJState` is disabled, POST the enable action: + +```bash +curl -sk -u ":" \ + -X POST \ + "https:///redfish/v1/Chassis/OAM_${X}/Actions/Oem/AMD/Chassis.ErrInjection" \ + -H "Content-Type: application/json" \ + -d '{"ErrInjection": "Enable"}' +``` + +After the POST, re-query the OAM slot to verify the state changed: + +```bash +curl -sk -u ":" \ + "https:///redfish/v1/Chassis/OAM_${X}" \ + | python3 -c " +import sys, json +data = json.load(sys.stdin) +oem = data.get('Oem', {}) +einj = oem.get('EINJState') or oem.get('Ami', {}).get('AMD', {}).get('EINJState') +print(f'OAM_{sys.argv[1]}: EINJState={einj}') +" "${X}" +``` + +Expected: `EINJState` should now show `"pending to enable"`. + +If the POST returns an error (HTTP 4xx/5xx), report the full response body — common causes: +- BMC firmware too old +- OAM slot in a failed state +- Insufficient BMC user privileges (need admin role) + +## Step 4 — Power cycle via Redfish (FullPowerCycle required) + +EINJ enablement only takes effect after a **full power cycle** (cold boot). A GracefulRestart (warm reboot) is NOT sufficient — the GPU/OAM firmware needs a complete power-off/power-on cycle. + +First, discover the system reset endpoint: + +```bash +curl -sk -u ":" \ + "https:///redfish/v1/Systems/1" \ + | python3 -c " +import sys, json +data = json.load(sys.stdin) +actions = data.get('Actions', {}) +reset = actions.get('#ComputerSystem.Reset', {}) +target = reset.get('target', 'NOT FOUND') +allowed = reset.get('ResetType@Redfish.AllowableValues', []) +power_state = data.get('PowerState', 'Unknown') +print(f'PowerState: {power_state}') +print(f'Reset target: {target}') +print(f'Allowed types: {allowed}') +" +``` + +The system URI may be `/redfish/v1/Systems/1` or `/redfish/v1/Systems/Self` — try both if the first returns 404. + +**Ask the user for confirmation before power cycling**, then issue a `FullPowerCycle` using the discovered reset target URI from above: + +```bash +curl -sk -u ":" \ + -X POST \ + "https://" \ + -H "Content-Type: application/json" \ + -d '{"ResetType": "FullPowerCycle"}' +``` + +Use the `target` value from the discovery step (e.g. `/redfish/v1/Systems/1/Actions/ComputerSystem.Reset`). Do NOT hardcode the Systems URI. + +If `FullPowerCycle` is not in the allowed values, fall back to `PowerCycle`, then `ForceRestart` as a last resort (but warn that it may not activate EINJ). + +After issuing the reset, report: + +``` +=== Power Cycle Initiated === + +BMC: +Reset Type: FullPowerCycle +OAM Slot(s): OAM_ [list all slots that were enabled] + +The host is rebooting. This typically takes 5-10 minutes for GPU servers. +``` + +## Step 5 — Post-reboot verification + +Poll the BMC every 60 seconds to check if the host is back. Wait at least 3 minutes before checking EINJ state (GPU firmware needs time to initialize after power-on): + +```bash +curl -sk -u ":" \ + "https:///redfish/v1/Systems/1" \ + | python3 -c " +import sys, json +data = json.load(sys.stdin) +state = data.get('PowerState', 'Unknown') +print(f'PowerState: {state}') +" +``` + +Once `PowerState` is `"On"` (and at least 3 minutes have elapsed), re-query the OAM slot(s): + +```bash +curl -sk -u ":" \ + "https:///redfish/v1/Chassis/OAM_${X}" \ + | python3 -c " +import sys, json +data = json.load(sys.stdin) +oem = data.get('Oem', {}) +einj = oem.get('EINJState') or oem.get('Ami', {}).get('AMD', {}).get('EINJState') +print(f'OAM_{sys.argv[1]}: EINJState={einj}') +" "${X}" +``` + +Expected: `EINJState = "Enabled"`. + +Print the final summary: + +``` +=== EINJ Enable Complete === + +BMC: +OAM Slot(s): OAM_ +EINJState: Enabled + +Host is powered on and EINJ is active. + +=== Next Steps (on the GPU host) === + +SSH to the GPU host and run amdgpuras to inject RAS errors: + sudo amdgpuras -l # list available GPUs + sudo amdgpuras -d 0 -b 2 -s 0 -t 4 # inject UE on GPU 0, GFX block + +Note: amdgpuras must be run directly on the GPU host, not remotely. +``` + +If `EINJState` is still `"pending to enable"` after reboot, the reset may not have completed a full cold boot. Try `FullPowerCycle` again, or use the BMC WebUI to power down then power up. + +# Important Notes + +- **Minimize user prompts.** Do NOT ask for user confirmation on read-only queries (Steps 1, 2, 5). Only ask for confirmation before state-changing actions: enabling EINJ (Step 3) and issuing the power cycle (Step 4). Run discovery and verification steps automatically without prompting. +- **SMCI-only.** If the Redfish endpoint returns 404 or lacks `EINJState`, the BMC vendor likely does not support this OEM extension. +- **FullPowerCycle required.** GracefulRestart (warm reboot) does NOT activate EINJ — the GPU OAM firmware only picks up the change on a cold boot. +- **Admin credentials required.** The BMC user must have administrator privileges to POST the enable action. +- **Idempotent.** Re-running the skill when EINJ is already enabled is safe — it will detect the `"Enabled"` state and report no action needed. +- **amdgpuras is host-only.** This skill only configures the BMC via Redfish. The actual RAS error injection (`amdgpuras`) must be run by the user directly on the GPU host via SSH. diff --git a/.claude/skills/ras-inject-test/SKILL.md b/.claude/skills/ras-inject-test/SKILL.md new file mode 100644 index 00000000..336e0687 --- /dev/null +++ b/.claude/skills/ras-inject-test/SKILL.md @@ -0,0 +1,275 @@ +--- +name: ras-inject-test +description: Run RAS error injection tests on AMD GPUs via amdgpuras, verify ECC metrics in both amd-smi (ground truth) and device-metrics-exporter, collect AFID data, and generate a test report with Confluence publishing. Requires EINJ enabled, amdgpuras installed, and DME running on the target host. +--- + +You are a skill that performs end-to-end RAS (Reliability, Availability, Serviceability) error injection testing on AMD GPUs. You inject errors via `amdgpuras`, verify ECC counters in both `amd-smi` (ground truth) and the device-metrics-exporter (DME), collect AFID data, and generate a structured test report published to Confluence. + +# Invocation + +``` +/ras-inject-test [--release v1.5.1] [--metrics-port 5000] [--num-gpus auto] [--blocks gfx,mmhub] +``` + +Example: +``` +/ras-inject-test +/ras-inject-test --release v1.5.2 +/ras-inject-test --release v1.5.1 --blocks gfx,mmhub,pcie_bif,xgmi_wafl +/ras-inject-test --metrics-port 5000 --num-gpus 4 +``` + +All SSH commands use `sshpass -p ssh -o StrictHostKeyChecking=no @`. All `amdgpuras` and `amd-smi` commands require `sudo`. + +**Security note:** `sshpass -p` exposes the password via `ps`. For production use, prefer `SSHPASS` env var with `sshpass -e`, or SSH key-based auth. The `-p` form is acceptable for lab/test environments. + +Default metrics port is 5000. If `--num-gpus` is `auto` (default), detect GPU count from the host. Default `--blocks` is `gfx,mmhub` (the two reliably testable blocks). Never include `umc` or `sdma` unless explicitly requested — they cause fatal GPU resets. + +Default `--release` is the current GPU operator release. This controls which Confluence parent page the results are published under. + +# Prerequisites + +Check ALL of these at the start. If any fail, report clearly and stop: + +```bash +# 1. SSH connectivity +sshpass -p "" ssh -o StrictHostKeyChecking=no @ "hostname" + +# 2. amdgpu driver loaded +sshpass ... "lsmod | grep -w amdgpu" + +# 3. amdgpuras installed +sshpass ... "which amdgpuras" + +# 4. amd-smi installed +sshpass ... "which amd-smi" + +# 5. DME serving metrics +sshpass ... "curl -sf http://localhost:/metrics | head -1" +``` + +If prerequisites fail, print which ones failed and suggest fixes: +- No driver: "Run ansible-playbook install-amdgpu-driver.yml or modprobe amdgpu" +- No amdgpuras: "Install from http://10.67.79.109/artifactory/linux-ci-generic-local/amdgpuras-tool/" +- No DME: "Start device-metrics-exporter container or deploy via GPU operator" +- EINJ not enabled: "Run /bmc-einj-enable first" + +# Block-to-Metric Mapping + +This maps amdgpuras block IDs to both amd-smi per-block output and DME Prometheus metrics: + +| Block ID | Name | amdgpuras flags | amd-smi per-block field | DME UE metric | DME CE metric | Risk | +|----------|-----------|-------------------------|--------------------------------------|--------------------------------|-------------------------------|------| +| 0 | umc | `-b 0 -s 0 -t 4 -a 0x800000000` | `UMC: UNCORRECTABLE_COUNT` | `gpu_ecc_uncorrect_umc` | `gpu_ecc_correct_umc` | Fatal reset | +| 1 | sdma | `-b 1 -s 0 -t 4` | `SDMA: UNCORRECTABLE_COUNT` | `gpu_ecc_uncorrect_sdma` | `gpu_ecc_correct_sdma` | Fatal reset | +| 2 | gfx | `-b 2 -s 0 -t 4` | `GFX: UNCORRECTABLE_COUNT` | `gpu_ecc_uncorrect_gfx` | `gpu_ecc_correct_gfx` | Safe | +| 3 | mmhub | `-b 3 -s 0 -t 4` | `MMHUB: UNCORRECTABLE_COUNT` | `gpu_ecc_uncorrect_mmhub` | `gpu_ecc_correct_mmhub` | Safe | +| 5 | pcie_bif | `-b 5 -s 1 -m 2 -t 4` | `PCIE_BIF: UNCORRECTABLE_COUNT` | `gpu_ecc_uncorrect_bif` | `gpu_ecc_correct_bif` | No HW counters | +| 7 | xgmi_wafl | `-b 7 -s 0 -t 4` | `XGMI_WAFL: UNCORRECTABLE_COUNT` | `gpu_ecc_uncorrect_xgmi_wafl` | `gpu_ecc_correct_xgmi_wafl` | Unreliable | + +Also track totals: `gpu_ecc_uncorrect_total`, `gpu_ecc_correct_total`, `gpu_health`, `gpu_afid_errors`. + +**Block-specific notes:** +- **UMC (b=0)**: Requires address flag `-a 0x800000000`. Causes fatal GPU reset — all GPUs enter "resuming" state requiring FullPowerCycle. +- **SDMA (b=1)**: Also causes fatal GPU reset. Exclude unless explicitly requested. +- **GFX (b=2)**: Reliably works. No GPU reset. Generates AFID 30 (FATAL). +- **MMHUB (b=3)**: Works when GPU is not recovering. Also generates AFID 30. May fail with "device resuming" if injected too soon after another block. +- **PCIe BIF (b=5)**: Requires `-s 1 -m 2` (sub-block 1, method ecrc_tx). amd-smi reports `N/A` for this block — hardware does not expose ECC counters via RAS sysfs. Injection is accepted but not observable. Mark as SKIP. +- **XGMI/WAFL (b=7)**: Injection accepted on specific XGMI link but counters don't increment. May need different injection method or firmware support. Mark as FAIL if counters don't change. + +# Procedure + +## Step 1 — Gather System Info + +Collect ALL of the following from the host and display as a table: + +```bash +# Hostname, kernel, OS +sshpass ... "hostname && uname -r && grep PRETTY_NAME /etc/os-release" + +# Driver version +sshpass ... "cat /sys/module/amdgpu/version" + +# amd-smi version (includes ROCm version) +sshpass ... "amd-smi version 2>&1 | head -1" + +# amdgpuras version +sshpass ... "dpkg -l amdgpuras 2>/dev/null | grep amdgpuras || rpm -qa amdgpuras 2>/dev/null" + +# GPU count +sshpass ... "ls /dev/dri/renderD* 2>/dev/null | wc -l" + +# GPU model from metrics +sshpass ... "curl -s http://localhost:/metrics | grep -m1 'card_model=' | sed 's/.*card_model=\"\([^\"]*\)\".*/\1/'" + +# GPU inventory (BDF, UUID, serial) +sshpass ... "sudo amd-smi list 2>&1" + +# DME container version +sshpass ... "sudo docker ps --filter name=device-metrics-exporter --format '{{.Image}}' 2>/dev/null || echo 'not docker'" + +# ROCm firmware info +sshpass ... "sudo amd-smi version 2>&1" +``` + +Print summary: +``` +=== System Info === +Host: () +Kernel: +OS: +Driver: amdgpu +ROCm: +amd-smi: +amdgpuras: +GPUs: × +DME: +``` + +## Step 2 — Discover Injectable Blocks + +Run `amdgpuras -l` on the host. Build the command table based on the `--blocks` argument, filtered to only blocks that exist in the device listing. + +Default safe blocks: `gfx,mmhub` + +Reference: `tests/pytests/scripts/amdgpuras_util.py:get_amdgpuras_valid_command_list()` for parsing patterns. + +## Step 3 — Test Loop (per GPU, per block, sequentially) + +**CRITICAL: Inject ONE block at a time per GPU, verify, wait for recovery, then move to the next. Stacking injections causes GPU resets that clear counters before they can be read.** + +For each GPU (0 to N-1), for each requested block: + +### 3a. Capture Baseline (three sources) + +**amd-smi per-block ECC (ground truth):** +```bash +sshpass ... "timeout 20 sudo amd-smi metric --ecc-block --gpu 2>&1" +``` +Parse the per-block output to extract `: UNCORRECTABLE_COUNT` and `CORRECTABLE_COUNT`. This command can be slow (10-15s); the `timeout 20` prevents hangs. + +**amd-smi aggregate ECC:** +```bash +sshpass ... "sudo amd-smi metric --ecc --gpu 2>&1" +``` +Extract `TOTAL_UNCORRECTABLE_COUNT`, `TOTAL_CORRECTABLE_COUNT`, `CACHE_UNCORRECTABLE_COUNT`. + +**amd-smi AFID:** +```bash +sshpass ... "sudo amd-smi ras --cper --gpu= --folder /tmp/.amdsmi_afid_temp 2>&1" +``` +Count the number of CPER entries for diff later. + +**DME per-block metrics:** +```bash +sshpass ... "curl -s http://localhost:/metrics | grep '^gpu_ecc_uncorrect_{' | grep 'gpu_id=\"\"'" +sshpass ... "curl -s http://localhost:/metrics | grep '^gpu_ecc_uncorrect_total{' | grep 'gpu_id=\"\"'" +sshpass ... "curl -s http://localhost:/metrics | grep '^gpu_health{' | grep 'gpu_id=\"\"'" +``` + +### 3b. Check GPU Ready + +```bash +sshpass ... "sudo amdgpuras -l 2>&1 | grep -c 'resuming'" +``` +If any device shows "resuming", wait up to 120 seconds polling every 10 seconds. If still stuck, the host needs a reboot (see Important Notes). + +### 3c. Inject Error + +```bash +sshpass ... "sudo amdgpuras -d -b -s [-m ] -t [-a 0x800000000]" +``` + +Record the injection result. Check output for: +- `"Error Inject Successfully"` → injection OK +- `"Error Inject Failed"` or `"resuming"` → record as failed +- `"Invalid injection parameters"` → missing `-m` method flag, retry with method + +### 3d. Wait for Metrics Update + +Wait 35 seconds for the exporter to poll and update metrics. + +### 3e. Capture Post-Injection (same three sources as 3a) + +### 3f. Cross-Validate and Record Result + +Compare baseline vs post-injection across ALL sources: + +1. **amd-smi per-block diff**: Did the specific block's `UNCORRECTABLE_COUNT` increment? +2. **amd-smi total diff**: Did `TOTAL_UNCORRECTABLE_COUNT` increment? +3. **DME per-block diff**: Did `gpu_ecc_uncorrect_` increment? +4. **DME total diff**: Did `gpu_ecc_uncorrect_total` increment? +5. **Health**: Did `gpu_health` change from 1 to 0? +6. **AFID**: Did new AFID entries appear in `amd-smi ras --cper`? + +Determine result: +- **PASS**: amd-smi per-block counter incremented AND DME per-block metric matches +- **PARTIAL**: amd-smi shows increment but DME does NOT match (potential DME bug) +- **FAIL**: amd-smi shows NO increment despite injection success +- **RESET**: GPU reset detected (counters cleared, device "resuming") +- **SKIP**: Block not testable (amd-smi reports N/A) or injection command failed + +### 3g. Wait for GPU Recovery + +Poll until GPU is no longer in resuming state (max 120s, 10s intervals). Then proceed to the next block. + +## Step 4 — Generate Report + +Save a markdown report to a timestamped file: `ras-inject-report--.md` + +The report must contain all of: +1. **System Info table** — hostname, kernel, OS, driver version, ROCm version, amd-smi version, amdgpuras version, GPU series/model, DME version, date +2. **GPU Inventory table** — GPU ID, BDF, UUID, serial number +3. **Results Summary table** — GPU, block, type, injection status, amd-smi per-block before/after, DME per-block before/after, health, AFID, result +4. **Cross-Validation Detail** — per block, showing amd-smi per-block, amd-smi total, DME per-block, DME total, health, with match indicators +5. **Totals** — PASS/PARTIAL/FAIL/RESET/SKIP counts +6. **AFID Summary** — per GPU, AFID count, severity breakdown +7. **Detailed AFID Log** — timestamp, GPU, severity, filename, AFID number +8. **Notes** — block-specific observations, known limitations + +## Step 5 — Publish to Confluence + +Results are published to Confluence under a release-based page hierarchy: + +``` +GPU Operator (page ID: 985276253, space ID: 566493184, space key: EN) +├── RAS Error Injection Test Results — v1.5.1 (parent, one per release) +│ ├── Run: exporter-0.0.1-342 / MI350X / SMCI / 2026-06-24 +│ └── Run: exporter-0.0.1-350 / MI350X / SMCI / 2026-06-27 +├── RAS Error Injection Test Results — v1.5.2 +│ └── Run: ... +└── ... +``` + +**Procedure:** + +1. **Find or create the release parent page.** Search for a child page of `985276253` titled `"RAS Error Injection Test Results — "`. If it exists, use it. If not, create it with the standard overview template (overview table, per-block validation mapping, block coverage, result legend). + +2. **Create the run child page** under the release parent with title: `"Run: / / / "`. Body contains the full report with system info, GPU inventory, results summary, cross-validation, AFID log, and notes. + +3. **Update the release parent overview table** — add/update a row for this run showing DME version, platform, GPU series, driver, date, per-block results (PASS/FAIL/SKIP lozenges), and overall score. + +If the Atlassian MCP tools are not connected, print the report path and instruct the user to authenticate: +``` +=== Report Generated (local only) === +File: + +Confluence upload skipped — Atlassian MCP not connected. +Run /mcp to authenticate, then re-run or manually upload. +``` + +# Important Notes + +- **amd-smi is the source of truth** for ECC counters and AFID data. Both per-block (`amd-smi metric --ecc-block`) and aggregate (`amd-smi metric --ecc`) should be captured. DME metrics are the system under test — compare per-block (`gpu_ecc_uncorrect_`) and total (`gpu_ecc_uncorrect_total`) against amd-smi. +- **One injection at a time.** Never stack multiple injections without waiting for verification and GPU recovery. Rapid injections cause GPU resets that clear counters. +- **UMC block requires `-a 0x800000000`** address flag. Other blocks do not. +- **PCIe BIF requires `-s 1 -m 2`** (sub-block 1, method ecrc_tx). Sub-block 0 does not exist. Without `-m`, amdgpuras returns "Invalid injection parameters". +- **PCIe BIF has no HW counters** — amd-smi reports N/A. Mark as SKIP, not FAIL. +- **XGMI/WAFL injection succeeds but counters may not increment** on current firmware. Mark as FAIL if no change observed. +- **GPU recovery can take 30-120 seconds.** Poll `amdgpuras -l` for "resuming" state before proceeding. +- **Exporter polling interval is ~30 seconds.** Wait at least 35 seconds after injection before checking DME metrics. +- **All commands run on the remote host via SSH**, not locally. The skill only generates the report locally. +- **Do not ask for user confirmation** on read-only queries or individual injections. Only ask if something unexpected happens (e.g., all GPUs are in resuming state, or SSH drops). +- **Reboot when GPUs are stuck.** If GPUs are in "resuming" state and do not recover within 120 seconds, the host needs a full power cycle. Use the BMC Redfish `FullPowerCycle` endpoint (same as `/bmc-einj-enable`). After reboot: wait for SSH, reload the amdgpu driver (`sudo modprobe amdgpu`), restart the DME container (`sudo docker start device-metrics-exporter`), then resume testing. A `GracefulRestart` is NOT sufficient — only `FullPowerCycle` clears the GPU firmware state. +- **BMC credentials are separate from host credentials.** The skill takes host SSH creds. If a reboot is needed, ask the user for the BMC IP and credentials, or check if `/bmc-einj-enable` was run earlier in the session and reuse those. +- **Cleanup after testing.** Stop and remove the DME container, and reboot the host to clear all GPU error state before releasing the node. diff --git a/docs-internal/knowledge/plans/2026-06-24-bmc-einj-enable-skill.md b/docs-internal/knowledge/plans/2026-06-24-bmc-einj-enable-skill.md new file mode 100644 index 00000000..99c5f717 --- /dev/null +++ b/docs-internal/knowledge/plans/2026-06-24-bmc-einj-enable-skill.md @@ -0,0 +1,72 @@ +# BMC EINJ Enable Skill + +- **Date:** 2026-06-24 +- **Author:** Claude / Srivatsa +- **Related PR(s):** TBD +- **Related issue(s) / JIRA:** N/A + +## Context + +Before `amdgpuras` can inject RAS (Reliability, Availability, Serviceability) +errors on AMD GPU hardware, the EINJ (Error Injection) capability must be +enabled on the BMC via the Redfish API. This is a manual multi-step process +involving OAM slot discovery, Redfish POST calls, and a host power cycle. + +Currently only SMCI (Supermicro) BMCs are known to expose the required OEM +endpoint (`Chassis.ErrInjection` under `Oem.Ami.AMD` or directly under `Oem` depending on firmware version). + +This plan adds a Claude Code skill to automate the entire flow from a single +invocation. + +## Approach + +Create a new skill at `.claude/skills/bmc-einj-enable/SKILL.md` that: + +1. **Discovers EINJ-capable OAM slots** — iterates OAM_0..OAM_7 via + `GET /redfish/v1/Chassis/OAM_X`, checks for `EINJState` at both + `Oem.EINJState` (newer firmware) and `Oem.Ami.AMD.EINJState` (older). +2. **Enables EINJ** — POSTs `{"ErrInjection": "Enable"}` to each slot + where state is `"Disable"`. +3. **Verifies state transition** — re-queries to confirm `"Pending to Enable"`. +4. **Power cycles the host via Redfish** — discovers the reset endpoint at + `/redfish/v1/Systems/1` (or `/Systems/Self`), asks user for confirmation, + then issues `FullPowerCycle` (required — `GracefulRestart` does NOT activate EINJ). +5. **Post-reboot verification** — polls `PowerState` until `"On"`, then + re-queries OAM slots to confirm `EINJState` is `"Enabled"` or `"Enable"` (varies by firmware). +6. **Reports next steps** — instructs user to SSH to the GPU host and run + `amdgpuras` (which can only execute locally on the host). + +All Redfish calls use `curl -k` (self-signed BMC certs). + +### Alternatives considered + +- **Script-based approach** (standalone Python/shell script under + `tests/pytests/scripts/`) — rejected because the workflow is interactive + (needs user confirmation before power cycle) and benefits from Claude's + ability to adapt to unexpected Redfish responses. +- **IPMI-only** (`ipmitool chassis power cycle`) — doesn't cover the EINJ + enable step which requires the Redfish OEM endpoint. + +## Scope + +- **In scope:** OAM EINJ discovery, enable, power cycle via Redfish, + post-reboot verification. Skill README entry. +- **Out of scope:** `amdgpuras` execution (host-only), non-SMCI BMC support, + automated SSH to GPU host, NPD/ANR integration testing. + +## Validation + +- Manual: invoke `/bmc-einj-enable ` against a SMCI BMC + with AMD GPUs. Confirm OAM discovery, EINJ enable, power cycle, and + post-reboot EINJState = Enable. +- Negative: invoke against a non-SMCI BMC and confirm graceful error + reporting (no EINJState found). + +## Risks and rollback + +- **Host downtime** — the skill performs a power cycle. Mitigated by + requiring user confirmation before the reset. +- **BMC compatibility** — untested BMC vendors will fail at OAM discovery; + the skill reports this clearly. +- **Rollback** — delete `.claude/skills/bmc-einj-enable/` and revert the + README entry. No production code is affected. diff --git a/docs-internal/knowledge/plans/2026-06-24-ras-inject-test-skill.md b/docs-internal/knowledge/plans/2026-06-24-ras-inject-test-skill.md new file mode 100644 index 00000000..7ce8190b --- /dev/null +++ b/docs-internal/knowledge/plans/2026-06-24-ras-inject-test-skill.md @@ -0,0 +1,78 @@ +# RAS Error Injection Test Skill + +- **Date:** 2026-06-24 +- **Author:** Claude / Srivatsa +- **Related PR(s):** TBD +- **Related issue(s) / JIRA:** N/A + +## Context + +RAS error injection testing validates that the device-metrics-exporter (DME) +correctly surfaces hardware ECC errors detected by `amd-smi`. The test uses +`amdgpuras` to inject errors into specific GPU blocks (UMC, SDMA, GFX, MMHUB, +PCIe, XGMI) and verifies: + +1. `amd-smi` (ground truth) shows the ECC counter increment +2. DME Prometheus metrics match `amd-smi` +3. AFID (AMD Fault ID) data is generated and correlates with metrics + +This was validated manually on SMCI MI350X hardware: +- GFX UE injection: `gpu_ecc_uncorrect_gfx` incremented 0→1, `gpu_health` flipped 1→0 +- UMC and SDMA injections succeeded but stacking them rapidly caused a GPU reset +- Key learning: inject one block at a time, wait for recovery between injections + +## Approach + +Create a Claude Code skill at `.claude/skills/ras-inject-test/SKILL.md` that: + +1. Connects to GPU host via SSH, validates prerequisites +2. Gathers system info (driver version, GPU count, DME version) +3. Discovers injectable blocks via `amdgpuras -l` +4. For each GPU × block: captures baseline from amd-smi + DME, injects, waits, + captures post-injection, cross-validates, records PASS/PARTIAL/FAIL/RESET/SKIP +5. Collects AFID data via `amd-smi ras --cper` +6. Generates a markdown test report with per-GPU per-block results + +**amd-smi is the source of truth.** DME is the system under test. If amd-smi +shows an increment but DME doesn't, that's flagged as PARTIAL (potential DME bug). + +### Separate concerns + +- **Driver installation** → existing `ci-internal/ansible/install-amdgpu-driver.yml` +- **DME deployment** → future `/deploy-dme` skill (Docker, Debian, K8s modes) +- **EINJ enablement** → existing `/bmc-einj-enable` skill +- **Confluence upload** → integrated in skill (Step 5, publishes when Atlassian MCP is connected, local-only otherwise) + +### Alternatives considered + +- Reusing `amdgpuras_util.py` directly as a pytest — rejected because the test + needs to run on bare-metal hosts that may not have the test framework installed. + The skill uses SSH from the dev machine instead. +- Running all blocks in parallel — rejected because stacking injections causes + GPU resets that clear counters before verification. + +## Scope + +- **In scope:** Error injection, amd-smi/DME cross-validation, AFID collection, + report generation, all GPUs in system. +- **Out of scope:** Driver/DME deployment, EINJ enablement, + CE (correctable error) injection testing (future enhancement). +- **Confluence publishing:** In scope — optional, publishes when Atlassian MCP + is connected, falls back to local-only report otherwise. + +## Validation + +1. Invoke `/ras-inject-test ` against a + SMCI350 host where EINJ is enabled and DME is running +2. Confirm system info is gathered correctly +3. Confirm at least GFX block injection produces PASS result +4. Confirm report file is generated with all GPUs and blocks +5. Confirm AFID data appears in the report + +## Risks and rollback + +- **GPU resets** — mitigated by sequential injection with recovery waits +- **Host unreachable after injection** — GPU resets can temporarily drop SSH; + the skill retries SSH after recovery wait +- **Rollback** — delete `.claude/skills/ras-inject-test/` and revert README. + No production code affected.