From 36cdd55e16bc6cf6679a7c1de5ac719b08fb6f31 Mon Sep 17 00:00:00 2001 From: Doug Holt Date: Mon, 13 Jul 2026 14:50:48 -0600 Subject: [PATCH] Add AGENTS.md operating guide and reusable skills Add an agent-first operating layer so AI agents (and new humans) can drive DeepOps end to end: - AGENTS.md: repository map, first-time setup, Slurm and Kubernetes golden paths anchored on the validation tools, operating rules (validate don't assume, review inventories, reboot awareness, check mode, idempotent reruns), and gotchas that look like failures but are not (login GPU hiding, fact caching, open vs proprietary kernel modules, uninitialized Kubespray submodule) - skills/: four self-contained procedures with preconditions, exact commands, expected outputs, and failure branches drawn from real validation runs: deploy-slurm-cluster, deploy-k8s-gpu-cluster, validate-gpu-cluster, diagnose-driver-install - README pointer to AGENTS.md, the validation contract, and skills/ --- AGENTS.md | 107 ++++++++++++++++++++++++ README.md | 2 + skills/README.md | 24 ++++++ skills/deploy-k8s-gpu-cluster/SKILL.md | 72 ++++++++++++++++ skills/deploy-slurm-cluster/SKILL.md | 72 ++++++++++++++++ skills/diagnose-driver-install/SKILL.md | 82 ++++++++++++++++++ skills/validate-gpu-cluster/SKILL.md | 51 +++++++++++ 7 files changed, 410 insertions(+) create mode 100644 AGENTS.md create mode 100644 skills/README.md create mode 100644 skills/deploy-k8s-gpu-cluster/SKILL.md create mode 100644 skills/deploy-slurm-cluster/SKILL.md create mode 100644 skills/diagnose-driver-install/SKILL.md create mode 100644 skills/validate-gpu-cluster/SKILL.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..b10263c98 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,107 @@ +# DeepOps agent operating guide + +This file is the entry point for AI agents (and new humans) operating this +repository. DeepOps deploys and manages GPU clusters — Slurm or Kubernetes on +NVIDIA GPU servers — using Ansible. Everything here is driven by playbooks +against an inventory you control; there is no server component. + +Load only what the task needs: this file for orientation and the golden +paths, `docs/` for depth, `skills/` for step-by-step procedures with failure +handling. + +## Repository map + +| Path | Purpose | +|------|---------| +| `playbooks/` | Entry points. `slurm-cluster.yml` and `k8s-cluster.yml` are the two top-level cluster deploys; subdirectories hold component playbooks. | +| `roles/` | Ansible roles (NVIDIA drivers, container toolkit, DGX software, monitoring, Slurm, storage). | +| `config.example/` | Template configuration. Copy to `config/` and edit; never edit `config.example/` for a deployment. | +| `config/` | Your site configuration and inventory (git-ignored; created by you). | +| `scripts/validation/` | Machine-readable preflight and post-deploy validation. Start here to know if anything worked. | +| `scripts/` | Setup and helper scripts (`setup.sh` installs Ansible and dependencies). | +| `submodules/kubespray` | Kubernetes deployment engine. Must be initialized before Kubernetes work. | +| `docs/` | Topic documentation: `deepops/`, `slurm-cluster/`, `k8s-cluster/`, `container/`, `airgap/`. | +| `skills/` | Reusable agent procedures with preconditions, commands, expected output, and failure branches. | + +## First-time setup (once per provisioning machine) + +```bash +git submodule update --init --recursive +./scripts/setup.sh # installs Ansible + Galaxy dependencies +cp -r config.example config # then edit config/inventory +python3 scripts/validation/deepops_doctor.py --json # verify before deploying +``` + +The doctor must report `"ok": true` (or you must understand every failure) +before you run any cluster playbook. With `--remote` it also proves SSH +reachability to every inventory host. + +## Golden path: Slurm GPU cluster + +```bash +# inventory groups: slurm-master, slurm-node (see config.example/inventory) +ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml +python3 scripts/validation/validate_slurm.py --json # run on a cluster node +``` + +The validator must report `"ok": true` with `gpu_job_ok: true`. See +`skills/deploy-slurm-cluster/` for the full procedure and failure branches. + +## Golden path: Kubernetes GPU cluster + +```bash +# inventory groups: kube_control_plane, etcd, kube_node (see config.example/inventory) +ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml +python3 scripts/validation/validate_k8s.py --json --cuda-smoke +``` + +The validator must report `"ok": true` with `cuda_smoke_ok: true`. See +`skills/deploy-k8s-gpu-cluster/` for the full procedure and failure branches. + +## Rules for operating this repository + +1. **Validate, don't assume.** Run the doctor before deploying and the + matching validator after. A playbook finishing with `failed=0` is not the + success signal; the validator's `"ok": true` is. +2. **Never run a cluster playbook against an unreviewed inventory.** These + playbooks install drivers, change container runtimes, and can reboot + machines. Confirm the inventory lists exactly the intended hosts + (`ansible-inventory --list`, or the doctor's host count) first. +3. **Driver installs can reboot nodes.** Schedule accordingly; never point a + first-time deploy at hosts with active users or workloads. +4. **Preview when unsure.** `ansible-playbook --check --diff -l ` shows + most pending changes without applying them (some tasks don't support + check mode). Use `--limit` to scope any run. +5. **Playbooks are idempotent; reruns are the normal recovery path.** After a + transient failure (package mirror timeout, network blip), rerun the same + playbook. A converged rerun reports `changed=0`. +6. **Configuration lives in `config/`, not in role defaults.** Override + variables in `config/group_vars/`; do not edit roles for site-specific + values. + +## Gotchas that look like failures but are not + +- **`nvidia-smi` over SSH reports "No devices were found" on Slurm nodes.** + DeepOps hides GPUs from ordinary SSH sessions on cluster nodes; GPUs are + visible inside Slurm jobs. Test with + `srun --gpus=1 nvidia-smi`, or `validate_slurm.py`, never with bare SSH + `nvidia-smi`. +- **Ansible fact caching can serve stale facts** when an inventory hostname + is reused for a different machine or after an OS reinstall. Rerun with + `--flush-cache`. +- **Open vs proprietary NVIDIA kernel modules matter per GPU generation.** + Turing and newer support the open kernel modules; older GPUs (e.g. Pascal) + need `nvidia_driver_ubuntu_use_open_kernel_modules: false`. A wrong choice + produces `nvidia-smi: No devices were found` after a clean-looking install. + See `skills/diagnose-driver-install/`. +- **Kubernetes playbooks fail on syntax/imports if `submodules/kubespray` is + not initialized** — the error mentions missing `kubespray_defaults` roles, + not submodules. Run `git submodule update --init --recursive`. + +## Contributing changes + +Run before pushing: `git diff --check`, YAML parse on changed files, +`./scripts/deepops/ansible-lint-roles.sh`, and a focused +`ansible-playbook --syntax-check` for changed playbooks. Public CI runs lint, +setup, and molecule checks on every PR. Deployment-affecting changes need +GPU-backed validation evidence in the PR body. diff --git a/README.md b/README.md index fd3f8ea84..c4e67fda3 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@ Latest release: [DeepOps 26.05 Release](https://github.com/NVIDIA/deepops/releas It is recommended to use the latest release branch for stable code (linked above). All development takes place on the master branch, which is generally [functional](docs/deepops/testing.md) but may change significantly between releases. +Operating this repository with an AI agent (or onboarding as a new human)? Start with [AGENTS.md](AGENTS.md) for the golden paths, operating rules, and [machine-readable validation tools](docs/deepops/validation.md), and [skills/](skills/README.md) for step-by-step procedures. + ## Deployment Requirements ### Provisioning System diff --git a/skills/README.md b/skills/README.md new file mode 100644 index 000000000..8822b2cac --- /dev/null +++ b/skills/README.md @@ -0,0 +1,24 @@ +# DeepOps skills + +Reusable, self-contained procedures for operating DeepOps, written for AI +agents and equally usable by humans. Each skill is a directory containing a +`SKILL.md` with YAML frontmatter (`name`, `description`) followed by +preconditions, exact commands, expected outputs, and failure branches. + +The format follows the emerging cross-tool agent-skills convention: agent +frameworks that support skill discovery can load these directly, and any +agent (or person) can simply read the relevant `SKILL.md` before acting. + +| Skill | Use when | +|-------|----------| +| [deploy-slurm-cluster](deploy-slurm-cluster/SKILL.md) | Deploying or rebuilding a Slurm GPU cluster. | +| [deploy-k8s-gpu-cluster](deploy-k8s-gpu-cluster/SKILL.md) | Deploying or rebuilding a Kubernetes GPU cluster. | +| [validate-gpu-cluster](validate-gpu-cluster/SKILL.md) | Health checks and post-deploy verification. | +| [diagnose-driver-install](diagnose-driver-install/SKILL.md) | NVIDIA driver failures, `nvidia-smi` errors, GPU pods crash-looping. | + +Start with [AGENTS.md](../AGENTS.md) at the repository root for orientation, +golden paths, and operating rules. + +Contributions should keep skills honest: every command must work as written +from a clean checkout, and failure branches should come from real observed +failures, not speculation. diff --git a/skills/deploy-k8s-gpu-cluster/SKILL.md b/skills/deploy-k8s-gpu-cluster/SKILL.md new file mode 100644 index 000000000..8af7e9e39 --- /dev/null +++ b/skills/deploy-k8s-gpu-cluster/SKILL.md @@ -0,0 +1,72 @@ +--- +name: deploy-k8s-gpu-cluster +description: Deploy a Kubernetes GPU cluster with DeepOps (Kubespray + GPU Operator) and prove it schedules GPU pods. Use when asked to deploy or rebuild Kubernetes on GPU servers with this repository. +--- + +# Deploy a Kubernetes GPU cluster + +## Preconditions + +- Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver + installs may reboot them; no active users or workloads). +- SSH access from the provisioning machine to every host as a sudo-capable + user. +- `submodules/kubespray` initialized — Kubernetes playbooks fail on missing + `kubespray_defaults` role imports without it. +- Run everything from the repository root. + +## Procedure + +1. Prepare the environment and verify it: + + ```bash + git submodule update --init --recursive + ./scripts/setup.sh + cp -r config.example config + ``` + +2. Edit `config/inventory`: control plane nodes under + `[kube_control_plane]` and `[etcd]`, workers under `[kube_node]` (a + single machine can hold all three roles). + +3. Preflight — must pass before deploying: + + ```bash + python3 scripts/validation/deepops_doctor.py --remote --json + ``` + +4. Deploy: + + ```bash + ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml + ``` + + This runs Kubespray and installs the NVIDIA GPU Operator. Expect + roughly 45–90 minutes on a first run. + +5. Validate — the success signal is this, not the play recap: + + ```bash + python3 scripts/validation/validate_k8s.py --json --cuda-smoke + ``` + + Require `"ok": true` with `nodes_ready == nodes_total`, + `gpus_allocatable > 0`, and `cuda_smoke_ok: true`. + +## Failure branches + +- **Playbook fails on a transient error**: rerun the same playbook; + Kubespray is rerun-safe. A converged rerun reports `changed=0`. +- **Syntax/import error mentioning `kubespray_defaults`**: the submodule is + not initialized; run `git submodule update --init --recursive`. +- **`gpus_allocatable: 0`**: the GPU Operator stack is not ready. Check + `kubectl get pods -A | grep -i nvidia` — the driver DaemonSet can take + 10+ minutes on first deploy; if pods are crash-looping, follow + `skills/diagnose-driver-install/`. +- **CUDA smoke pod stuck `Pending`**: `kubectl -n deepops-validate + describe pod deepops-validate-cuda` — usually no allocatable GPU + (see above) or an image pull problem on airgapped networks (use + `--cuda-image` to point at a mirrored image). +- **Single-node clusters**: control plane taints are handled by the + playbook for the single-node case; if pods stay Pending on a multi-role + node, check taints with `kubectl describe node | grep -i taint`. diff --git a/skills/deploy-slurm-cluster/SKILL.md b/skills/deploy-slurm-cluster/SKILL.md new file mode 100644 index 000000000..0ac432a42 --- /dev/null +++ b/skills/deploy-slurm-cluster/SKILL.md @@ -0,0 +1,72 @@ +--- +name: deploy-slurm-cluster +description: Deploy a Slurm GPU cluster with DeepOps and prove it works. Use when asked to deploy, install, or rebuild Slurm on one or more GPU servers with this repository. +--- + +# Deploy a Slurm GPU cluster + +## Preconditions + +- Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver + installs may reboot them; no active users or workloads). +- SSH access from the provisioning machine to every host as a sudo-capable + user. +- Run everything from the repository root. + +## Procedure + +1. Prepare the environment and verify it: + + ```bash + git submodule update --init --recursive + ./scripts/setup.sh + cp -r config.example config + ``` + +2. Edit `config/inventory`: put the controller under `[slurm-master]` and + compute nodes under `[slurm-node]` (a single machine can be both). Set + the connection user in `[all:vars]` if not root. + +3. Preflight — must pass before deploying: + + ```bash + python3 scripts/validation/deepops_doctor.py --remote --json + ``` + + Fix anything in `failures` (each check's `detail` says how) and rerun. + +4. Deploy: + + ```bash + ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml + ``` + + This installs NVIDIA drivers, builds and configures Slurm, and sets up + munge, NFS, and node health checks. Expect roughly 30–60 minutes on a + first run. + +5. Validate on a cluster node — the success signal is this, not the play + recap: + + ```bash + python3 scripts/validation/validate_slurm.py --json + ``` + + Require `"ok": true` with `gpu_job_ok: true` and + `nodes_unavailable: 0`. + +## Failure branches + +- **Playbook fails on a transient error** (mirror timeout, apt lock, + network blip): rerun the same playbook; it is idempotent. A converged + rerun ends with `changed=0`. +- **`nvidia-smi` works in the validator's srun job but "fails" over SSH**: + that is the login GPU-hide behavior, not an error (see AGENTS.md + gotchas). +- **`gpu_job_ok: false` with driver errors**: follow + `skills/diagnose-driver-install/`. +- **Node shows `down` or `drained` in `node_states`**: check + `scontrol show node ` for the reason; after fixing, resume with + `scontrol update nodename= state=resume`. +- **Wrong or stale host facts after reprovisioning a node**: rerun with + `--flush-cache`. diff --git a/skills/diagnose-driver-install/SKILL.md b/skills/diagnose-driver-install/SKILL.md new file mode 100644 index 000000000..34b2fb2ff --- /dev/null +++ b/skills/diagnose-driver-install/SKILL.md @@ -0,0 +1,82 @@ +--- +name: diagnose-driver-install +description: Diagnose NVIDIA driver installation failures on DeepOps-managed nodes — nvidia-smi errors, "No devices were found", DKMS build failures, or GPU pods crash-looping. Use before reinstalling anything. +--- + +# Diagnose an NVIDIA driver install + +Work through these in order; most "driver failures" are one of the first +three and need no reinstall. + +## 1. Are you being fooled by GPU hiding? + +On DeepOps **Slurm** nodes, GPUs are hidden from ordinary SSH sessions by +design. Bare `nvidia-smi` over SSH reporting `No devices were found` on an +otherwise healthy node is expected. + +```bash +srun --gpus=1 nvidia-smi # the authoritative test on Slurm nodes +``` + +If the srun job sees the GPU, the driver is fine. Stop here. + +## 2. Is the hardware visible at all? + +```bash +lspci | grep -i nvidia +``` + +No output → not a driver problem. The GPU is absent, unseated, or bound by +VFIO passthrough or platform firmware; escalate to hardware support before +touching software. + +## 3. Open vs proprietary kernel modules + +`nvidia-smi: No devices were found` immediately after a clean install is the +classic symptom of the wrong module flavor for the GPU generation: + +- Turing and newer (T4, A100, H100, RTX PRO Blackwell): open kernel modules + supported — DeepOps default `nvidia_driver_ubuntu_use_open_kernel_modules: true` + is correct. +- Pascal and older (P100, GTX 10xx): open modules are **not** supported — + set `nvidia_driver_ubuntu_use_open_kernel_modules: false` in + `config/group_vars/all.yml` and rerun the driver play. + +Check what is loaded: `modinfo nvidia | grep -i license` (open modules say +MIT/GPL, proprietary says NVIDIA). + +## 4. Did the kernel module actually build and load? + +```bash +dkms status # driver module state per kernel +dmesg | grep -iE 'nvidia|nvrm' | tail -20 +lsmod | grep nvidia +``` + +- DKMS shows an error for the running kernel → usually missing headers + (`linux-headers-$(uname -r)`) or a kernel updated after the driver + install. Install headers or reboot into the matching kernel, then rerun + the driver play. +- Module loaded but `nvidia-smi` fails → check `dmesg` for RmInitAdapter or + fallen-off-the-bus errors; these are hardware/firmware territory. + +## 5. Rerun the play, don't hand-fix + +After correcting configuration, converge with the playbook rather than +manual package surgery, then validate: + +```bash +ansible-playbook -l playbooks/nvidia-software/nvidia-driver.yml +python3 scripts/validation/validate_slurm.py --json # or validate_k8s.py +``` + +Reruns are idempotent; a healthy converged rerun reports `changed=0`. + +## Kubernetes-specific notes + +GPU Operator crash-looping driver pods: `kubectl logs -n ` usually names the same root causes — missing +headers, wrong module flavor, or a node reboot needed. Fix via the operator +values or node state, not by installing drivers by hand on the host (the +operator owns the driver on Kubernetes nodes unless DeepOps was configured +for host drivers). diff --git a/skills/validate-gpu-cluster/SKILL.md b/skills/validate-gpu-cluster/SKILL.md new file mode 100644 index 000000000..9b43f4e47 --- /dev/null +++ b/skills/validate-gpu-cluster/SKILL.md @@ -0,0 +1,51 @@ +--- +name: validate-gpu-cluster +description: Check whether a DeepOps-deployed Slurm or Kubernetes GPU cluster is healthy and report a machine-readable verdict. Use for health checks, post-deploy verification, "is the cluster working?" questions, and after any node or driver change. +--- + +# Validate a GPU cluster + +## Which validator + +- Slurm cluster → `scripts/validation/validate_slurm.py` (run on a cluster + node) +- Kubernetes cluster → `scripts/validation/validate_k8s.py` (run wherever + `kubectl` reaches the cluster) +- Not deployed yet / checking the provisioning environment → + `scripts/validation/deepops_doctor.py` (run from the repository root) + +All tools support `--json` and exit `0` only when every check passes. Full +contract: `docs/deepops/validation.md`. + +## Procedure + +1. Run the matching validator with `--json`. + + ```bash + python3 scripts/validation/validate_slurm.py --json + python3 scripts/validation/validate_k8s.py --json --cuda-smoke + ``` + + Use `--skip-gpu-job` (Slurm) or omit `--cuda-smoke` (Kubernetes) when a + strictly read-only check is required — for example on a busy production + cluster where a test job would queue behind real work. + +2. Read the verdict from the JSON, not from ad-hoc commands: + - `ok: true` — report healthy, include the key counts + (`nodes_total`, `gpus_configured`/`gpus_allocatable`). + - `ok: false` — report each entry in `failures` verbatim; they name the + failing subsystem and the next diagnostic step. + +3. When a GPU check fails, do not conclude "driver broken" from a bare + `nvidia-smi` over SSH — on Slurm nodes GPUs are hidden outside jobs. + Follow `skills/diagnose-driver-install/` instead. + +## Interpreting common results + +| Signal | Meaning | +|--------|---------| +| Slurm `nodes_unavailable > 0` | Node down/drained — `scontrol show node ` for the reason. | +| Slurm `gpus_configured: 0` | GRES not configured — check `config/group_vars/slurm-cluster.yml` GPU settings and rerun the Slurm playbook. | +| K8s `gpus_allocatable: 0` | GPU Operator stack not ready — `kubectl get pods -A \| grep -i nvidia`; first-deploy driver builds can take 10+ minutes. | +| K8s smoke pod `Pending` | No schedulable GPU or image pull failure — `kubectl -n deepops-validate describe pod`. | +| Direct `nvidia-smi` over SSH says "No devices were found" on a Slurm node | Expected GPU-hide behavior, not a failure. |