diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..b10263c98 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,107 @@ +# DeepOps agent operating guide + +This file is the entry point for AI agents (and new humans) operating this +repository. DeepOps deploys and manages GPU clusters — Slurm or Kubernetes on +NVIDIA GPU servers — using Ansible. Everything here is driven by playbooks +against an inventory you control; there is no server component. + +Load only what the task needs: this file for orientation and the golden +paths, `docs/` for depth, `skills/` for step-by-step procedures with failure +handling. + +## Repository map + +| Path | Purpose | +|------|---------| +| `playbooks/` | Entry points. `slurm-cluster.yml` and `k8s-cluster.yml` are the two top-level cluster deploys; subdirectories hold component playbooks. | +| `roles/` | Ansible roles (NVIDIA drivers, container toolkit, DGX software, monitoring, Slurm, storage). | +| `config.example/` | Template configuration. Copy to `config/` and edit; never edit `config.example/` for a deployment. | +| `config/` | Your site configuration and inventory (git-ignored; created by you). | +| `scripts/validation/` | Machine-readable preflight and post-deploy validation. Start here to know if anything worked. | +| `scripts/` | Setup and helper scripts (`setup.sh` installs Ansible and dependencies). | +| `submodules/kubespray` | Kubernetes deployment engine. Must be initialized before Kubernetes work. | +| `docs/` | Topic documentation: `deepops/`, `slurm-cluster/`, `k8s-cluster/`, `container/`, `airgap/`. | +| `skills/` | Reusable agent procedures with preconditions, commands, expected output, and failure branches. | + +## First-time setup (once per provisioning machine) + +```bash +git submodule update --init --recursive +./scripts/setup.sh # installs Ansible + Galaxy dependencies +cp -r config.example config # then edit config/inventory +python3 scripts/validation/deepops_doctor.py --json # verify before deploying +``` + +The doctor must report `"ok": true` (or you must understand every failure) +before you run any cluster playbook. With `--remote` it also proves SSH +reachability to every inventory host. + +## Golden path: Slurm GPU cluster + +```bash +# inventory groups: slurm-master, slurm-node (see config.example/inventory) +ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml +python3 scripts/validation/validate_slurm.py --json # run on a cluster node +``` + +The validator must report `"ok": true` with `gpu_job_ok: true`. See +`skills/deploy-slurm-cluster/` for the full procedure and failure branches. + +## Golden path: Kubernetes GPU cluster + +```bash +# inventory groups: kube_control_plane, etcd, kube_node (see config.example/inventory) +ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml +python3 scripts/validation/validate_k8s.py --json --cuda-smoke +``` + +The validator must report `"ok": true` with `cuda_smoke_ok: true`. See +`skills/deploy-k8s-gpu-cluster/` for the full procedure and failure branches. + +## Rules for operating this repository + +1. **Validate, don't assume.** Run the doctor before deploying and the + matching validator after. A playbook finishing with `failed=0` is not the + success signal; the validator's `"ok": true` is. +2. **Never run a cluster playbook against an unreviewed inventory.** These + playbooks install drivers, change container runtimes, and can reboot + machines. Confirm the inventory lists exactly the intended hosts + (`ansible-inventory --list`, or the doctor's host count) first. +3. **Driver installs can reboot nodes.** Schedule accordingly; never point a + first-time deploy at hosts with active users or workloads. +4. **Preview when unsure.** `ansible-playbook --check --diff -l ` shows + most pending changes without applying them (some tasks don't support + check mode). Use `--limit` to scope any run. +5. **Playbooks are idempotent; reruns are the normal recovery path.** After a + transient failure (package mirror timeout, network blip), rerun the same + playbook. A converged rerun reports `changed=0`. +6. **Configuration lives in `config/`, not in role defaults.** Override + variables in `config/group_vars/`; do not edit roles for site-specific + values. + +## Gotchas that look like failures but are not + +- **`nvidia-smi` over SSH reports "No devices were found" on Slurm nodes.** + DeepOps hides GPUs from ordinary SSH sessions on cluster nodes; GPUs are + visible inside Slurm jobs. Test with + `srun --gpus=1 nvidia-smi`, or `validate_slurm.py`, never with bare SSH + `nvidia-smi`. +- **Ansible fact caching can serve stale facts** when an inventory hostname + is reused for a different machine or after an OS reinstall. Rerun with + `--flush-cache`. +- **Open vs proprietary NVIDIA kernel modules matter per GPU generation.** + Turing and newer support the open kernel modules; older GPUs (e.g. Pascal) + need `nvidia_driver_ubuntu_use_open_kernel_modules: false`. A wrong choice + produces `nvidia-smi: No devices were found` after a clean-looking install. + See `skills/diagnose-driver-install/`. +- **Kubernetes playbooks fail on syntax/imports if `submodules/kubespray` is + not initialized** — the error mentions missing `kubespray_defaults` roles, + not submodules. Run `git submodule update --init --recursive`. + +## Contributing changes + +Run before pushing: `git diff --check`, YAML parse on changed files, +`./scripts/deepops/ansible-lint-roles.sh`, and a focused +`ansible-playbook --syntax-check` for changed playbooks. Public CI runs lint, +setup, and molecule checks on every PR. Deployment-affecting changes need +GPU-backed validation evidence in the PR body. diff --git a/README.md b/README.md index fd3f8ea84..c4e67fda3 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,8 @@ Latest release: [DeepOps 26.05 Release](https://github.com/NVIDIA/deepops/releas It is recommended to use the latest release branch for stable code (linked above). All development takes place on the master branch, which is generally [functional](docs/deepops/testing.md) but may change significantly between releases. +Operating this repository with an AI agent (or onboarding as a new human)? Start with [AGENTS.md](AGENTS.md) for the golden paths, operating rules, and [machine-readable validation tools](docs/deepops/validation.md), and [skills/](skills/README.md) for step-by-step procedures. + ## Deployment Requirements ### Provisioning System diff --git a/skills/README.md b/skills/README.md new file mode 100644 index 000000000..8822b2cac --- /dev/null +++ b/skills/README.md @@ -0,0 +1,24 @@ +# DeepOps skills + +Reusable, self-contained procedures for operating DeepOps, written for AI +agents and equally usable by humans. Each skill is a directory containing a +`SKILL.md` with YAML frontmatter (`name`, `description`) followed by +preconditions, exact commands, expected outputs, and failure branches. + +The format follows the emerging cross-tool agent-skills convention: agent +frameworks that support skill discovery can load these directly, and any +agent (or person) can simply read the relevant `SKILL.md` before acting. + +| Skill | Use when | +|-------|----------| +| [deploy-slurm-cluster](deploy-slurm-cluster/SKILL.md) | Deploying or rebuilding a Slurm GPU cluster. | +| [deploy-k8s-gpu-cluster](deploy-k8s-gpu-cluster/SKILL.md) | Deploying or rebuilding a Kubernetes GPU cluster. | +| [validate-gpu-cluster](validate-gpu-cluster/SKILL.md) | Health checks and post-deploy verification. | +| [diagnose-driver-install](diagnose-driver-install/SKILL.md) | NVIDIA driver failures, `nvidia-smi` errors, GPU pods crash-looping. | + +Start with [AGENTS.md](../AGENTS.md) at the repository root for orientation, +golden paths, and operating rules. + +Contributions should keep skills honest: every command must work as written +from a clean checkout, and failure branches should come from real observed +failures, not speculation. diff --git a/skills/deploy-k8s-gpu-cluster/SKILL.md b/skills/deploy-k8s-gpu-cluster/SKILL.md new file mode 100644 index 000000000..8af7e9e39 --- /dev/null +++ b/skills/deploy-k8s-gpu-cluster/SKILL.md @@ -0,0 +1,72 @@ +--- +name: deploy-k8s-gpu-cluster +description: Deploy a Kubernetes GPU cluster with DeepOps (Kubespray + GPU Operator) and prove it schedules GPU pods. Use when asked to deploy or rebuild Kubernetes on GPU servers with this repository. +--- + +# Deploy a Kubernetes GPU cluster + +## Preconditions + +- Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver + installs may reboot them; no active users or workloads). +- SSH access from the provisioning machine to every host as a sudo-capable + user. +- `submodules/kubespray` initialized — Kubernetes playbooks fail on missing + `kubespray_defaults` role imports without it. +- Run everything from the repository root. + +## Procedure + +1. Prepare the environment and verify it: + + ```bash + git submodule update --init --recursive + ./scripts/setup.sh + cp -r config.example config + ``` + +2. Edit `config/inventory`: control plane nodes under + `[kube_control_plane]` and `[etcd]`, workers under `[kube_node]` (a + single machine can hold all three roles). + +3. Preflight — must pass before deploying: + + ```bash + python3 scripts/validation/deepops_doctor.py --remote --json + ``` + +4. Deploy: + + ```bash + ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml + ``` + + This runs Kubespray and installs the NVIDIA GPU Operator. Expect + roughly 45–90 minutes on a first run. + +5. Validate — the success signal is this, not the play recap: + + ```bash + python3 scripts/validation/validate_k8s.py --json --cuda-smoke + ``` + + Require `"ok": true` with `nodes_ready == nodes_total`, + `gpus_allocatable > 0`, and `cuda_smoke_ok: true`. + +## Failure branches + +- **Playbook fails on a transient error**: rerun the same playbook; + Kubespray is rerun-safe. A converged rerun reports `changed=0`. +- **Syntax/import error mentioning `kubespray_defaults`**: the submodule is + not initialized; run `git submodule update --init --recursive`. +- **`gpus_allocatable: 0`**: the GPU Operator stack is not ready. Check + `kubectl get pods -A | grep -i nvidia` — the driver DaemonSet can take + 10+ minutes on first deploy; if pods are crash-looping, follow + `skills/diagnose-driver-install/`. +- **CUDA smoke pod stuck `Pending`**: `kubectl -n deepops-validate + describe pod deepops-validate-cuda` — usually no allocatable GPU + (see above) or an image pull problem on airgapped networks (use + `--cuda-image` to point at a mirrored image). +- **Single-node clusters**: control plane taints are handled by the + playbook for the single-node case; if pods stay Pending on a multi-role + node, check taints with `kubectl describe node | grep -i taint`. diff --git a/skills/deploy-slurm-cluster/SKILL.md b/skills/deploy-slurm-cluster/SKILL.md new file mode 100644 index 000000000..0ac432a42 --- /dev/null +++ b/skills/deploy-slurm-cluster/SKILL.md @@ -0,0 +1,72 @@ +--- +name: deploy-slurm-cluster +description: Deploy a Slurm GPU cluster with DeepOps and prove it works. Use when asked to deploy, install, or rebuild Slurm on one or more GPU servers with this repository. +--- + +# Deploy a Slurm GPU cluster + +## Preconditions + +- Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver + installs may reboot them; no active users or workloads). +- SSH access from the provisioning machine to every host as a sudo-capable + user. +- Run everything from the repository root. + +## Procedure + +1. Prepare the environment and verify it: + + ```bash + git submodule update --init --recursive + ./scripts/setup.sh + cp -r config.example config + ``` + +2. Edit `config/inventory`: put the controller under `[slurm-master]` and + compute nodes under `[slurm-node]` (a single machine can be both). Set + the connection user in `[all:vars]` if not root. + +3. Preflight — must pass before deploying: + + ```bash + python3 scripts/validation/deepops_doctor.py --remote --json + ``` + + Fix anything in `failures` (each check's `detail` says how) and rerun. + +4. Deploy: + + ```bash + ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml + ``` + + This installs NVIDIA drivers, builds and configures Slurm, and sets up + munge, NFS, and node health checks. Expect roughly 30–60 minutes on a + first run. + +5. Validate on a cluster node — the success signal is this, not the play + recap: + + ```bash + python3 scripts/validation/validate_slurm.py --json + ``` + + Require `"ok": true` with `gpu_job_ok: true` and + `nodes_unavailable: 0`. + +## Failure branches + +- **Playbook fails on a transient error** (mirror timeout, apt lock, + network blip): rerun the same playbook; it is idempotent. A converged + rerun ends with `changed=0`. +- **`nvidia-smi` works in the validator's srun job but "fails" over SSH**: + that is the login GPU-hide behavior, not an error (see AGENTS.md + gotchas). +- **`gpu_job_ok: false` with driver errors**: follow + `skills/diagnose-driver-install/`. +- **Node shows `down` or `drained` in `node_states`**: check + `scontrol show node ` for the reason; after fixing, resume with + `scontrol update nodename= state=resume`. +- **Wrong or stale host facts after reprovisioning a node**: rerun with + `--flush-cache`. diff --git a/skills/diagnose-driver-install/SKILL.md b/skills/diagnose-driver-install/SKILL.md new file mode 100644 index 000000000..34b2fb2ff --- /dev/null +++ b/skills/diagnose-driver-install/SKILL.md @@ -0,0 +1,82 @@ +--- +name: diagnose-driver-install +description: Diagnose NVIDIA driver installation failures on DeepOps-managed nodes — nvidia-smi errors, "No devices were found", DKMS build failures, or GPU pods crash-looping. Use before reinstalling anything. +--- + +# Diagnose an NVIDIA driver install + +Work through these in order; most "driver failures" are one of the first +three and need no reinstall. + +## 1. Are you being fooled by GPU hiding? + +On DeepOps **Slurm** nodes, GPUs are hidden from ordinary SSH sessions by +design. Bare `nvidia-smi` over SSH reporting `No devices were found` on an +otherwise healthy node is expected. + +```bash +srun --gpus=1 nvidia-smi # the authoritative test on Slurm nodes +``` + +If the srun job sees the GPU, the driver is fine. Stop here. + +## 2. Is the hardware visible at all? + +```bash +lspci | grep -i nvidia +``` + +No output → not a driver problem. The GPU is absent, unseated, or bound by +VFIO passthrough or platform firmware; escalate to hardware support before +touching software. + +## 3. Open vs proprietary kernel modules + +`nvidia-smi: No devices were found` immediately after a clean install is the +classic symptom of the wrong module flavor for the GPU generation: + +- Turing and newer (T4, A100, H100, RTX PRO Blackwell): open kernel modules + supported — DeepOps default `nvidia_driver_ubuntu_use_open_kernel_modules: true` + is correct. +- Pascal and older (P100, GTX 10xx): open modules are **not** supported — + set `nvidia_driver_ubuntu_use_open_kernel_modules: false` in + `config/group_vars/all.yml` and rerun the driver play. + +Check what is loaded: `modinfo nvidia | grep -i license` (open modules say +MIT/GPL, proprietary says NVIDIA). + +## 4. Did the kernel module actually build and load? + +```bash +dkms status # driver module state per kernel +dmesg | grep -iE 'nvidia|nvrm' | tail -20 +lsmod | grep nvidia +``` + +- DKMS shows an error for the running kernel → usually missing headers + (`linux-headers-$(uname -r)`) or a kernel updated after the driver + install. Install headers or reboot into the matching kernel, then rerun + the driver play. +- Module loaded but `nvidia-smi` fails → check `dmesg` for RmInitAdapter or + fallen-off-the-bus errors; these are hardware/firmware territory. + +## 5. Rerun the play, don't hand-fix + +After correcting configuration, converge with the playbook rather than +manual package surgery, then validate: + +```bash +ansible-playbook -l playbooks/nvidia-software/nvidia-driver.yml +python3 scripts/validation/validate_slurm.py --json # or validate_k8s.py +``` + +Reruns are idempotent; a healthy converged rerun reports `changed=0`. + +## Kubernetes-specific notes + +GPU Operator crash-looping driver pods: `kubectl logs -n ` usually names the same root causes — missing +headers, wrong module flavor, or a node reboot needed. Fix via the operator +values or node state, not by installing drivers by hand on the host (the +operator owns the driver on Kubernetes nodes unless DeepOps was configured +for host drivers). diff --git a/skills/validate-gpu-cluster/SKILL.md b/skills/validate-gpu-cluster/SKILL.md new file mode 100644 index 000000000..9b43f4e47 --- /dev/null +++ b/skills/validate-gpu-cluster/SKILL.md @@ -0,0 +1,51 @@ +--- +name: validate-gpu-cluster +description: Check whether a DeepOps-deployed Slurm or Kubernetes GPU cluster is healthy and report a machine-readable verdict. Use for health checks, post-deploy verification, "is the cluster working?" questions, and after any node or driver change. +--- + +# Validate a GPU cluster + +## Which validator + +- Slurm cluster → `scripts/validation/validate_slurm.py` (run on a cluster + node) +- Kubernetes cluster → `scripts/validation/validate_k8s.py` (run wherever + `kubectl` reaches the cluster) +- Not deployed yet / checking the provisioning environment → + `scripts/validation/deepops_doctor.py` (run from the repository root) + +All tools support `--json` and exit `0` only when every check passes. Full +contract: `docs/deepops/validation.md`. + +## Procedure + +1. Run the matching validator with `--json`. + + ```bash + python3 scripts/validation/validate_slurm.py --json + python3 scripts/validation/validate_k8s.py --json --cuda-smoke + ``` + + Use `--skip-gpu-job` (Slurm) or omit `--cuda-smoke` (Kubernetes) when a + strictly read-only check is required — for example on a busy production + cluster where a test job would queue behind real work. + +2. Read the verdict from the JSON, not from ad-hoc commands: + - `ok: true` — report healthy, include the key counts + (`nodes_total`, `gpus_configured`/`gpus_allocatable`). + - `ok: false` — report each entry in `failures` verbatim; they name the + failing subsystem and the next diagnostic step. + +3. When a GPU check fails, do not conclude "driver broken" from a bare + `nvidia-smi` over SSH — on Slurm nodes GPUs are hidden outside jobs. + Follow `skills/diagnose-driver-install/` instead. + +## Interpreting common results + +| Signal | Meaning | +|--------|---------| +| Slurm `nodes_unavailable > 0` | Node down/drained — `scontrol show node ` for the reason. | +| Slurm `gpus_configured: 0` | GRES not configured — check `config/group_vars/slurm-cluster.yml` GPU settings and rerun the Slurm playbook. | +| K8s `gpus_allocatable: 0` | GPU Operator stack not ready — `kubectl get pods -A \| grep -i nvidia`; first-deploy driver builds can take 10+ minutes. | +| K8s smoke pod `Pending` | No schedulable GPU or image pull failure — `kubectl -n deepops-validate describe pod`. | +| Direct `nvidia-smi` over SSH says "No devices were found" on a Slurm node | Expected GPU-hide behavior, not a failure. |