Add AGENTS.md operating guide and agent skills

AGENTS.md

@@ -0,0 +1,107 @@
+# DeepOps agent operating guide
+
+This file is the entry point for AI agents (and new humans) operating this
+repository. DeepOps deploys and manages GPU clusters — Slurm or Kubernetes on
+NVIDIA GPU servers — using Ansible. Everything here is driven by playbooks
+against an inventory you control; there is no server component.
+
+Load only what the task needs: this file for orientation and the golden
+paths, `docs/` for depth, `skills/` for step-by-step procedures with failure
+handling.
+
+## Repository map
+
+| Path | Purpose |
+|------|---------|
+| `playbooks/` | Entry points. `slurm-cluster.yml` and `k8s-cluster.yml` are the two top-level cluster deploys; subdirectories hold component playbooks. |
+| `roles/` | Ansible roles (NVIDIA drivers, container toolkit, DGX software, monitoring, Slurm, storage). |
+| `config.example/` | Template configuration. Copy to `config/` and edit; never edit `config.example/` for a deployment. |
+| `config/` | Your site configuration and inventory (git-ignored; created by you). |
+| `scripts/validation/` | Machine-readable preflight and post-deploy validation. Start here to know if anything worked. |
+| `scripts/` | Setup and helper scripts (`setup.sh` installs Ansible and dependencies). |
+| `submodules/kubespray` | Kubernetes deployment engine. Must be initialized before Kubernetes work. |
+| `docs/` | Topic documentation: `deepops/`, `slurm-cluster/`, `k8s-cluster/`, `container/`, `airgap/`. |
+| `skills/` | Reusable agent procedures with preconditions, commands, expected output, and failure branches. |
+
+## First-time setup (once per provisioning machine)
+
+```bash
+git submodule update --init --recursive
+./scripts/setup.sh                      # installs Ansible + Galaxy dependencies
+cp -r config.example config             # then edit config/inventory
+python3 scripts/validation/deepops_doctor.py --json   # verify before deploying
+```
+
+The doctor must report `"ok": true` (or you must understand every failure)
+before you run any cluster playbook. With `--remote` it also proves SSH
+reachability to every inventory host.
+
+## Golden path: Slurm GPU cluster
+
+```bash
+# inventory groups: slurm-master, slurm-node (see config.example/inventory)
+ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml
+python3 scripts/validation/validate_slurm.py --json    # run on a cluster node
+```
+
+The validator must report `"ok": true` with `gpu_job_ok: true`. See
+`skills/deploy-slurm-cluster/` for the full procedure and failure branches.
+
+## Golden path: Kubernetes GPU cluster
+
+```bash
+# inventory groups: kube_control_plane, etcd, kube_node (see config.example/inventory)
+ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml
+python3 scripts/validation/validate_k8s.py --json --cuda-smoke
+```
+
+The validator must report `"ok": true` with `cuda_smoke_ok: true`. See
+`skills/deploy-k8s-gpu-cluster/` for the full procedure and failure branches.
+
+## Rules for operating this repository
+
+1. **Validate, don't assume.** Run the doctor before deploying and the
+   matching validator after. A playbook finishing with `failed=0` is not the
+   success signal; the validator's `"ok": true` is.
+2. **Never run a cluster playbook against an unreviewed inventory.** These
+   playbooks install drivers, change container runtimes, and can reboot
+   machines. Confirm the inventory lists exactly the intended hosts
+   (`ansible-inventory --list`, or the doctor's host count) first.
+3. **Driver installs can reboot nodes.** Schedule accordingly; never point a
+   first-time deploy at hosts with active users or workloads.
+4. **Preview when unsure.** `ansible-playbook --check --diff -l <host>` shows
+   most pending changes without applying them (some tasks don't support
+   check mode). Use `--limit` to scope any run.
+5. **Playbooks are idempotent; reruns are the normal recovery path.** After a
+   transient failure (package mirror timeout, network blip), rerun the same
+   playbook. A converged rerun reports `changed=0`.
+6. **Configuration lives in `config/`, not in role defaults.** Override
+   variables in `config/group_vars/`; do not edit roles for site-specific
+   values.
+
+## Gotchas that look like failures but are not
+
+- **`nvidia-smi` over SSH reports "No devices were found" on Slurm nodes.**
+  DeepOps hides GPUs from ordinary SSH sessions on cluster nodes; GPUs are
+  visible inside Slurm jobs. Test with
+  `srun --gpus=1 nvidia-smi`, or `validate_slurm.py`, never with bare SSH
+  `nvidia-smi`.
+- **Ansible fact caching can serve stale facts** when an inventory hostname
+  is reused for a different machine or after an OS reinstall. Rerun with
+  `--flush-cache`.
+- **Open vs proprietary NVIDIA kernel modules matter per GPU generation.**
+  Turing and newer support the open kernel modules; older GPUs (e.g. Pascal)
+  need `nvidia_driver_ubuntu_use_open_kernel_modules: false`. A wrong choice
+  produces `nvidia-smi: No devices were found` after a clean-looking install.
+  See `skills/diagnose-driver-install/`.
+- **Kubernetes playbooks fail on syntax/imports if `submodules/kubespray` is
+  not initialized** — the error mentions missing `kubespray_defaults` roles,
+  not submodules. Run `git submodule update --init --recursive`.
+
+## Contributing changes
+
+Run before pushing: `git diff --check`, YAML parse on changed files,
+`./scripts/deepops/ansible-lint-roles.sh`, and a focused
+`ansible-playbook --syntax-check` for changed playbooks. Public CI runs lint,
+setup, and molecule checks on every PR. Deployment-affecting changes need
+GPU-backed validation evidence in the PR body.

README.md

@@ -32,6 +32,8 @@ Latest release: [DeepOps 26.05 Release](https://github.com/NVIDIA/deepops/releas
 
 It is recommended to use the latest release branch for stable code (linked above). All development takes place on the master branch, which is generally [functional](docs/deepops/testing.md) but may change significantly between releases.
 
+Operating this repository with an AI agent (or onboarding as a new human)? Start with [AGENTS.md](AGENTS.md) for the golden paths, operating rules, and [machine-readable validation tools](docs/deepops/validation.md), and [skills/](skills/README.md) for step-by-step procedures.
+
 ## Deployment Requirements
 
 ### Provisioning System

skills/README.md

@@ -0,0 +1,24 @@
+# DeepOps skills
+
+Reusable, self-contained procedures for operating DeepOps, written for AI
+agents and equally usable by humans. Each skill is a directory containing a
+`SKILL.md` with YAML frontmatter (`name`, `description`) followed by
+preconditions, exact commands, expected outputs, and failure branches.
+
+The format follows the emerging cross-tool agent-skills convention: agent
+frameworks that support skill discovery can load these directly, and any
+agent (or person) can simply read the relevant `SKILL.md` before acting.
+
+| Skill | Use when |
+|-------|----------|
+| [deploy-slurm-cluster](deploy-slurm-cluster/SKILL.md) | Deploying or rebuilding a Slurm GPU cluster. |
+| [deploy-k8s-gpu-cluster](deploy-k8s-gpu-cluster/SKILL.md) | Deploying or rebuilding a Kubernetes GPU cluster. |
+| [validate-gpu-cluster](validate-gpu-cluster/SKILL.md) | Health checks and post-deploy verification. |
+| [diagnose-driver-install](diagnose-driver-install/SKILL.md) | NVIDIA driver failures, `nvidia-smi` errors, GPU pods crash-looping. |
+
+Start with [AGENTS.md](../AGENTS.md) at the repository root for orientation,
+golden paths, and operating rules.
+
+Contributions should keep skills honest: every command must work as written
+from a clean checkout, and failure branches should come from real observed
+failures, not speculation.

skills/deploy-k8s-gpu-cluster/SKILL.md

@@ -0,0 +1,72 @@
+---
+name: deploy-k8s-gpu-cluster
+description: Deploy a Kubernetes GPU cluster with DeepOps (Kubespray + GPU Operator) and prove it schedules GPU pods. Use when asked to deploy or rebuild Kubernetes on GPU servers with this repository.
+---
+
+# Deploy a Kubernetes GPU cluster
+
+## Preconditions
+
+- Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver
+  installs may reboot them; no active users or workloads).
+- SSH access from the provisioning machine to every host as a sudo-capable
+  user.
+- `submodules/kubespray` initialized — Kubernetes playbooks fail on missing
+  `kubespray_defaults` role imports without it.
+- Run everything from the repository root.
+
+## Procedure
+
+1. Prepare the environment and verify it:
+
+   ```bash
+   git submodule update --init --recursive
+   ./scripts/setup.sh
+   cp -r config.example config
+   ```
+
+2. Edit `config/inventory`: control plane nodes under
+   `[kube_control_plane]` and `[etcd]`, workers under `[kube_node]` (a
+   single machine can hold all three roles).
+
+3. Preflight — must pass before deploying:
+
+   ```bash
+   python3 scripts/validation/deepops_doctor.py --remote --json
+   ```
+
+4. Deploy:
+
+   ```bash
+   ansible-playbook -l k8s_cluster playbooks/k8s-cluster.yml
+   ```
+
+   This runs Kubespray and installs the NVIDIA GPU Operator. Expect
+   roughly 45–90 minutes on a first run.
+
+5. Validate — the success signal is this, not the play recap:
+
+   ```bash
+   python3 scripts/validation/validate_k8s.py --json --cuda-smoke
+   ```
+
+   Require `"ok": true` with `nodes_ready == nodes_total`,
+   `gpus_allocatable > 0`, and `cuda_smoke_ok: true`.
+
+## Failure branches
+
+- **Playbook fails on a transient error**: rerun the same playbook;
+  Kubespray is rerun-safe. A converged rerun reports `changed=0`.
+- **Syntax/import error mentioning `kubespray_defaults`**: the submodule is
+  not initialized; run `git submodule update --init --recursive`.
+- **`gpus_allocatable: 0`**: the GPU Operator stack is not ready. Check
+  `kubectl get pods -A | grep -i nvidia` — the driver DaemonSet can take
+  10+ minutes on first deploy; if pods are crash-looping, follow
+  `skills/diagnose-driver-install/`.
+- **CUDA smoke pod stuck `Pending`**: `kubectl -n deepops-validate
+  describe pod deepops-validate-cuda` — usually no allocatable GPU
+  (see above) or an image pull problem on airgapped networks (use
+  `--cuda-image` to point at a mirrored image).
+- **Single-node clusters**: control plane taints are handled by the
+  playbook for the single-node case; if pods stay Pending on a multi-role
+  node, check taints with `kubectl describe node <name> | grep -i taint`.

skills/deploy-slurm-cluster/SKILL.md

@@ -0,0 +1,72 @@
+---
+name: deploy-slurm-cluster
+description: Deploy a Slurm GPU cluster with DeepOps and prove it works. Use when asked to deploy, install, or rebuild Slurm on one or more GPU servers with this repository.
+---
+
+# Deploy a Slurm GPU cluster
+
+## Preconditions
+
+- Ubuntu 22.04/24.04 or RHEL/Rocky 8/9 hosts you may fully manage (driver
+  installs may reboot them; no active users or workloads).
+- SSH access from the provisioning machine to every host as a sudo-capable
+  user.
+- Run everything from the repository root.
+
+## Procedure
+
+1. Prepare the environment and verify it:
+
+   ```bash
+   git submodule update --init --recursive
+   ./scripts/setup.sh
+   cp -r config.example config
+   ```
+
+2. Edit `config/inventory`: put the controller under `[slurm-master]` and
+   compute nodes under `[slurm-node]` (a single machine can be both). Set
+   the connection user in `[all:vars]` if not root.
+
+3. Preflight — must pass before deploying:
+
+   ```bash
+   python3 scripts/validation/deepops_doctor.py --remote --json
+   ```
+
+   Fix anything in `failures` (each check's `detail` says how) and rerun.
+
+4. Deploy:
+
+   ```bash
+   ansible-playbook -l slurm-cluster playbooks/slurm-cluster.yml
+   ```
+
+   This installs NVIDIA drivers, builds and configures Slurm, and sets up
+   munge, NFS, and node health checks. Expect roughly 30–60 minutes on a
+   first run.
+
+5. Validate on a cluster node — the success signal is this, not the play
+   recap:
+
+   ```bash
+   python3 scripts/validation/validate_slurm.py --json
+   ```
+
+   Require `"ok": true` with `gpu_job_ok: true` and
+   `nodes_unavailable: 0`.
+
+## Failure branches
+
+- **Playbook fails on a transient error** (mirror timeout, apt lock,
+  network blip): rerun the same playbook; it is idempotent. A converged
+  rerun ends with `changed=0`.
+- **`nvidia-smi` works in the validator's srun job but "fails" over SSH**:
+  that is the login GPU-hide behavior, not an error (see AGENTS.md
+  gotchas).
+- **`gpu_job_ok: false` with driver errors**: follow
+  `skills/diagnose-driver-install/`.
+- **Node shows `down` or `drained` in `node_states`**: check
+  `scontrol show node <name>` for the reason; after fixing, resume with
+  `scontrol update nodename=<name> state=resume`.
+- **Wrong or stale host facts after reprovisioning a node**: rerun with
+  `--flush-cache`.

skills/diagnose-driver-install/SKILL.md

@@ -0,0 +1,82 @@
+---
+name: diagnose-driver-install
+description: Diagnose NVIDIA driver installation failures on DeepOps-managed nodes — nvidia-smi errors, "No devices were found", DKMS build failures, or GPU pods crash-looping. Use before reinstalling anything.
+---
+
+# Diagnose an NVIDIA driver install
+
+Work through these in order; most "driver failures" are one of the first
+three and need no reinstall.
+
+## 1. Are you being fooled by GPU hiding?
+
+On DeepOps **Slurm** nodes, GPUs are hidden from ordinary SSH sessions by
+design. Bare `nvidia-smi` over SSH reporting `No devices were found` on an
+otherwise healthy node is expected.
+
+```bash
+srun --gpus=1 nvidia-smi          # the authoritative test on Slurm nodes
+```
+
+If the srun job sees the GPU, the driver is fine. Stop here.
+
+## 2. Is the hardware visible at all?
+
+```bash
+lspci | grep -i nvidia
+```
+
+No output → not a driver problem. The GPU is absent, unseated, or bound by
+VFIO passthrough or platform firmware; escalate to hardware support before
+touching software.
+
+## 3. Open vs proprietary kernel modules
+
+`nvidia-smi: No devices were found` immediately after a clean install is the
+classic symptom of the wrong module flavor for the GPU generation:
+
+- Turing and newer (T4, A100, H100, RTX PRO Blackwell): open kernel modules
+  supported — DeepOps default `nvidia_driver_ubuntu_use_open_kernel_modules: true`
+  is correct.
+- Pascal and older (P100, GTX 10xx): open modules are **not** supported —
+  set `nvidia_driver_ubuntu_use_open_kernel_modules: false` in
+  `config/group_vars/all.yml` and rerun the driver play.
+
+Check what is loaded: `modinfo nvidia | grep -i license` (open modules say
+MIT/GPL, proprietary says NVIDIA).
+
+## 4. Did the kernel module actually build and load?
+
+```bash
+dkms status                        # driver module state per kernel
+dmesg | grep -iE 'nvidia|nvrm' | tail -20
+lsmod | grep nvidia
+```
+
+- DKMS shows an error for the running kernel → usually missing headers
+  (`linux-headers-$(uname -r)`) or a kernel updated after the driver
+  install. Install headers or reboot into the matching kernel, then rerun
+  the driver play.
+- Module loaded but `nvidia-smi` fails → check `dmesg` for RmInitAdapter or
+  fallen-off-the-bus errors; these are hardware/firmware territory.
+
+## 5. Rerun the play, don't hand-fix
+
+After correcting configuration, converge with the playbook rather than
+manual package surgery, then validate:
+
+```bash
+ansible-playbook -l <host> playbooks/nvidia-software/nvidia-driver.yml
+python3 scripts/validation/validate_slurm.py --json    # or validate_k8s.py
+```
+
+Reruns are idempotent; a healthy converged rerun reports `changed=0`.
+
+## Kubernetes-specific notes
+
+GPU Operator crash-looping driver pods: `kubectl logs -n <gpu-operator
+namespace> <driver-pod>` usually names the same root causes — missing
+headers, wrong module flavor, or a node reboot needed. Fix via the operator
+values or node state, not by installing drivers by hand on the host (the
+operator owns the driver on Kubernetes nodes unless DeepOps was configured
+for host drivers).