Skip to content

Replace third-party Slurm exporter with a DeepOps-owned implementation#1366

Merged
michael-balint merged 1 commit into
masterfrom
dholt/owned-slurm-exporter
Jul 15, 2026
Merged

Replace third-party Slurm exporter with a DeepOps-owned implementation#1366
michael-balint merged 1 commit into
masterfrom
dholt/owned-slurm-exporter

Conversation

@dholt

@dholt dholt commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Replace the third-party Slurm exporter (unmaintained since 2024; required a build-time parser patch as of fix: build the Slurm exporter image locally with a current base #1365) with a small DeepOps-owned exporter maintained in this repository: standard-library-only Go, no external modules, built by the role from in-repo source — the external clone is gone entirely.
  • Emits exactly the metric names the shipped Grafana dashboard consumes (slurm_nodes_*, slurm_queue_*, slurm_scheduler_*) plus the aggregate slurm_cpus_* gauges, with all series always present (idle clusters omit backfill statistics from sdiag; the gauges still emit as 0).
  • Locale- and truncation-safe Slurm client invocations by construction; collectors degrade gracefully (slurm_exporter_collector_errors) instead of crash-looping; 8 unit tests including fixtures for Slurm 26.05 output shapes.
  • Intentionally dropped relative to the previous exporter, as nothing in this repository consumes them: per-node, per-user, and fair-share series (the sshare bind mount is removed accordingly). Sites needing them can set slurm_exporter_build_image: false and supply their own image.

Validation

  • go vet, gofmt, and the unit suite pass; on a Slurm-less host the exporter serves HTTP 200 with collector_errors 4 rather than failing.
  • Full role lint (0 failures) and playbook syntax check.
  • Live single-GPU deployment (Ubuntu 22.04, Slurm 26.05.1 via a fresh slurm-cluster.yml run): image 2.0.0 built from in-repo source by the role, service active with NRestarts=0, /metrics HTTP 200 serving 31 slurm_* series covering the dashboard contract completely, collector_errors 0, and automatic recovery after systemctl restart docker.

Notes

The exporter previously built from a third-party repository that has
been unmaintained since 2024 and needed a build-time patch for a parser
panic. Replace it with a small dependency-free Go exporter maintained
in this repository (standard library only, no external modules).

The owned exporter emits exactly the metric names the shipped Grafana
dashboard consumes (slurm_nodes_*, slurm_queue_*, slurm_scheduler_*)
plus the aggregate slurm_cpus_* gauges, uses locale- and
truncation-safe Slurm client invocations, degrades gracefully when a
collector fails (slurm_exporter_collector_errors), and ships unit
tests. Intentionally dropped relative to the previous exporter, as
nothing in the repo consumes them: per-node series, per-user series,
and fair-share series (the sshare bind mount is removed accordingly).
Sites needing those can set slurm_exporter_build_image: false and
supply their own image.
@dholt dholt marked this pull request as ready for review July 15, 2026 22:04
@dholt dholt requested a review from michael-balint July 15, 2026 22:04
@michael-balint michael-balint merged commit 477e651 into master Jul 15, 2026
41 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants