fix: build the Slurm exporter image locally with a current base

[dholt]

Summary

• Stop pulling deepops/prometheus-slurm-exporter:latest — a 2020-era external image with a single unpinned tag and no in-repo build path. Its old glibc is incompatible with the Slurm client binaries bind-mounted from modern hosts, so the service crash-loops on current deployments (The Docker image 'deepops/prometheus-slurm-exporter' is based on too old version of Ubuntu. #1328, Errors in deepops/slurm-exporter #1307).
• Build the exporter image locally from a bundled Dockerfile instead: vpenso prometheus-slurm-exporter 0.20 (same metric names as the shipped Grafana dashboard) on an Ubuntu 24.04 base with the system libraries the mounted Slurm binaries require. The tag is pinned; no registry access is needed.
• Sites can keep using their own image with slurm_exporter_build_image: false and a slurm_exporter_container override (the unit then retains the pull step).
• Mount the full Slurm install lib directory (superset of the previous lib/slurm plugin mount) so libslurm resolves alongside the plugins, and mount sshare, which the exporter's fair-share collector executes but the unit never provided.
• Create the slurm system user in the image (the mounted Slurm clients refuse to parse slurm.conf without it) and patch an upstream exporter bug at build time: the node collector's sinfo -O call uses default field widths, so node names of 20+ characters merge into the next column and panic the exporter on every scrape (present through upstream master; patch is a candidate for upstream contribution).

Validation

• YAML parse and full role lint: 0 failures; ansible-playbook --syntax-check playbooks/slurm-cluster.yml.
• Live single-node GPU validation summary will be posted as a follow-up comment before review: fresh Slurm deployment with the exporter role, asserting the image builds locally, the service stays active (no crash loop), and /metrics serves slurm_* series.

Notes

• Fixes The Docker image 'deepops/prometheus-slurm-exporter' is based on too old version of Ubuntu. #1328. The previously shipped Grafana dashboard is unchanged — the 0.20 exporter emits the same metric names.
• Alternative considered: switching to the actively maintained rivosinc exporter; rejected for this fix because its metric names would break the shipped dashboard. Candidate for a future train.

[dholt]

Live validation summary

Validated on a disposable dual-GPU server (Ubuntu 22.04) with a fresh Slurm 26.05.1 deployment; the exporter role was applied from this branch. Live validation surfaced three real defects that are all addressed in the final head — each was reproduced, fixed, and re-verified on the node:

1. The container needs a resolvable slurm system user or the mounted Slurm clients refuse to parse slurm.conf — added to the image.
2. The exporter's fair-share collector executes sshare, which the unit never bind-mounted (a latent gap that predates this change) — mount added.
3. An upstream exporter bug present through vpenso master: the node collector calls sinfo -O without field widths, so node names of 20+ characters truncate into the next column and the exporter panics with index out of range on every scrape. Patched at image build with explicit no-truncation field widths; candidate for an upstream contribution.

Final state on the node: image deepops/prometheus-slurm-exporter:0.20-1 built locally, service active with NRestarts=0 (the original issue was a crash loop), /metrics returning HTTP 200 with 48 slurm_* series (slurm_nodes_idle 1, slurm_cpus_total 32, slurm_queue_pending 0), and the service recovering automatically after a Docker restart.

[dholt]

The upstream exporter bug patched at image build here (node collector panic from sinfo -O field-width truncation on 20+ character node names) has been submitted upstream: vpenso/prometheus-slurm-exporter#128. If it merges, a future release can drop the build-time patch by bumping the pinned exporter version.