Skip to content

fix: restart containerized exporters when Docker restarts#1363

Merged
michael-balint merged 1 commit into
masterfrom
dholt/exporter-service-partof
Jul 15, 2026
Merged

fix: restart containerized exporters when Docker restarts#1363
michael-balint merged 1 commit into
masterfrom
dholt/exporter-service-partof

Conversation

@dholt

@dholt dholt commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add PartOf=docker.service to the six containerized monitoring service units (node exporter, DCGM exporter, Slurm exporter, Prometheus, Grafana, Alertmanager).
  • Root cause of node exporters don't work after initial run of slurm playbook #1267: the units declare Requires=docker.service without PartOf=, so an explicit Docker restart propagates as a one-way stop to the dependents and nothing starts them again. A first slurm-cluster.yml run on GPU nodes triggers exactly this — the NVIDIA container runtime configuration restarts Docker after the node exporter is already up — leaving exporters down until reboot.
  • With PartOf=, a Docker restart propagates as a restart, so the units come back automatically (including after any manual systemctl restart docker).

Validation

  • git diff --check; full role lint: 0 failures, 0 warnings.
  • Live single-node GPU validation summary will be posted as a follow-up comment before review: fresh slurm-cluster.yml run, then asserting the node exporter is active at the end of the run and recovers after an explicit Docker restart.

Notes

The monitoring container units (node exporter, DCGM exporter, Slurm
exporter, Prometheus, Grafana, Alertmanager) declare Requires=docker.service
without PartOf=. When Docker is restarted after the units are up — which
the NVIDIA container runtime configuration does during a first
slurm-cluster.yml run on GPU nodes — systemd stops the dependent units
and nothing starts them again, so exporters stay down until reboot.

Add PartOf=docker.service so a Docker restart propagates as a restart to
the container units instead of a one-way stop.

Fixes #1267
@dholt

dholt commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Live validation summary

Validated on a disposable dual-GPU server (Ubuntu 22.04) with a fresh full slurm-cluster.yml deployment from a branch carrying this change:

  • The rendered unit files contain PartOf=docker.service.
  • Reproduction of the reported failure sequence: the node exporter was deployed and active, then the DCGM exporter playbook's NVIDIA container runtime configuration fired its restart docker handler — the exact sequence that leaves exporters dead on master. With this change the node exporter was active at the end of the run.
  • Explicit restart recovery: systemctl restart docker on the node; both the node exporter and Slurm exporter units returned to active automatically, and :9100/metrics served node metrics.
  • Slurm remained healthy throughout (node idle, srun --gpus=1 nvidia-smi passing).

@dholt dholt marked this pull request as ready for review July 15, 2026 17:52
@dholt dholt requested a review from michael-balint July 15, 2026 17:52
@michael-balint michael-balint merged commit 8093c9a into master Jul 15, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

node exporters don't work after initial run of slurm playbook

2 participants