fix(jobs): Fix pause/resume checks in kube jobs

[matthewgrossman]

Summary

Fixes AIRCORE-853: Job pause/resume races with errored-pod detection in the K8s reconciler.

When a K8s Job is suspended (paused), Kubernetes kills running pods via SIGTERM. The reconciler's sync_step was checking for errored pods before checking if the job was suspended. The terminated pod (exit code 137 from SIGTERM) was misclassified as an error, causing the job to transition to error instead of paused.

Root cause: In kubernetes_job.py, the if errored_pods: check had priority over the if is_suspended: check. The same race existed for cancellation.

Fix: Moved the is_suspended and is_cancelling checks before the errored_pods check. Pods killed during suspension or cancellation are expected — they must not cause the reconciler to report ERROR.

Changes

• services/core/jobs/.../kubernetes_job.py — Reordered sync_step to check suspended/cancelling state before errored pods
• services/core/jobs/tests/.../test_kubernetes_backend.py — Added 3 regression tests:
  • test_sync_job_paused_with_errored_pods_from_sigterm — suspended + errored pod → PAUSED
  • test_sync_job_pausing_with_errored_pods_from_sigterm — suspended + mix of running/errored → PAUSING
  • test_sync_job_cancelling_with_errored_pods — cancelling + errored pod → CANCELLED
• e2e/test_jobs.py — Removed @pytest.mark.skip(reason="AIRCORE-853") from test_job_pause_resume and test_job_pause_and_cancel; reduced sleep 300 → sleep 30 so tests complete within the default 120s timeout

Test plan

• All 63 unit tests in test_kubernetes_backend.py pass
• test_job_pause_resume e2e passes on minikube (3/3 consecutive runs, no flakes)
• test_job_pause_and_cancel e2e passes on minikube (3/3 consecutive runs, no flakes)

Summary by CodeRabbit

• Bug Fixes

  • Improved job pause, resume, and cancel handling so Kubernetes-terminated pods no longer incorrectly mark jobs as failed.
  • Fixed edge cases where paused, pausing, or cancelling jobs could briefly show an error state instead of the expected status.
  • Reduced the runtime of pause-related end-to-end checks to make them faster while preserving coverage.

• Tests

  • Added regression coverage for suspended and cancelling jobs with errored pods to verify correct status transitions.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fdf8053c-ecae-4fdf-a41e-3833c2af31ca

📥 Commits

Reviewing files that changed from the base of the PR and between a012393 and ac4e5d1.

📒 Files selected for processing (3)

• e2e/test_jobs.py
• services/core/jobs/src/nmp/core/jobs/controllers/backends/kubernetes/kubernetes_job.py
• services/core/jobs/tests/controllers/test_kubernetes_backend.py

---

📝 Walkthrough

Walkthrough

Kubernetes job status mapping now checks suspended/cancelling before errored pods. Regression tests cover suspended, pausing, and cancelling jobs with errored pods. Two pause-related E2E tests are re-enabled with shorter step sleeps.

Changes

Kubernetes pause and cancel handling

Layer / File(s)	Summary
Status mapping order services/core/jobs/src/nmp/core/jobs/controllers/backends/kubernetes/kubernetes_job.py	The Kubernetes job status mapper evaluates suspended and cancelling states before treating errored pods as an error condition.
Backend regression coverage services/core/jobs/tests/controllers/test_kubernetes_backend.py	Three Kubernetes backend sync tests assert paused, pausing, and cancelled outcomes when pods report errors during suspend or cancel timing.
Pause E2E tests e2e/test_jobs.py	Two pause-related E2E tests are unskipped and their long-running step sleeps are reduced from 300 seconds to 30 seconds.

Suggested reviewers

• svvarom
• a2bondar

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: fixing Kubernetes job pause/resume checks and related status handling.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch mgrossman/aircore-853-job-pauseresume-races-with-errored-pod-detection-in-k8s

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	21898/28725	76.2%	61.0%
Integration Tests	12583/27405	45.9%	19.2%