Skip to content

fix(transfer): Prevent JobPollingService thread crash on stale job claims#1491

Merged
yukthagaduputi merged 1 commit into
dtinit:masterfrom
yukthagaduputi:fix-job-polling-crash
May 19, 2026
Merged

fix(transfer): Prevent JobPollingService thread crash on stale job claims#1491
yukthagaduputi merged 1 commit into
dtinit:masterfrom
yukthagaduputi:fix-job-polling-crash

Conversation

@yukthagaduputi
Copy link
Copy Markdown
Collaborator

This PR fixes a critical process-terminating thread crash in the JobPollingService loop that occurs when a transfer worker attempts to claim a job that has already been claimed and has credentials stored by another worker instance.

Problem
In a distributed, highly concurrent deployment with multiple worker instances, a worker can poll a job ID from an eventually consistent database index that looks free (CREDS_AVAILABLE) but has actually already been claimed by a faster peer instance.

When our worker performs a strongly consistent read (store.findJob) to retrieve the job details, it detects the instanceId mismatch (indicating the job belongs to another worker) and marks the job's state as CANCELED in memory.

However, in the old implementation:

  1. JobPollingService.tryToClaimJob failed to verify whether the retrieved existingJob was null or canceled, proceeding blindly to build the updatedJob to claim it.
  2. Constructing this job object to transition to state CREDS_ENCRYPTION_KEY_GENERATED threw a validation exception (IllegalStateException) because credentials (encryptedAuthData) had already been set by the peer worker.
  3. Because the PortabilityJob builder execution occurred outside the main try-catch block in tryToClaimJob, this exception was uncaught.
  4. This uncaught thread failure propagated up, terminating the periodic JobPollingService thread and causing the entire container sandbox to crash.

Solution
Implemented two layers of safety in JobPollingService.java to resolve this:

  1. Proactive Prevention (Early Abort): Added null and CANCELED state checks immediately after retrieving the job from the JobStore. If the job has been deleted or marked canceled (which happens on instanceId mismatch), the worker aborts the claim attempt early and returns false safely.
  2. Reactive Safety (Validation Safety Net): Wrapped the PortabilityJob builder execution in a local try-catch block to safely handle any unexpected IllegalStateException validation errors during object construction, returning false (handled failure) instead of propagating and crashing the thread.

These changes ensure that failing to claim a job (due to losing a race) is treated as a handled, temporary failure, allowing the worker to complete the current polling iteration normally and try again in the next cycle instead of crashing.

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 14, 2026

CLA assistant check
All committers have signed the CLA.

@yukthagaduputi yukthagaduputi requested a review from lisad May 15, 2026 02:24
Copy link
Copy Markdown
Member

@lisad lisad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good for robustness

@yukthagaduputi yukthagaduputi merged commit a384d58 into dtinit:master May 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants