Truncate failure_message so a large failure cannot crash the worker#508
Merged
Merged
Conversation
…e worker markJobFailed() stored the failure message and captured output verbatim into the failure_message / output TEXT columns. A failure larger than the column (e.g. a database error that echoes a huge multi-row query) made the save throw, so the worker crashed while recording the failure: the job got stuck and the queue stalled. Byte-truncate both fields to the column size before saving.
A newer phpstan no longer reports an error on the dynamic $this->$action() call, so the @phpstan-ignore-next-line became an unmatched (failing) ignore. Cast the result to int to satisfy the declared return type without an ignore.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
QueuedJobsTable::markJobFailed()persisted the failure message (and captured output) verbatim into thefailure_message/outputTEXT columns. A failure larger than the column - for example a databaseQueryExceptionwhose message echoes a huge multi-rowINSERTwith all of its JSON values - made thesave()itself throwSQLSTATE[22001]: Data too long for column 'failure_message'.That means the worker crashes while recording a job failure: the job is left fetched-but-not-completed, the worker process dies, and (with stale
queue_processesslots) new workers can be blocked - the queue stalls instead of cleanly failing the one job.Fix
Byte-truncate
failure_messageandoutputto the column size (mb_strcut, multibyte-safe) before saving, via a newFAILURE_MESSAGE_MAX_LENGTHconstant (default 65535, the TEXT limit). Recording a failure can no longer overflow the column, so the worker fails the job cleanly and keeps running.No schema change required.
Notes
FAILURE_MESSAGE_MAX_LENGTHispublic const, so an app that widened the column can override it.testMarkJobFailedTruncatesOversizedMessage.