Emit celery.failed_task=0 on success so the failure metric stays dense#339
Merged
Conversation
failed_task was a TIMER emitted only on failure, so lemur.celery.failed_task.count was sparse and the persistent-failures monitor had to default_zero it. Make failed_task a counter and emit 0 on every task success (low-cardinality task_name tag only), so it reads 0 when healthy and >0 on failures without query-side gap-filling. This renames the DD metric to lemur.celery.failed_task (counters have no .count suffix); the monitor query changes in a companion terraform-config PR. Deploy this first so the counter is flowing before that switch.
maperu
marked this pull request as ready for review
July 20, 2026 20:57
maperu
enabled auto-merge (squash)
July 20, 2026 21:00
evan-datadog
approved these changes
Jul 20, 2026
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: queued
mergequeue-status: in_progress
mergequeue-status: retrying
mergequeue-status: removed
mergequeue-status: error
and removed
mergequeue-status: queued
mergequeue-status: in_progress
mergequeue-status: retrying
mergequeue-status: removed
labels
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: waiting
and removed
mergequeue-status: error
labels
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: done
and removed
mergequeue-status: waiting
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
celery.failed_task was emitted only on task failure as a TIMER, so lemur.celery.failed_task.count was sparse (no data when nothing fails) and the persistent-failures monitor (299971237) had to wrap it in default_zero to read 0 when healthy.
This makes failed_task a counter and emits 0 on every task success, with a low-cardinality task_name tag only, not the full request tags that carry task_id. So the metric is continuously present, 0 when healthy and greater than 0 on failures, and the monitor can drop default_zero.
Heads up: this renames the Datadog metric from lemur.celery.failed_task.count to lemur.celery.failed_task (counters have no .count suffix). The monitor query changes in a companion terraform-config PR, and this one should deploy first so the new counter is flowing before that monitor switches, otherwise it briefly queries a metric with no data. Anything else reading failed_task.count (e.g. the Celery Task Health group on the nqu-dbc-mtt dashboard) needs the same rename. This diverges from upstream, which emits failed_task as a TIMER.