Skip to content

feat: add a broadcast channel for job state event notifications#1891

Open
Jeadie wants to merge 2 commits into
apache:mainfrom
spiceai:upstream/job-state-broadcast-channel
Open

feat: add a broadcast channel for job state event notifications#1891
Jeadie wants to merge 2 commits into
apache:mainfrom
spiceai:upstream/job-state-broadcast-channel

Conversation

@Jeadie

@Jeadie Jeadie commented Jun 23, 2026

Copy link
Copy Markdown

Which issue does this PR close?

None Closes #. Suggested by @milenkovicm that we should upstream our changes (see comment).

Rationale for this change

Useful in spiceai/datafusion-ballista for production features.

What changes are included in this PR?

Summary

Adds a tokio::sync::broadcast sender to SchedulerServer, subscribable via subscribe_job_updates(). On job state changes (queued / running / completed / failed / cancelled) the scheduler broadcasts a JobStateEvent carrying the job id and new state. Lets components holding a handle to the scheduler observe job lifecycle transitions without polling.

Relationship to the existing JobStatusSubscriber

Similar to the existing JobStatusSubscriber (mpsc) mechanism wired through submit_job(..., subscriber). This broadcast channel is a separate, fan-out style notification path; worth deciding the best DX between these.

JobStatusSubscriber (existing) JobStateEvent broadcast (this PR)
Channel mpsc - single consumer broadcast - fan-out
Scope 1 subscriber per submitted job, request-scoped Global; fires for all jobs
Consumer submitting gRPC client, via ExecuteQuery / GetJobStatusResult. in-process component(s), via subscribe_job_updates()
Payload Full protobuf JobStatus Lightweight JobStateEvent { job_id: String, state }

Both fire from the same lifecycle points in query_stage_scheduler.rs.

Ported from spiceai#15.

Jeadie added 2 commits June 23, 2026 13:47
Adds a tokio::sync::broadcast Sender to SchedulerServer, subscribable via
subscribe_job_updates(). On job state changes (queued/running/completed/
failed/cancelled) the scheduler broadcasts a JobStateEvent carrying the job
id and new state.

Ported from #15 for upstreaming.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant