Skip to content

Add hierarchical queue capacities#29

Merged
steve-the-edwards merged 14 commits intoblock:mainfrom
steve-the-edwards:sedwards/no-ticket/resource-aware-scheduling
Apr 21, 2026
Merged

Add hierarchical queue capacities#29
steve-the-edwards merged 14 commits intoblock:mainfrom
steve-the-edwards:sedwards/no-ticket/resource-aware-scheduling

Conversation

@steve-the-edwards
Copy link
Copy Markdown
Collaborator

Summary

  • add /-delimited hierarchical queue names and --queue-capacity support across the MCP server and tq CLI
  • preserve FIFO when parent scopes are the bottleneck, and tighten the shared-connection transaction handling around task acquisition
  • document the Android multi-emulator pattern and add regression coverage for queue scheduling and installed-entrypoint argument parsing

Testing

  • uv run --extra dev pytest tests/test_queue.py tests/test_tq_cli.py
  • amp review \"origin/main...HEAD\"

steve-the-edwards and others added 9 commits April 20, 2026 11:32
@steve-the-edwards
Copy link
Copy Markdown
Collaborator Author

@codex review

@steve-the-edwards
Copy link
Copy Markdown
Collaborator Author

@builderbot0 review

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4000bbdd66

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread queue_core.py
@steve-the-edwards
Copy link
Copy Markdown
Collaborator Author

[AI:] Follow-up pushed in 9c4b97d.

This narrows the fix to the blocker we discussed:

  • queue_core.is_task_queue_process() now recognizes the installed tq entrypoint as a live queue owner, so sibling-scope cleanup does not incorrectly reap active tq jobs
  • added a regression test covering an installed tq command line

I also clarified the intended behavior in the docs:

  • sibling queues sharing a parent scope are best-effort for parent-capacity contention, not strict FIFO across siblings
  • --queue-capacity settings are process-local and should match across entrypoints sharing the same queue DB

Validation run locally:

  • python -m py_compile queue_core.py task_queue.py tq.py tests/test_queue.py tests/test_tq_cli.py
  • uv run --extra dev pytest tests/test_queue.py tests/test_tq_cli.py
  • uv run --extra dev ruff check queue_core.py task_queue.py tq.py tests/test_queue.py tests/test_tq_cli.py README.md

@steve-the-edwards steve-the-edwards marked this pull request as ready for review April 21, 2026 15:54
Comment thread README.md Outdated

```python
run_task("./gradlew connectedDebugAndroidTest", queue_name="gradle/emu-5557", env_vars="ANDROID_SERIAL=127.0.0.1:5557", ...)
run_task("./gradlew connectedDebugAndroidTest", queue_name="gradle/emu-5559", env_vars="ANDROID_SERIAL=127.0.0.1:5559", ...)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if it's just not a particularly illustrative example or an actual flaw, but this Gradle command also runs assemble, which is still undesirable to be run in parallel at the same time, right?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI:] Good catch. The current queue-capacity model applies to the entire command lifetime, not to phases inside a single command, so if each leaf task is ./gradlew connectedDebugAndroidTest, any assemble work inside those invocations can still overlap.

I do not think that needs a code change in this PR; phase-aware scheduling would be a different model. I pushed 1622101, which updates the README to clarify that limitation and show the supported pattern: queue shared Gradle prep/build once, then fan out per-emulator commands that reuse those outputs. If the emulator phase no longer needs Gradle at all, it should be queued outside the shared gradle scope entirely.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still feel like it's a lot of words in the README that no one is going to read, which could be replaced by a simple example that doesn't include any gradle assembly and is truly parallelizable between emulators.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI:] Agreed, and I updated the example accordingly in 0324a72. The hierarchical-queues section now leads with the split pattern directly: one shared assemble/prep step, then separate per-emulator run steps that reuse those outputs, instead of showing connectedDebugAndroidTest as the primary leaf example.

@steve-the-edwards steve-the-edwards merged commit 6c66540 into block:main Apr 21, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants