Skip to content

[codex] Fix init provider detection and runtime support#11

Draft
rendrag-git wants to merge 1 commit into
masterfrom
codex/fix-init-runtime-provider-detection
Draft

[codex] Fix init provider detection and runtime support#11
rendrag-git wants to merge 1 commit into
masterfrom
codex/fix-init-runtime-provider-detection

Conversation

@rendrag-git

@rendrag-git rendrag-git commented May 2, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes #1, #2, #4, #7, #9, and #10. Partially addresses the scoring cleanup from #3.

  • make oc-bench init fail closed when detection finds only detect-only providers instead of silently writing default vLLM config
  • route explicit --no-detect --vllm-* config through vLLM provider shaping so GPT-OSS gets reasoning_effort=low
  • add --probe-hosts plumbing for init detection and scan OpenClaw profiles inside Incus/Docker runtimes
  • support incus:<instance> command execution paths for OpenClaw version, gateway start/stop, preflight, provider health, and backend runs while preserving Docker container behavior
  • clarify top-level help for benchmark preflight vs provider-route provider-preflight
  • clean up scoring intent comments and the duplicate preserved-file failure branch

Review follow-up

  • moved runtime parsing helpers into openclaw_bench.container to avoid duplicate backend/preflight logic
  • made provider-health runtime parsing unconditional so legacy plain container names still resolve as Docker
  • added runtime-aware gateway stop and profile-scoped Incus gateway logs
  • confirmed _parse_csv already exists in openclaw_bench/cli.py and is not a new runtime crash
  • expanded --probe-hosts help text to mention bridge/LAN addresses

Validation

  • python3 -m pytest tests/test_providers_detect.py tests/test_providers_init_integration.py tests/test_quickstart.py tests/test_preflight.py tests/test_backend.py tests/test_scoring.py tests/test_container.py tests/test_providers_preflight.py -q -> 153 passed
  • Full python3 -m pytest -q was also attempted before PR creation and hit an unrelated local HTTP fixture failure: tests/test_serve.py::ServeTests::test_api_base_context_window_probe_failure_is_classified raised ConnectionResetError: [Errno 54] Connection reset by peer; rerunning that single test reproduced the same failure.

Notes

  • .github/ is intentionally not included because it was already untracked before this work.
  • The task-suite expansion ideas from Behavior coverage thin spots in tier suites #5 are intentionally deferred until live calibration data exists; changing the benchmark surface now would be premature.

@rendrag-git rendrag-git force-pushed the codex/fix-init-runtime-provider-detection branch from bd707ef to ca2dc8f Compare May 2, 2026 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

init silently writes broken vLLM config when detection finds non-vLLM provider

2 participants