Skip to content

docs: add documentation for automated EvalBench evaluation workflow and test configuration #172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
omkargaikwad23 merged 2 commits into from
May 8, 2026
+25 −0
Merged

docs: add documentation for automated EvalBench evaluation workflow and test configuration #172

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8296b10
docs: add documentation for automated EvalBench evaluation workflow a…
omkargaikwad23 May 8, 2026
0dedfcc
docs: update evaluation pipeline instructions to reference Cloud Buil…
omkargaikwad23 May 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions DEVELOPER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,31 @@ All tools are currently tested in the [MCP Toolbox GitHub](https://github.com/go

The skills themselves are validated using the `skills-validate.yml` workflow.

### Automated Skill Evaluations (EvalBench)

This repository uses the [EvalBench framework](https://github.com/GoogleCloudPlatform/evalbench) to automatically evaluate the quality, multi-turn conversational capabilities, and skill execution of the extension.

Evaluations run automatically via Cloud Build (`cloudbuild.yaml`) on pull requests when the `ci:run-evals` or `autorelease: pending` label is applied. Because tests run against a live Cloud SQL instance, credentials are securely injected by Secret Manager during CI.

#### Understanding Evaluation Files

All evaluation configurations and datasets are located in the [`evals/`](evals/) directory:

* **Conversational Datasets (`*_dataset.json`):** Define test scenarios for different models (e.g., `gemini_dataset.json`, `claude_dataset.json`). Each scenario contains:
* `starting_prompt`: The initial prompt sent to the agent.
* `conversation_plan`: Instructions for the simulated user LLM to drive multi-turn interactions.
* `expected_trajectory`: The sequence of tool/skill calls expected to successfully complete the task.
* **Run Configurations (`*_run_config.yaml`):** Configure the EvalBench orchestrator, target model configs, and qualitative/performance scorers (e.g., goal completion, behavioral metrics, latency, token consumption).

#### Maintaining and Adding Scenarios

When adding new skills or modifying existing behavior, you should add or update corresponding scenarios in the dataset files:

1. Open `evals/gemini_dataset.json` (and/or `evals/claude_dataset.json`).
2. Add a new scenario block with a unique `id`, a clear `starting_prompt`, a detailed `conversation_plan`, and the `expected_trajectory` of tool calls.
3. Apply the `ci:run-evals` label while creating your pull request to trigger the evaluation pipeline.
4. The evaluation pipeline runs securely via Cloud Build. A maintainer will review the internal logs and results to verify your scenarios pass successfully.

### Other GitHub Checks

* **License Header Check:** A workflow ensures all necessary files contain the
Expand Down
Loading