fix: Route Sentry telemetry and simplify evals#617
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
dcramer
changed the title
dcramer
force-pushed
the
fix/evals-native-session-gh-615
branch
from
b6fccd4 to
8bd0ff3
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8bd0ff3. Configure here.
Route Sentry product telemetry and feature-usage requests through the Sentry skill while keeping Hex as the internal data primitive. Clarify the skill descriptions so explicit Sentry telemetry requests do not steer directly to Hex. Move Junior evals onto the normalized vitest-evals session surface for judge scoring and deterministic assertions. Simplify rubrics to pass/fail and document the package eval commands so agents do not guess at direct Vitest invocation. Keep ordered full-turn transcript behavior owned by the native vitest-evals Pi harness instead of adding repo-local event log layers. Refs GH-615 Co-Authored-By: GPT-5 Codex <noreply@openai.com>
dcramer
force-pushed
the
fix/evals-native-session-gh-615
branch
from
8bd0ff3 to
717581e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sentry product telemetry and feature-usage requests now steer through the Sentry skill instead of falling through to Hex as the visible skill. Hex remains the internal data primitive, but its skill description now explicitly excludes direct use for Sentry product telemetry when the Sentry skill is available.
The eval cleanup supports that fix by moving Junior evals onto native vitest-evals session/tool primitives instead of repo-local output buckets. The Slack eval harness returns a standard HarnessRun, rubric criteria are reduced to pass/fail, and deterministic checks use helpers such as toolCalls(result.session) and assistantMessages(result.session).
The docs and local agent instructions now point eval authors at the package eval scripts, direct file/-t arguments, and the native Pi harness boundary for ordered full-turn transcript behavior. This avoids adding a second event-log or sequencing layer in Junior while keeping Slack evals focused on Slack-visible behavior.
Refs GH-615