Emit aggregate analytics for scope.models.embed / generate#779
Open
kriszyp wants to merge 1 commit into
Open
Conversation
Successful embed/generate/generateStream calls now also flow through
recordAction() into hdb_raw_analytics → hdb_analytics, mirroring the
db-read / db-message pattern in Table.ts. Phase 1 of the model-usage
license-enforcement chain — Phase 2 extends the CM Plan schema + signed
license blob with the new fields, Phase 3 adds fabric-ai-level-1..3 SKUs.
Per-call rows in hdb_model_calls remain for forensics; aggregate metrics
fire on success only.
Metrics emitted (path = backend.name):
- model-<method> count = 1
- model-<method>-tokens sum of embeddingTokens + promptTokens +
completionTokens, only when > 0
Multi-modal embeds (e.g. CLIP image patches) report a synthetic
embeddingTokens count from the backend, so the image-vs-text multiplier
is baked into TokenUsage rather than applied here.
The metric emitter is constructor-injected so unit tests can assert on
the exact (value, metric, path) tuples without touching the global
analytics pipeline. Production wires up the module-scope recordAction.
Reviewed by:
- Codex CLI: no discrete correctness issues
- Gemini-3.1-pro: DI justified, perf safe (sync + lazy), no security
concerns, no regressions (no existing consumer aggregates by 'model-'
prefix)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Reviewed; no blockers found. |
This was referenced May 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Let's track and charge for this stuff!
Summary
embed/generate/generateStreamcalls now flow throughrecordAction()intohdb_raw_analytics→hdb_analytics, mirroring the existingdb-read/db-messagepattern inTable.tshdb_model_callsare unchanged — they remain the forensics trail; aggregate metrics fire on success only(value, metric, path)tuples without touching the global pipeline; production wires up the module-scoperecordActionWhy
Phase 1 of the model-usage license-enforcement chain. Phase 2 will extend the central-manager
Plan.planLimitsschema + signed license blob with the new fields, Phase 3 addsfabric-ai-level-1..3SKUs with embed/generate budgets. Splitting the work this way lets the Harper-side emission land independently and start populating analytics tables before the CM-side enforcement is wired up.Metrics emitted (path =
backend.name)model-<method>1per successful callmodel-<method>-tokensembeddingTokens + promptTokens + completionTokens, only when > 0<method>isembed/generate/generateStream. Multi-modal embeds (e.g. CLIP image patches) report a syntheticembeddingTokensfrom the backend, so the image-vs-text multiplier is baked intoTokenUsagerather than applied here.Where to look
resources/models/Models.ts—#record()now also calls the injected emitter after the per-call writer rowunitTests/resources/models/Models.test.js— 8 new tests inaggregate analytics emissiondescribe blockReviewed by
recordActionsafe in hot path, no security concerns, confirmed no existing consumer aggregates bymodel-prefix so no regression risk. Only finding was an unrelatedpackage-lock.jsondrift from npm install (already reverted before commit).Test plan
npx mocha unitTests/resources/models/Models.test.js— 28/28 passing (20 prior + 8 new)npm run test:unit:resources— 544/544npm run test:unit:main— 2130/2130npm run test:integration:all— 313/313npx prettier --write+npx oxlint --quiet— cleanGenerated by Claude (Opus 4.7) as part of the embed-analytics integration work.