Skip to content

Emit aggregate analytics for scope.models.embed / generate#779

Open
kriszyp wants to merge 1 commit into
mainfrom
kris/model-analytics-emission
Open

Emit aggregate analytics for scope.models.embed / generate#779
kriszyp wants to merge 1 commit into
mainfrom
kris/model-analytics-emission

Conversation

@kriszyp
Copy link
Copy Markdown
Member

@kriszyp kriszyp commented May 25, 2026

Let's track and charge for this stuff!

Summary

  • Successful embed/generate/generateStream calls now flow through recordAction() into hdb_raw_analyticshdb_analytics, mirroring the existing db-read / db-message pattern in Table.ts
  • Per-call rows in hdb_model_calls are unchanged — they remain the forensics trail; aggregate metrics fire on success only
  • Metric emitter is constructor-injected so tests can assert on exact (value, metric, path) tuples without touching the global pipeline; production wires up the module-scope recordAction

Why

Phase 1 of the model-usage license-enforcement chain. Phase 2 will extend the central-manager Plan.planLimits schema + signed license blob with the new fields, Phase 3 adds fabric-ai-level-1..3 SKUs with embed/generate budgets. Splitting the work this way lets the Harper-side emission land independently and start populating analytics tables before the CM-side enforcement is wired up.

Metrics emitted (path = backend.name)

Metric Value
model-<method> 1 per successful call
model-<method>-tokens sum of embeddingTokens + promptTokens + completionTokens, only when > 0

<method> is embed / generate / generateStream. Multi-modal embeds (e.g. CLIP image patches) report a synthetic embeddingTokens from the backend, so the image-vs-text multiplier is baked into TokenUsage rather than applied here.

Where to look

  • resources/models/Models.ts#record() now also calls the injected emitter after the per-call writer row
  • unitTests/resources/models/Models.test.js — 8 new tests in aggregate analytics emission describe block

Reviewed by

  • Codex CLI (uncommitted diff) — "The changes appear consistent with the existing analytics patterns... I did not find any discrete correctness issues in the modified files."
  • Gemini-3.1-pro (git diff main) — DI justified, sync+lazy recordAction safe in hot path, no security concerns, confirmed no existing consumer aggregates by model- prefix so no regression risk. Only finding was an unrelated package-lock.json drift from npm install (already reverted before commit).

Test plan

  • npx mocha unitTests/resources/models/Models.test.js — 28/28 passing (20 prior + 8 new)
  • npm run test:unit:resources — 544/544
  • npm run test:unit:main — 2130/2130
  • npm run test:integration:all — 313/313
  • npx prettier --write + npx oxlint --quiet — clean
  • Phase 2 follow-up PR in central-manager will close the license-enforcement loop

Generated by Claude (Opus 4.7) as part of the embed-analytics integration work.

Successful embed/generate/generateStream calls now also flow through
recordAction() into hdb_raw_analytics → hdb_analytics, mirroring the
db-read / db-message pattern in Table.ts. Phase 1 of the model-usage
license-enforcement chain — Phase 2 extends the CM Plan schema + signed
license blob with the new fields, Phase 3 adds fabric-ai-level-1..3 SKUs.

Per-call rows in hdb_model_calls remain for forensics; aggregate metrics
fire on success only.

Metrics emitted (path = backend.name):
- model-<method>            count = 1
- model-<method>-tokens     sum of embeddingTokens + promptTokens +
                            completionTokens, only when > 0

Multi-modal embeds (e.g. CLIP image patches) report a synthetic
embeddingTokens count from the backend, so the image-vs-text multiplier
is baked into TokenUsage rather than applied here.

The metric emitter is constructor-injected so unit tests can assert on
the exact (value, metric, path) tuples without touching the global
analytics pipeline. Production wires up the module-scope recordAction.

Reviewed by:
- Codex CLI: no discrete correctness issues
- Gemini-3.1-pro: DI justified, perf safe (sync + lazy), no security
  concerns, no regressions (no existing consumer aggregates by 'model-'
  prefix)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kriszyp kriszyp requested review from cb1kenobi and heskew May 25, 2026 19:19
@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 25, 2026

Reviewed; no blockers found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant