feat(project-engine-client): typed client wrapper + IMS Bearer auth + retry + spec-correction overlay (LLMO-5461)#1661
Conversation
…peline Create @adobe/spacecat-shared-project-engine-client and wire the Swagger 2.0 → type-generation pipeline for the Semrush Project Engine API. - Vendor projectengine_swagger_public.yaml (manual refresh; no live spec access) - spec:convert (swagger2openapi --patch) → generate:ts (openapi-typescript) → generate:pydantic (datamodel-codegen, pydantic v2); `generate` runs all three - mock script: Counterfact off the raw v2 spec (no conversion on the mock path) - JS + ESM + JSDoc, mocha/chai/c8, eslint-helix — matches spacecat-shared - linguist-generated markers + gitignore for build/ intermediate - smoke test asserts vendored spec + generated types presence datamodel-codegen emits a package dir (model.py/aiseo.py/http_server.py) rather than a single models.py: the spec's Go-style dotted schema names are modular references. Tool-flag layout choice; the spec is unmodified. The client wrapper, IMS handler move, and stateful mock land in follow-ups. Refs: LLMO-5461, LLMO-5459 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… ci] Generated by `npm run generate` from spec/projectengine_swagger_public.yaml: - src/generated/types.ts (openapi-typescript) - python/serenity_project_engine/ (datamodel-codegen, pydantic v2) Committed separately and marked linguist-generated (see .gitattributes) so the diff is collapsed in review. Do not hand-edit — re-run `npm run generate`. Refs: LLMO-5461 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…arding, idempotency-aware retry Port the scaffold client surface to spacecat-shared conventions (JS + ESM + JSDoc, mocha + chai + sinon + c8). Behaviour is unchanged from the design: - createSerenityProjectEngineApiClient: thin openapi-fetch client over the generated `paths`, owning base URL, retries, and the Auth-Data-Jwt header only. - Caller's raw IMS JWT is forwarded verbatim via the Auth-Data-Jwt header (static or per-request async getter). No token minting or exchange. - Retry: 429 retried for any method; 5xx + network errors retried only for idempotent methods (GET/HEAD/PUT/DELETE/OPTIONS). Defaults: 2 retries, exponential backoff base 200ms. - Hand-written index.d.ts re-exports the typed surface + generated paths/components. 100% test coverage; lint clean. LLMO-5461 (epic LLMO-5459) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pty token Addresses review findings on the retry/auth wrapper: - Retry replayed the same Request object openapi-fetch hands the injected fetch; fetch() consumes the body on first use, so a bodied request that hit 429/5xx either threw 'Request already used' (POST+429) or silently skipped the retry and returned the first failure (PUT/DELETE+5xx). Clone the Request per attempt and never touch the original so every retry sends a fresh body. - authMiddleware now fails fast when the token getter resolves to an empty value instead of sending the literal 'undefined' / an empty header. - Bump new-file copyright headers to 2026 to match the repo convention. Tests: add body-replay regression tests (PUT+5xx re-sends body, POST+429 retries on a bodied request) and an empty-token guard test. 27 passing, 100% coverage; lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This PR will trigger a minor release when merged. |
aliciadriani
left a comment
There was a problem hiding this comment.
Self-review follow-up: pushed 4c7c521 addressing all findings from the review (1 Must Fix, 1 Should Fix, 2 nits). Inline comments below map each fix to its finding. Verified end-to-end against a real http server before and after; suite now 27 passing, 100% coverage, lint clean.
| // Regression: openapi-fetch hands us a Request object, and reading its body (as real | ||
| // fetch does) marks it used — a bare replay would throw "Request ... already used". | ||
| // These two assert the per-attempt clone actually re-sends the body on retry. | ||
| it('re-sends a bodied idempotent Request on retry (clones rather than replaying)', async () => { |
There was a problem hiding this comment.
Should Fix #2 — resolved. The original retry tests passed a string URL + stubbed fetch, so they never exercised the Request-with-body path openapi-fetch actually produces — which is how 100% line coverage coexisted with a broken retry. These two tests drive createRetryingFetch with a real Request carrying a body and a stub that reads req.text() each call (mirroring real fetch's consumption). They fail against the pre-fix code and pass after the clone() fix.
| expect(fetch.callCount).to.equal(2); | ||
| }); | ||
|
|
||
| it('fails fast (and never sends the request) when the token resolves to empty', async () => { |
There was a problem hiding this comment.
Nit (auth guard) — test. Asserts the empty-token getter makes the call fail fast and that the underlying fetch is never invoked.
- mark package private so semantic-release skips the npm publish on merge; a new package cannot use OIDC for its first publish (see scripts/setup-npm-trusted-publishers.sh), so leaving it publishable would fail the main release job. Publish + trust binding will be set up deliberately when the client lands. - declare a `types` entry pointing at the generated types so the package has a defined type surface (full main/exports arrives with the client wrapper). - fix `clean` script: no per-package lockfile exists in this monorepo; clean the dirs this package actually generates (build/, .counterfact/). - correct stale .gitignore comment: the generated Python package (model.py, aiseo.py, http_server.py, __init__.py) is committed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-Data-Jwt header The generated `paths` mark `Auth-Data-Jwt` as a required per-operation header (faithful to the API contract), but the client injects that header at runtime via middleware (headers.set) and overwrites whatever a caller passes. So the typed surface was forcing every consumer to pass a value that's always ignored — a footgun (a "real" token passed there silently does nothing). Narrow the *exported* client type only: - Add a type-only mapped transform over the generated `paths` (ClientPaths) that removes `Auth-Data-Jwt` from each operation's header params. Auth-Data-Jwt is the sole header in the contract, so the emptied bag collapses to `never` — passing a header is now a compile error, not just optional, which actively closes the footgun. A genuine consumer header (if ever added) is preserved. - Type SerenityProjectEngineApiClient / the factory return against ClientPaths instead of raw Client<paths>. Generated types, the vendored spec, and the Pydantic models are untouched — they remain the honest API contract; only the client surface hides the header it supplies itself. Runtime (client.js, middleware) is unchanged. Validated with a throwaway tsc harness (not committed): a consumer calls an op without Auth-Data-Jwt (compiles), path params stay required, passing Auth-Data-Jwt is rejected, and the bogus-path drift canary still trips. Lint clean; unit suite 51 passing at 100% coverage (type-only change). Guard reconciliation (cross-PR): the type-surface guard fixture lives in #1665 (test/types/index.type-test.ts) and currently asserts the un-narrowed surface (it passes header: { 'Auth-Data-Jwt' }). That fixture must be updated when these land together at main — see the follow-up note on #1665. Relates to the #4 adoption, where consumer call sites drop the header argument. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Carry `private: true` from the foundation PR (#1660) onto the wrapper so neither PR's merge attempts a publish. A new package name cannot use the monorepo's OIDC trusted-publishing path for its first publish (see scripts/setup-npm-trusted-publishers.sh) — that binding is a privileged adobe-bot / @spacecat-admins step. Keeping both PRs private decouples review/merge from that step; "go public" lands as its own explicit PR once the trust binding exists. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The merged-to-main package.json is the wrapper's, so the foundation's clean-script fix (e440235) would be silently reverted at merge. Apply the same fix here: no per-package lockfile exists in this npm-workspaces monorepo; clean the dirs this package actually generates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…461-project-engine-client-wrapper
Two non-blocking findings from the MysticatBot review (verdict: Comment): - CLAUDE.md: bump the monorepo package count 22 -> 23 (this package is the 23rd workspace), in the prose and the structure tree. - foundation.test.js: assert the vendored Swagger version with a regex (`/^swagger:\s*['"]?2\.0['"]?/m`) instead of a literal substring, so a future spec refresh that reformats the quote style doesn't break the test on a harmless change. Verified the regex still matches the vendored spec. Lint clean; 6 foundation smoke tests pass. The two settled findings (nycrc branches:100, publishConfig under private:true) are addressed by reply, not code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Four "should-fix" items from the human review of PR #1660 plus one nit, folded into a single commit to keep the stack ripple to one rebase point: - spec:convert: prepend `mkdir -p build`. build/ is gitignored, and swagger2openapi --outfile does not create the output dir, so the README's refresh flow (`npm run generate`) hit ENOENT on a fresh clone. Verified: with no build/ present, the fixed script exits 0 and writes build/openapi3.json. It only passed before because build/ already existed. - Add LICENSE.txt, CODE_OF_CONDUCT.md, CONTRIBUTING.md (copied from a sibling package). The package declares Apache-2.0 and intends to publish; every sibling carries these, and a LICENSE belongs in the tarball. - Declare c8, mocha, mocha-multi-reporters in devDependencies. The test script and mocha config invoke them; they previously resolved only via root hoisting. Versions pinned to match siblings. (chai's lockfile node entry was also missing and is now recorded.) - foundation.test.js: copyright 2025 -> 2026 (file created June 2026). - .gitattributes: drop the dead `build/** linguist-generated` entry (build/ is gitignored, so the attribute never applies). Lockfile change is scoped to this package's node only; pre-existing sibling version drift left untouched. spec:convert/generate:ts, 6 unit tests, and lint all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses MysticatBot review nit: the Package Catalog table enumerated all API client packages but omitted project-engine-client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…foundation' into llmo-5461-project-engine-client-wrapper # Conflicts: # package-lock.json
The foundation merge brought in main's data-access 3.74.2 manifest, but the package-lock node still pinned 3.74.1, leaving npm ci inconsistent. Corrected by npm install (lockfile node now matches the manifest). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ine-client-wrapper # Conflicts: # package-lock.json # packages/spacecat-shared-project-engine-client/package.json
There was a problem hiding this comment.
Hey @aliciadriani,
Verdict: Comment - clean, well-layered client with solid test coverage; only minor polish items.
Changes: Adds typed Project Engine API client wrapper (openapi-fetch + Auth-Data-Jwt + retry with idempotency-awareness) with 18 tests at 100% coverage (9 files).
Non-blocking (4): minor issues and suggestions
- nit: Missing
baseUrlvalidation at the factory boundary - passingundefinedor''yields a cryptic error from inside openapi-fetch rather than a fail-fast guard like the auth token has -packages/spacecat-shared-project-engine-client/src/client.js:52 - suggestion: Add jitter to retry backoff (
delay * (0.5 + Math.random() * 0.5)) to prevent thundering-herd alignment when multiple Lambda instances hit the same 429/503 -packages/spacecat-shared-project-engine-client/src/internal.js:92 - suggestion: Respect
Retry-Afterheader on 429 responses - the current fixed exponential backoff may be shorter than the server's requested wait, leading to repeated 429s under aggressive rate limiting - nit: Missing test for a throwing token getter (e.g. IMS outage rejecting the promise) - distinct from the empty-string path and a scenario real callers will hit -
packages/spacecat-shared-project-engine-client/test/client.test.js
Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 0m 52s | Cost: $3.86 | Commit: b052eac652c59ab4e454ae3ee0b937396d9d5536
If this code review was useful, please react with 👍. Otherwise, react with 👎.
A README is durable package docs, not a PR description — "this PR vendors…"
reads wrong the moment it merges. Replace the scope blockquote with a roadmap
note ("Not here yet: …") describing what the package still lacks, not what a
particular PR did. The top bullets already state what the package provides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Hey @aliciadriani,
Verdict: Approve - prior blocking finding addressed cleanly; only minor polish items remain.
Changes: Fixes async onRetry rejection handling, refactors applyOverlay to return structured results instead of logging, updates types and tests (7 files).
Non-blocking (2): minor issues and suggestions
- nit:
@callback OnRetryJSDoc declares@returns {void}butindex.d.tsnow correctly types the return asvoid | Promise<void>- the two are out of sync for IDE tooling consumers of the JSDoc -packages/spacecat-shared-project-engine-client/src/internal.js:158 - suggestion: The async-rejection test proves "the request succeeds" rather than "no rejection escaped" - it relies on Node 18+ crashing the process on unhandled rejections rather than asserting explicitly. Consider a scoped
process.on('unhandledRejection', fail)listener for a positive assertion. Low priority since the test runner enforces this at the process level. -packages/spacecat-shared-project-engine-client/test/internal.test.js:210
Previously flagged, now resolved
- Async onRetry rejection now sunk via duck-typed
.catch()on any thenable return value - Overlay tests cleaned up: sinon stubs removed after applyOverlay refactored to pure function returning structured results
Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 0m 42s | Cost: $4.55 | Commit: 9023e1dd6b9f8a00b2e37313967d6a6b139f9f16
If this code review was useful, please react with 👍. Otherwise, react with 👎.
…eview Two non-blocking items from the approval: - OnRetry @callback JSDoc said @returns {void} while index.d.ts now types it void | Promise<void>; sync the JSDoc so IDE tooling matches. - The async-rejection test asserted "the request succeeds" but only relied on the runner crashing on an unhandled rejection. Add a scoped process.on('unhandledRejection') listener and assert nothing escaped, after a macrotask turn — a positive assertion that notifyRetry sinks the rejection. 70 passing, 100% coverage, lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Cleared the two non-blocking nits from the approval (
70 passing, 100% coverage, lint clean. Re-requesting review. |
There was a problem hiding this comment.
Hey @aliciadriani,
Verdict: Approve - prior blocking finding cleanly addressed; only minor polish items remain.
Changes: Addresses async onRetry rejection handling (the prior blocking finding) and refactors overlay script to return structured results instead of logging directly (7 files).
Non-blocking (2): minor issues and suggestions
- nit: Duck-typing
.catchis looser than the standard thenable check -typeof result?.then === 'function'is the ecosystem idiom and aligns better with thevoid | Promise<void>return type. Practical risk is negligible sinceonRetryis caller-controlled. -packages/spacecat-shared-project-engine-client/src/internal.js:141 - suggestion: The async onRetry test implicitly relies on mocha surfacing unhandled rejections as failures. An explicit scoped
process.on('unhandledRejection')listener would make the test intent more visible and guard against future runners that swallow them. Strictly optional. -packages/spacecat-shared-project-engine-client/test/internal.test.js:210
Previously flagged, now resolved
- Async onRetry rejection now sunk via duck-typed
.catch()with dedicated test applyOverlay()returns structured{ total, results }with no console output; CLImain()handles logging- JSDoc expanded to document the deliberate silent-failure design choice
Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 1m 24s | Cost: $4.60 | Commit: dc6c3269cc05b558e69c45236b543086591be148
If this code review was useful, please react with 👍. Otherwise, react with 👎.
There was a problem hiding this comment.
Hey @aliciadriani,
Verdict: Approve - prior nits addressed cleanly; no new issues.
Changes: Syncs OnRetry JSDoc return type with index.d.ts and adds positive unhandledRejection assertion to the async hook test (2 files).
Non-blocking (1): minor issues and suggestions
- nit: Test comment says "macrotask turn" but
setImmediateis technically a check-phase callback, not a macrotask - "event-loop turn" would be more precise -packages/spacecat-shared-project-engine-client/test/internal.test.js:217
Previously flagged, now resolved
- OnRetry
@callbackJSDoc@returnsnowvoid | Promise<void>, matchingindex.d.ts - Async-rejection test uses scoped
process.on('unhandledRejection')listener for positive assertion with proper cleanup
Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 5m 29s | Cost: $1.78 | Commit: 1e4411b53b6e64190554f59177d8024c5154441a
If this code review was useful, please react with 👍. Otherwise, react with 👎.
…ution Carry the user-manager-client review nit into this (shared) overlay applier so the two copies stay byte-identical bar the header: deepMerge now skips __proto__/constructor/prototype keys, with a guard test asserting Object.prototype stays clean. No behaviour change for the hand-authored overlays. 71 passing, 100% coverage, lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Hey @aliciadriani,
Verdict: Approve - clean, focused security hardening with correct test coverage.
Changes: Hardens deepMerge against prototype pollution by skipping __proto__, constructor, and prototype keys, with a regression test exercising the real attack vector via JSON.parse (2 files).
Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 1m 2s | Cost: $2.16 | Commit: 1c7411792bee73a96cac2582afd348ecbc18e7a7
If this code review was useful, please react with 👍. Otherwise, react with 👎.
## [@adobe/spacecat-shared-project-engine-client-v1.1.0](https://github.com/adobe/spacecat-shared/compare/@adobe/spacecat-shared-project-engine-client-v1.0.0...@adobe/spacecat-shared-project-engine-client-v1.1.0) (2026-06-23) ### Features * **project-engine-client:** typed client wrapper + IMS Bearer auth + retry + spec-correction overlay (LLMO-5461) ([#1661](#1661)) ([0465ab0](0465ab0))
|
🎉 This PR is included in version @adobe/spacecat-shared-project-engine-client-v1.1.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
What
Adds the typed Project Engine client wrapper to
@adobe/spacecat-shared-project-engine-client— a thinopenapi-fetchlayer over the generatedpathsthat owns the base URL, retries, and request authentication — together with a generation-time spec-correction overlay that aligns the vendored Semrush swagger with the live API.Part of LLMO-5459 (epic) / LLMO-5461. Step 2 of the work plan.
Behaviour
createSerenityProjectEngineApiClient(options)—openapi-fetchclient over the generatedpaths.Authorization: Bearer <token>header — matching the deployedspacecat-api-servicetransport (src/support/serenity/rest-transport.js). Semrush accepts the IMS bearer directly; the client mints or exchanges nothing. Accepts a static token or a (sync/async) per-request getter, so the header reflects the current caller, not a token captured at construction. Fails fast on an empty/missing token, and propagates a throwing getter (e.g. IMS outage), rather than sendingBearer undefined./enterprise/projects/apiprefix. It normalises the configured base URL to its origin (dropping any path/credentials) and appends the prefix, mirroringrest-transport.js. Idempotent for callers already passing the prefix. Invalid or non-http(s) base URLs fail fast at construction.429retried for any method;5xx+ network errors retried only for idempotent methods (GET/HEAD/PUT/DELETE/OPTIONS), so a POST that may have created a resource is never replayed. Bodied requests areclone()d per attempt so retries send a fresh, unconsumed body. Backoff is exponential with equal jitter (0.5x–1x) to de-correlate concurrent clients, and honours aRetry-Afterheader (delta-seconds or HTTP-date) when it asks for longer than the backoff — clamped to a 20s ceiling. Defaults: 2 retries, base 200ms. After exhausting retries, returns the last retryable response (e.g. final 503) or rethrows the last network error.Spec-correction overlay
The vendored Semrush swagger (
spec/projectengine_swagger_public.yaml) is kept pristine for re-vendoring. A small OpenAPI Overlay (spec/overlays/corrections.yaml), applied to the converted OAS3 artifact at generation time byscripts/apply-overlay.mjs, corrects three places where the published spec diverges from the live API:GET /v1/ai_models(the live global model catalog, absent from the upstream swagger).Auth-Data-Jwtheader from every operation and adds animsBearersecurity scheme. The live API authenticates onAuthorization: Bearer <IMS>;Auth-Data-Jwtis rejected.GET /v2/workspaces/{id}/projectsdescription: it also returns initial, never-yet-published draft projects, not only published ones.Guard tests pin CR1 and CR2, so a future Semrush spec refresh that silently drops the overlay fails loudly instead of regressing the generated types.
Auth-header note
An earlier revision forwarded the IMS token as the spec's native
Auth-Data-Jwtheader. That was verified to 401 against the live Semrush endpoint (adobe-hackathon.semrush.com), whileAuthorization: Bearer <IMS>returns 200. The client uses Bearer, and the CR2 overlay removesAuth-Data-Jwtfrom the generated TypeScript types and Pydantic models so the contract matches reality.Convention port
The scaffold's TypeScript surface (
internal.ts/client.ts/index.ts+ Vitest tests) is ported one-for-one to JS + ESM + JSDoc, mocha + chai + sinon + c8 — the design does not change, only the surface. A hand-writtensrc/index.d.tsre-exports the typed surface plus the generatedpaths/components.Validation
npm test— 46 passing, 100% statements / branches / functions / linesnpm run lint— clean🤖 Generated with Claude Code