feat(project-engine-client): foundation — vendor spec + generation pipeline (LLMO-5461)

[aliciadriani]

What

Foundation slice for the new @adobe/spacecat-shared-project-engine-client package — the typed integration with the Semrush Project Engine API (/enterprise/projects/api).

Part of LLMO-5459 (epic) / LLMO-5461 (client + generated types). This PR is foundation only: vendor the spec and wire the conversion-and-type-generation pipeline. The client wrapper, IMS handler move, and stateful mock land in follow-up PRs.

Scope of this PR

• Vendored spec — spec/projectengine_swagger_public.yaml (Swagger 2.0, basePath: /enterprise/projects/api). Manual-refresh only; Semrush provides the file, no endpoint access. Not patched/edited.
• Generation pipeline (npm scripts):
  • spec:convert — swagger2openapi --patch v2→3.x → build/openapi3.json (intermediate, gitignored). Exists only to feed the type generators.
  • generate:ts — openapi-typescript → src/generated/types.ts
  • generate:pydantic — datamodel-code-generator → python/serenity_project_engine/ (Pydantic v2)
  • mock — Counterfact reads the raw v2 spec directly (:4010), never the converted artifact.
• Conventions — follows spacecat-shared: JS + ESM, JSDoc-typed source, mocha + chai + c8, @adobe/eslint-config-helix. The scaffold's TS surface is ported to JS+JSDoc; types.ts is a type artifact only.
• Smoke test — test/foundation.test.js asserts the vendored spec invariants (Swagger 2.0, basePath, Auth-Data-Jwt header) and that generated types.ts carries the expected paths interface + known v1/v2 routes.

Commit structure

1. feat(...) — vendored spec + generation pipeline + config + smoke test
2. chore(...) [skip ci] — generated TS + Pydantic types, committed separately and marked linguist-generated (collapsed in review, excluded from diff counts)

Validation

• npm run generate runs end-to-end (build/openapi3.json → types.ts → python package)
• npm test — 6 smoke tests pass
• npm run lint — clean

🤖 Generated with Claude Code

[github-actions]

This PR will trigger a minor release when merged.

[aliciadriani]

Must fix — first merge would fail the release job.

This monorepo's release job runs npx -ws semantic-release on every merge to main and publishes each package via OIDC Trusted Publishing (no npm token, NPM_CONFIG_PROVENANCE=true). Per scripts/setup-npm-trusted-publishers.sh, a new package cannot use OIDC for its first publish — it needs a manual npm publish as adobe-bot plus a trust binding, and this package isn't in that script's PACKAGES array. As written, the first merge would attempt an OIDC publish of @adobe/spacecat-shared-project-engine-client@1.0.0 and turn the release job red.

[aliciadriani]

Fixed in e440235 — marked the package "private": true, so @semantic-release/npm auto-skips publishing (the release job still versions/tags it, just no npm push), and the SR_NO_NPM_AUTH guard in .releaserc.cjs stays satisfied.

When the client lands and we actually want this on npm, flip private off and do the deliberate first publish: npm publish --access public as adobe-bot, add the name to the PACKAGES array, then re-run setup-npm-trusted-publishers.sh.

[aliciadriani]

Should fix — no entry point. Sibling packages (e.g. spacecat-shared-utils) declare main + an exports map; this one declared neither, so even the generated types weren't reachable through the package name.

[aliciadriani]

Fixed in e440235 — added a types entry pointing at the generated src/generated/types.ts, giving the package a defined type surface.

Deliberately held off on a full main/exports map: there's no runtime yet, the client exports arrives with the wrapper PR (so this avoids fighting that stacked branch), and an exports map here would prematurely restrict deep imports.

[aliciadriani]

Nit — clean script. rm -rf package-lock.json node_modules references a per-package lockfile that doesn't exist in this npm-workspaces monorepo (the lockfile lives at the repo root), and it skips the dirs this package actually generates.

[aliciadriani]

Fixed in e440235 — clean is now rm -rf node_modules build .counterfact: drops the phantom lockfile and clears the real generated artifacts (the build/ conversion intermediate and the Counterfact scratch dir).

[aliciadriani]

Should fix — stale comment. This said "only models.py is committed," but the committed file is model.py (singular) and four modules are actually committed: model.py, aiseo.py, http_server.py, __init__.py.

[aliciadriani]

Fixed in e440235 — the comment now states that the generated Python package (model.py, aiseo.py, http_server.py, __init__.py) is committed and only the bytecode caches are ignored.

[aliciadriani]

Nit — coverage gate is currently vacuous. include: ["src/**/*.js"] matches no files in this foundation PR (only src/generated/types.ts exists, and it's excluded), so with all: true + check-coverage: 100 c8 reports 0/0 and the threshold passes trivially.

[aliciadriani]

Intentional — left as-is. This is forward-looking config: the gate starts enforcing 100% the moment the client's src/**/*.js lands in the wrapper PR. Confirmed npm test still exits 0 today (6 smoke tests pass), so no change needed here.

[MysticatBot]

Hey @aliciadriani,

Verdict: Comment - clean foundation scaffold with only minor consistency notes.
Changes: new @adobe/spacecat-shared-project-engine-client package vendoring the Semrush Project Engine Swagger 2.0 spec and wiring a type-generation pipeline (swagger2openapi, openapi-typescript, datamodel-code-generator) with smoke tests (18 files).

Non-blocking (4): minor issues and suggestions

• nit: .nycrc.json sets "branches": 100 while the monorepo convention (documented in root CLAUDE.md) is 97% branches. Stricter is fine but creates inconsistency with sibling packages - consider aligning to 97 or documenting why this package opts for 100. - packages/spacecat-shared-project-engine-client/.nycrc.json:8
• nit: "publishConfig": { "access": "public" } is inert while "private": true and could mislead a future contributor who flips private off without the OIDC trust binding. Consider removing it until the deliberate first-publish commit. - packages/spacecat-shared-project-engine-client/package.json:41
• nit: root CLAUDE.md says "containing 22 packages" which becomes inaccurate (23) once this merges. Consider updating or switching to "20+ packages". - CLAUDE.md (Architecture Overview)
• suggestion: the spec assertion expect(spec).to.include('swagger: "2.0"') is brittle to YAML quote-style variation (swagger: '2.0' or unquoted). Since the spec is vendored and manually refreshed this is low-risk, but parsing YAML and asserting on the value would be more robust. - packages/spacecat-shared-project-engine-client/test/foundation.test.js:18

Suggestions for follow-up

• The committed Python models (Pydantic v2) have no CI validation gate - unlike the TS types which the smoke test exercises. When the spec is refreshed, someone could regenerate TS but forget Pydantic, and nothing would signal the drift. Consider adding a minimal Python smoke test (e.g. python3 -c "from serenity_project_engine.model import ...") or a generate:check script that diffs committed output against a fresh run.
• Consider adding a SHA-256 checksum sidecar for the vendored spec (spec/projectengine_swagger_public.yaml.sha256) so future refreshes can be verified against what Semrush provides - supply-chain hygiene for when the spec becomes load-bearing.

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 1m 39s | Cost: $3.52 | Commit: e4402353669fdaf678230cef6d51cdc08b6e0530
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[aliciadriani]

Publishing note: private: true is intentional here

This package (@adobe/spacecat-shared-project-engine-client) is intentionally marked
private: true in this PR and in #1661, so both can merge without attempting a publish.

Going public requires a one-time prerequisite: the npm trusted-publisher binding for this
new package name must be configured before the release job can publish it via OIDC. Without it,
a first publish fails the release job — so keeping the package private decouples these merges
from that setup step.

Go-public sequence, when ready: configure the trusted-publisher binding for
@adobe/spacecat-shared-project-engine-client (release-owner / npm-org action), then drop
private so it publishes on merge.

This is the gate on LLMO-5461 #4 (adopting the client in spacecat-api-service and llmo): the
package has to publish before those can consume it. Flagging here so whoever owns spacecat-shared
releases can sequence the binding when appropriate.

[rainer-friederich]

Review — foundation slice (reviewed tip e440235)

Verdict: solid foundation slice, approve-with-changes. CI is green and the structure follows repo conventions closely, but there is one genuine bug the MysticatBot review missed (the generate pipeline fails on a fresh clone — verified empirically), plus a few convention gaps.

Scope of review: every hand-written file in full (package.json, scripts, configs, README, test, the committed Pydantic output). The 6,547-line vendored spec, the 7,907-line generated types.ts, and the package-lock were assessed via the pipeline config, smoke tests, and the Python output — not line-by-line.

Should fix before merge

1. npm run spec:convert fails on a fresh clone. swagger2openapi --outfile build/openapi3.json does not create the output directory, and build/ is gitignored — so the next person following the README's refresh flow ("drop in the newer file, re-run npm run generate") hits ENOENT. Reproduced with the exact pinned version (7.0.8) in a clean directory. Fix: "spec:convert": "mkdir -p build && swagger2openapi ...". It validated end-to-end locally only because build/ already existed.

2. Missing LICENSE.txt, CODE_OF_CONDUCT.md, CONTRIBUTING.md. Every sibling package carries these, and the package declares Apache-2.0 with intent to publish to npm — publishing without a LICENSE file in the tarball is a real gap (.npmignore couldn't even exclude it; it just isn't there).

3. devDependencies omit the tools its own scripts invoke. test uses c8 / mocha / mocha-multi-reporters and none are declared — it only works via hoisting from the monorepo root. Siblings (e.g. tier-client) declare them per-package. Declare c8, mocha, mocha-multi-reporters to match convention and survive root devDep changes.

4. Copyright header says 2025 in test/foundation.test.js — new file created June 2026.

Worth a decision (not blocking)

5. .nycrc.json branches: 100 vs the repo's 97 convention (MysticatBot finding, confirmed — spacecat-shared-utils uses 97). Also note the whole coverage gate is currently vacuous: include: src/**/*.js matches zero files (only types.ts exists, and it's excluded). Fine until the client PR — just be aware the 100/100 gate isn't being exercised yet.

6. No main/exports in package.json — types points at the generated .ts and nothing else. Any runtime import of the package throws. Acceptable for a types-only foundation slice, but the client PR must add it; anything consuming the package in the interim fails confusingly.

7. Python distribution story is undefined. python/serenity_project_engine/ is committed but npmignored, has no pyproject.toml, no Python lint/import check in CI, and spacecat-shared only releases via npm. Worth deciding how Serenity/DRS actually consumes it (copy? git ref? separate publish?) before it becomes load-bearing. Related hazard: generated names like TypeModel / TypeModel1 are order-dependent generator artifacts — a spec refresh can silently renumber them and break any Python consumer pinning those class names.

8. generate partial-failure drift: datamodel-codegen is a Python PATH tool most JS contributors won't have; npm run generate then dies after regenerating types.ts, leaving the TS and Python committed artifacts out of sync in the working tree. A generate:check CI gate (diff committed output vs fresh regen) would catch both this and MysticatBot finding 5 — good follow-up ticket.

On the MysticatBot triage

The triage verdicts look sound: branches-97 alignment (item 5 above), the CLAUDE.md "22 packages" → 23 fix belongs in this PR, declining the YAML-parse refactor of the smoke test (string tripwire is appropriate for a vendored file), and deferring the Pydantic CI gate + spec checksum to follow-ups. One point deserves explicit reviewer attention: up-stack (the mock PR, 4c510c1) private: true is already removed while publishConfig.access: public remains — the first publish of this package via the npm trusted-publisher binding needs explicit confirmation before that stack tip reaches main.

Nits

• .gitattributes marks build/** linguist-generated, but build/ is gitignored — dead entry.
• The vendored spec leaks the vendor's TODO: LJ: Validate into KeywordsTaggedRequest's description — correct consequence of the vendor-as-is policy, just visible.
• homepage URL style and .releaserc.cjs (including the SR_NO_NPM_AUTH guard) match siblings exactly — checked, not findings.

Items 1–4 are small enough to fold into the foundation branch in one commit, which also keeps the stack ripple to a single rebase point.

🤖 Generated with Claude Code

[aliciadriani]

Correction — supersedes my earlier triage comment above. That triage was written against tip e4402353 and said "no ref-mutating changes were made," recommended folding #3 in by hand, and recommended declining #4. All three are now wrong: commit 95f9830 ("address PR #1660 review nits") has since landed and been pushed — it is the current foundation tip (e4402353 → 95f9830, a clean linear child). Reconciling each finding against what actually landed:

#3 — done. 95f9830 bumps the monorepo package count 22 → 23 in both the CLAUDE.md prose and the structure tree. Correct (this package is the 23rd workspace).

#4 — done, and keeping it is the right call (I withdraw the earlier "decline"). 95f9830 replaces the literal expect(spec).to.include('swagger: "2.0"') with expect(spec).to.match(/^swagger:\s*['"]?2\.0['"]?/m). On the merits this is a safe robustness tweak, not a loosening: an OpenAPI 3.x document has no swagger: key at all, so the regex still fails on a wrong-version spec exactly as the literal did — version detection is unchanged. It only tolerates cosmetic quote/whitespace reformatting (swagger: '2.0', unquoted, etc.) of a legitimate 2.0 declaration on a future re-vendor.

#1 (.nycrc branches: 100 vs the 97% convention) — author's call, kept at 100 by reply, and the reply's rationale holds. On the foundation tree the gate is currently vacuous: there is no non-generated src/**/*.js under coverage yet (only the excluded generated/), so nothing is measured here — independently confirmed in the human review above. The genuine 100% lands on the stacked branches (client.js, store.js, stateful.js, …), which are already at 100%, so the gate holds end-to-end. The only standing point is the convention divergence (siblings use 97); keeping 100 and dropping to 97 only if a genuinely hard-to-cover branch ever appears is fine. Not blocking.

#2 (publishConfig / private) — already settled by the author's publishing note above; correcting my earlier framing. My prior wording called the package "publishable up-stack" and asked to "confirm the first publish is deliberate," implying a deliberate go-public. That was the wrong read on two counts. (a) The missing private: true at the mock head (4c510c1) is stack staleness, not a flip: the mock branched off the wrapper at 8d9b908, before private: true was present there, and neither the current wrapper (044a54f) nor foundation (95f9830) — both of which carry private: true — is an ancestor of the mock. (b) The author's publishing note above already establishes that private: true is intentional in #1660/#1661 and that go-public is a deliberate, gated, post-binding step. So there is no open "did you mean to publish" question. The only residual is mechanical: when the mock rebases onto the current wrapper, confirm private: true lands in the merged package.json — the stale non-private mock tree must not win that merge.

#5 / #6 (Pydantic CI gate, spec checksum sidecar) — agreed, follow-up tickets. Not #1660 blockers.

Net: #3 and #4 are addressed and pushed; #1 is settled by reply (rationale confirmed); #2 is settled by the author's publishing note, with one mechanical private-survives-the-rebase check at convergence. No further code change needed here.

[Edited for honesty about what this comment originally got wrong. An earlier version mis-framed #2 as "publishable up-stack … likely intentional … the first real publish should be explicitly confirmed deliberate," implying a possible go-public. That was wrong: the missing private: true at the mock head is stack staleness (the mock branched at 8d9b908, before the flag existed on the wrapper), and the author's publishing note above had already settled that private is intentional and go-public is a gated step — so #2 was never an open publish question. The #1 line was also restated accurately (vacuous on foundation, genuine 100% on the stacked branches) rather than left neutral.]

[aliciadriani]

The fresh-clone spec:convert bug especially. Triage:

Fixed in fb36bfa (folded into one commit to keep the stack ripple to a single rebase point):

• spec:convert ENOENT on a fresh clone. build/ is gitignored and swagger2openapi --outfile doesn't create the dir. Fixed with mkdir -p build && …. Reproduced under the exact no-build/ condition: the fixed script exits 0 and writes build/openapi3.json.
• Missing LICENSE.txt / CODE_OF_CONDUCT.md / CONTRIBUTING.md. Added, copied from a sibling package.
• devDeps omit c8 / mocha / mocha-multi-reporters. Declared, versions pinned to match siblings. (chai was also missing from the lockfile node — now recorded.) Lockfile change is scoped to this package's node only.
• Copyright 2025 → 2026 in test/foundation.test.js.
• Dead .gitattributes entry. Dropped build/** linguist-generated (build/ is gitignored).

spec:convert/generate:ts, 6 unit tests, and lint all green.

Deferred to follow-ups (tracked):

• generate:check CI gate. Agreed this is the right catch-all — one gate (re-run generate, fail on a diff vs committed output) covers both the TS/Python partial-failure drift and the earlier Pydantic-no-gate finding. Filed as a follow-up; it pulls a Python runtime + datamodel-codegen into CI so it's out of this slice.
• Python distribution story + order-dependent TypeModel/TypeModel1 naming. Captured in the same follow-up — needs a decision on how Serenity/DRS consumes the Python before it's load-bearing.

Acknowledged, no change here:

• main/exports absent. Acceptable for the types-only foundation slice; the wrapper PR (feat(project-engine-client): typed client wrapper + IMS Bearer auth + retry + spec-correction overlay (LLMO-5461) #1661) adds them.
• .nycrc branches:100 vs 97 / vacuous gate. Confirmed — the gate matches zero files today; the covered code lands up-stack at 100%. Kept at 100, will drop to the 97 floor only if a genuinely hard-to-cover branch appears.
• Vendored-spec TODO leak — consequence of the vendor-as-is policy, visible by design.

On the cross-stack publish flag: confirmed and escalated separately — the mock stack tip is go-public-shaped (private removed) and the fix (re-add private: true, keep go-public a separate gated PR after the trusted-publisher binding) is being handled on that branch.

Pre-existing lockfile/manifest drift on this branch (athena/drs/utils behind the tree) is noted for a separate look so npm ci isn't installing stale versions — kept out of fb36bfa.

[MysticatBot]

Hey @aliciadriani,

Verdict: Comment - clean foundation slice, all prior review findings addressed.
Changes: new @adobe/spacecat-shared-project-engine-client package vendoring the Semrush Project Engine Swagger 2.0 spec and wiring a type-generation pipeline (swagger2openapi, openapi-typescript, datamodel-code-generator) with smoke tests (22 files).

Non-blocking (1): minor issues and suggestions

• nit: CLAUDE.md Package Catalog table explicitly enumerates all API client packages but does not include project-engine-client after this PR merges. The Monorepo Structure glob (spacecat-shared-*-client/) still covers it, so the table is merely incomplete rather than misleading. Consider adding it when the package goes public. - CLAUDE.md (Package Catalog section)

Previously flagged, now resolved

• CLAUDE.md package count updated 22 to 23
• Spec assertion now uses regex instead of brittle literal match
• spec:convert ENOENT on fresh clone fixed with mkdir -p build
• LICENSE.txt, CODE_OF_CONDUCT.md, CONTRIBUTING.md added
• devDependencies (c8, mocha, chai, mocha-multi-reporters) declared
• Copyright year corrected to 2026
• Dead .gitattributes entry for gitignored build/ removed

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 0m 50s | Cost: $4.92 | Commit: fb36bfa5028dd4c9d14f8041fba3fd629c876d22
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[rainer-friederich]

LGTM

[MysticatBot]

Hey @aliciadriani,

Verdict: Approve - no new concerns; all prior findings resolved.
Changes: new @adobe/spacecat-shared-project-engine-client package vendoring the Semrush Project Engine Swagger 2.0 spec and wiring a type-generation pipeline (swagger2openapi, openapi-typescript, datamodel-code-generator) with smoke tests (22 files).

Previously flagged, now resolved

• CLAUDE.md Package Catalog now includes project-engine-client
• CLAUDE.md package count updated 22 to 23
• Spec assertion uses regex instead of brittle literal match
• spec:convert ENOENT on fresh clone fixed with mkdir -p build
• LICENSE.txt, CODE_OF_CONDUCT.md, CONTRIBUTING.md added
• devDependencies (c8, mocha, chai, mocha-multi-reporters) declared
• Copyright year corrected to 2026
• Dead .gitattributes entry for gitignored build/ removed

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 0m 28s | Cost: $2.23 | Commit: b6a689fa4bdf5a108bec4cd60d07f9690cacb145
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[solaris007]

🎉 This PR is included in version @adobe/spacecat-shared-project-engine-client-v1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀