Skip to content

chore(ci): Streamline CI setup to split bundle, layer, tarball generation#20396

Merged
mydea merged 49 commits intodevelopfrom
fn/ci-test-structure
Apr 24, 2026
Merged

chore(ci): Streamline CI setup to split bundle, layer, tarball generation#20396
mydea merged 49 commits intodevelopfrom
fn/ci-test-structure

Conversation

@mydea
Copy link
Copy Markdown
Member

@mydea mydea commented Apr 20, 2026

Summary

Restructures the CI Build & Test pipeline to split the monolithic build step into parallel jobs and eliminate redundant work, reducing the critical path for test execution.

Key changes

Split build into parallel jobs:

  • The main Build job now only runs build:transpile, build:types, and build:extension (via yarn build:ci)
  • Bundle builds (job_build_bundles) run in a separate parallel job after Build — only when browser integration tests are affected
  • Lambda layer build (job_build_layer) runs in its own job — only when @sentry/aws-serverless or related E2E tests are affected
  • Tarball packaging (job_build_tarballs) is a dedicated job that replaces the old job_e2e_prepare step

Smarter build artifact handling:

  • Replaced the hand-maintained CACHED_BUILD_PATHS list with a dynamic Nx-derived artifact path computation (scripts/ci-print-build-artifact-paths.mjs) that reads the Nx project graph to determine
    exactly which output directories to upload
  • Removed the tarball cache (BUILD_CACHE_TARBALL_KEY) — tarballs are now built once and passed as artifacts, not cached across runs
  • Each build stage uploads its own artifact (build-output, build-bundle-output, build-layer-output), so downstream jobs only download what they need

E2E test matrix generation moved into Build:

  • The E2E test matrix (ci:build-matrix) is now computed inside the Build job instead of a separate job_e2e_prepare job
  • This removes job_e2e_prepare as a separate step and shaves off the sequential dependency

NX cache improvements:

  • Moved NX cache directory from .nxcache to .nx/cache (includes workspace-data for better cache hits)
  • Changed from actions/cache (save+restore) to actions/cache/save in Build + actions/cache/restore in downstream jobs
  • Cache is now scoped per-PR (${{ github.head_ref }}-${{ github.run_id }}) for more predictable behavior

Other cleanups:

  • getTestMatrix.tsgetTestMatrix.mjs — removed the ts-node, glob, and yaml dependencies by rewriting as plain ESM with node:fs and node:util
  • Fixed 1aws-serverless1 to include the lambda extension in the transpile step, to ensure it is properly cached and restored, and as we depend on this for the tarball as well.
  • Fixed angular dependency resolution for CI builds
  • Fixes nestjs types output for proper caching

Before → After pipeline structure

BEFORE:
  Metadata → Build (transpile+types+bundles+layer+tarballs) → All tests

AFTER:
  Metadata → Build (transpile+types only)
               ├── Build bundles ──→ Browser Playwright tests
               ├── Build layer ───→ (only when needed)
               ├── Build tarballs → E2E tests
               └── Unit/Integration/Remix tests (no extra build needed)

Somehow, cache restoration with nx seems hit or miss, it works sometimes but not other times. I opened an issue here: nrwl/nx#35403 - fixing this would improve this by a couple minutes, so would be nice to get there...

mydea and others added 13 commits April 16, 2026 16:23
…install

Replace `file:` directory references with `npm pack` tarballs when
installing @sentry/* packages into the Lambda layer's node_modules.

Previously, `yarn install` used `file:` references to workspace packages
with a fresh cache folder (to avoid stale cache issues), taking ~52s.
With tarballs, yarn can use its global cache and the install completes
in ~8s. Only transitive dependencies of @sentry/aws-serverless are
packed (~5 packages), keeping the packing step fast.

Total layer build time: ~97s → ~15s.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instead, only build this before uploading artifacts, as we do not really need this before.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 20, 2026

size-limit report 📦

Path Size % Change Change
@sentry/browser 25.88 kB - -
@sentry/browser - with treeshaking flags 24.35 kB - -
@sentry/browser (incl. Tracing) 43.81 kB - -
@sentry/browser (incl. Tracing + Span Streaming) 45.5 kB - -
@sentry/browser (incl. Tracing, Profiling) 48.73 kB - -
@sentry/browser (incl. Tracing, Replay) 82.98 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 72.5 kB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 87.67 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 99.93 kB - -
@sentry/browser (incl. Feedback) 42.7 kB - -
@sentry/browser (incl. sendFeedback) 30.55 kB - -
@sentry/browser (incl. FeedbackAsync) 35.55 kB - -
@sentry/browser (incl. Metrics) 27.16 kB - -
@sentry/browser (incl. Logs) 27.29 kB - -
@sentry/browser (incl. Metrics & Logs) 27.98 kB - -
@sentry/react 27.62 kB - -
@sentry/react (incl. Tracing) 46.05 kB - -
@sentry/vue 30.71 kB - -
@sentry/vue (incl. Tracing) 45.62 kB - -
@sentry/svelte 25.89 kB - -
CDN Bundle 28.57 kB - -
CDN Bundle (incl. Tracing) 46.08 kB - -
CDN Bundle (incl. Logs, Metrics) 29.95 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 47.12 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 68.92 kB - -
CDN Bundle (incl. Tracing, Replay) 83.14 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 84.17 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 88.61 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 89.69 kB - -
CDN Bundle - uncompressed 83.59 kB - -
CDN Bundle (incl. Tracing) - uncompressed 137.62 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 87.73 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 141.03 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 211.31 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 255.06 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 258.46 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 267.97 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 271.36 kB - -
@sentry/nextjs (client) 48.58 kB - -
@sentry/sveltekit (client) 44.22 kB - -
@sentry/node-core 58.35 kB +0.02% +8 B 🔺
@sentry/node 175.66 kB +0.01% +13 B 🔺
@sentry/node - without tracing 98.3 kB +0.02% +11 B 🔺
@sentry/aws-serverless 115.33 kB +0.01% +9 B 🔺

View base workflow run

@mydea mydea force-pushed the fn/ci-test-structure branch 3 times, most recently from 4481233 to 8c330ce Compare April 22, 2026 07:29
@mydea mydea self-assigned this Apr 23, 2026
"outputs": [
"{projectRoot}/build/types",
"{projectRoot}/build/types-ts3.8",
"{projectRoot}/*.d.ts"
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was not properly cached before, but we generate types into the root directory here.

@mydea mydea marked this pull request as ready for review April 23, 2026 08:51
@mydea mydea requested review from JPeer264, logaretm and s1gr1d April 23, 2026 08:51
Comment thread packages/aws-serverless/package.json
Comment thread packages/aws-serverless/package.json
Comment thread .github/workflows/build.yml
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c36e943. Configure here.

Comment thread .github/workflows/build.yml
Comment thread scripts/ci-print-build-artifact-paths.mjs
@mydea mydea force-pushed the fn/ci-test-structure branch from 1a9ca06 to 4f8cd17 Compare April 23, 2026 11:22
@mydea mydea merged commit 3bb1722 into develop Apr 24, 2026
250 checks passed
@mydea mydea deleted the fn/ci-test-structure branch April 24, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants