Skip to content

feat(engine): add HDR two-pass compositing — DOM layer + native HLG video#288

Open
vanceingalls wants to merge 1 commit intofix/hdr-output-pipelinefrom
feat/hdr-phase-1
Open

feat(engine): add HDR two-pass compositing — DOM layer + native HLG video#288
vanceingalls wants to merge 1 commit intofix/hdr-output-pipelinefrom
feat/hdr-phase-1

Conversation

@vanceingalls
Copy link
Copy Markdown
Collaborator

@vanceingalls vanceingalls commented Apr 16, 2026

Summary

Compositions with HDR video AND DOM overlays (text, graphics, SDR video) couldn't render both correctly — either HDR data was lost (Chrome captures sRGB only) or DOM overlays were missing (FFmpeg pass-through skips Chrome). This PR adds in-memory alpha compositing that combines both.

What it does

Per-frame two-pass capture:

  1. DOM pass — Chrome screenshots the page with a transparent background (CDP alpha). HDR videos are hidden, leaving transparent holes where they go.
  2. HDR pass — Pre-extracted native HLG/PQ frames (16-bit PNG from FFmpeg) are read from disk.
  3. Composite — DOM pixels (sRGB RGBA8) are alpha-composited over HDR pixels (rgb48le) in Node.js memory, with sRGB→HLG/PQ conversion via a 256-entry lookup table.

Key components:

  • decodePng() / decodePngToRgb48le() — Pure Node.js PNG decoders (no native dependencies). Support all 5 PNG filter types.
  • blitRgba8OverRgb48le() — Alpha composite with per-pixel sRGB→HDR LUT conversion. Fast paths for alpha=0 (skip) and alpha=255 (overwrite).
  • initTransparentBackground() + captureAlphaPng() — Split CDP transparent background setup (once) from per-frame screenshot capture (eliminates 2 CDP round-trips per frame).
  • Single-pass FFmpeg extraction — All HDR frames extracted in one sequential FFmpeg run (avoids duplicate frames from per-frame -ss fast seek).

Key design decisions

Decision Why
In-memory compositing (not FFmpeg overlay) Eliminates ~2400 process spawns + temp files per render. Pure pixel math is 10x faster.
16-bit PNG intermediate Raw -f rawvideo loses color metadata, causing moiré artifacts. PNG is self-describing.
sRGB→HLG LUT (256 entries) DOM content is sRGB. Without conversion, it appears orange-shifted in HLG stream.
Native HDR detection before extraction extractAllVideoFrames converts SDR→HDR. Pre-extraction probe identifies original HDR sources so only truly-HDR videos get native extraction.

Files changed

File What changed
packages/engine/src/utils/alphaBlit.ts NEW — PNG decode, sRGB→HDR LUT, alpha compositing (14 tests)
packages/engine/src/services/screenshotService.ts Transparent background CDP, captureAlphaPng()
packages/engine/src/services/videoFrameInjector.ts hideVideoElements() / showVideoElements()
packages/engine/src/services/streamingEncoder.ts Input color space tags for rgb48le
packages/producer/src/services/renderOrchestrator.ts Two-pass HDR capture loop, native HDR detection

How to test

Render a composition with an HDR video background and text overlays. Both should be visible — HDR video at full quality, text crisp with correct colors (not orange-shifted).

Stack position

3 of 6 — Stacked on #265 (HDR output pipeline). This is the foundation for all layered compositing that follows.

🤖 Generated with Claude Code

Copy link
Copy Markdown
Collaborator Author

vanceingalls commented Apr 16, 2026

Per-frame in-memory alpha compositing: Chrome screenshots DOM with
transparent background (PNG alpha), FFmpeg extracts native HLG/PQ
frames as 16-bit PNG. DOM pixels (sRGB RGBA8) composited over HDR
(rgb48le) via 256-entry sRGB-to-HLG/PQ LUT. Pure Node.js PNG decoder,
single-pass FFmpeg extraction, native HDR detection before SDR
conversion.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant