fix(export): mix all source audio tracks instead of only the first

[josiahcoad]

Problem

Recordings made with both system audio and microphone enabled export with no audible audio. The native capture helpers write system audio and the microphone as two separate audio tracks in the recording; the export pipeline reads only the demuxer's default (first) track — the system-audio track — which is pure silence when nothing was playing on the machine. The user's voice (track 2) is dropped entirely.

Reproduced on macOS 1.6.0: source recording has a:0 system audio at −91 dB (silence) and a:1 mic at −15.9 dB peak; the exported MP4's audio is −91 dB throughout.

Fix

• Trim-only path (the default for projects without speed edits): enumerate all audio streams via getAVStreams(), decode each stream (readAVPacket per stream index), and sum them into one mixed timeline, hard-clamped to [-1, 1] and re-chunked at the same granularity the single-track path uses for trim skipping. Single-track sources take the existing path unchanged.
• Pitch-preserved (media element) path: enable every audio track on the render element, mirroring what VideoPlayback already does for the editor preview.

Known gaps, intentionally out of scope and noted in the code: the source-copy fast path returns the source verbatim (all tracks preserved, but players will default to the silent first track), and the offline WSOLA path (segments >16× speed) still reads the default track only.

Testing

• New browser-mode regression test (multiTrackAudio.browser.test.ts) with a two-track fixture — silent first track + 440 Hz tone second track, the exact shape the native helpers produce. It exports through the real WebCodecs pipeline and asserts the decoded result is audible. It fails with peak=0 on the previous code and passes with this change.
• tsc --noEmit, biome check, full unit suite (352 tests) and all browser-mode tests (7) pass.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Improved audio export for media containing multiple audio tracks by combining tracks into a single output.
  • Preserved audio timing and quality when trimming, resampling, or adjusting channel layouts.
  • Ensured all available audio tracks are included during real-time recording exports.

• Tests

  • Added browser coverage to verify multi-track audio exports complete successfully and contain audible sound.

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes

The audio exporter now preserves multiple audio tracks in trim-only and speed-preserved export paths. Trim-only exports decode and mix audio streams with channel conversion, resampling, clamping, and trim-aware re-chunking. Browser coverage validates a two-track fixture.

Multi-track audio export

Layer / File(s)	Summary
Enable media audio tracks src/lib/exporter/audioEncoder.ts	The capture path enables all available media audio tracks before recording.
Decode and mix audio streams src/lib/exporter/audioEncoder.ts	The trim-only path selects single-track decoding or multi-track mixing, then passes the resulting sample rate and channel count to encoding.
Validate multi-track export src/lib/exporter/multiTrackAudio.browser.test.ts	A browser test exports a two-track fixture, checks that decoded output is non-silent, and closes the audio context.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant VideoExporter
  participant WebDemuxer
  participant AudioDecoder
  participant AudioEncoder
  VideoExporter->>WebDemuxer: enumerate audio streams
  WebDemuxer->>AudioDecoder: decode each stream
  AudioDecoder->>AudioEncoder: provide decoded audio data
  AudioEncoder->>AudioEncoder: mix, resample, clamp, and trim slices
  AudioEncoder-->>VideoExporter: encode processed audio

Loading

Suggested reviewers: etiennelescot

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is detailed but missing several required template sections, including Summary, Related issue, Type of change, Release impact, Desktop impact, and Screenshots/video.	Add the missing template sections and complete the required issue reference, checklists, platform impact, and any screenshots or video.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: exporting all source audio tracks instead of only the first.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/exporter/audioEncoder.ts (1)

641-674: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

A mid-stream decode error aborts the whole export and leaks pending frames.

The stated intent is to “skip undecodable audio streams,” but only config-unsupported streams are skipped (Line 566). If a stream passes isConfigSupported yet a packet fails at runtime, the error callback logs while WebCodecs closes the decoder; the next decoder.decode(...) on Line 645 then throws InvalidStateError, which propagates out of decodeAndMixAudioStreams and fails the entire audio export — even for tracks that decoded cleanly. Any AudioData still sitting in pending at that point is also never close()d, leaking native resources.

Consider isolating per-stream decode failures so one bad stream is dropped rather than aborting the mix.

🛡️ Sketch: contain per-stream failures

 			const reader = packetStream.getReader();
 			try {
 				while (!this.cancelled) {
 					const { done, value: packet } = await reader.read();
 					if (done || !packet) break;
 					decoder.decode(
 						new EncodedAudioChunk({ /* ... */ }),
 					);
 					while (decoder.decodeQueueSize > DECODE_BACKPRESSURE_LIMIT && !this.cancelled) {
 						await new Promise((resolve) => setTimeout(resolve, 1));
 					}
 					drainPending();
 				}
+			} catch (e) {
+				console.warn("[AudioProcessor] Skipping stream after decode error:", e);
 			} finally {
 				try {
 					await reader.cancel();
 				} catch {
 					/* reader already closed */
 				}
+				for (const d of pending) d.close();
+				pending.length = 0;
 			}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/exporter/audioEncoder.ts` around lines 641 - 674, Update the
per-stream decode flow in decodeAndMixAudioStreams around the reader loop and
decoder.flush so runtime decode or flush failures are caught, logged, and cause
only the current audio stream to be dropped rather than propagating to the
overall export. Ensure all AudioData entries remaining in pending are closed
when a stream fails, while preserving normal draining and cleanup for successful
or cancelled streams.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/exporter/audioEncoder.ts`:
- Around line 641-674: Update the per-stream decode flow in
decodeAndMixAudioStreams around the reader loop and decoder.flush so runtime
decode or flush failures are caught, logged, and cause only the current audio
stream to be dropped rather than propagating to the overall export. Ensure all
AudioData entries remaining in pending are closed when a stream fails, while
preserving normal draining and cleanup for successful or cancelled streams.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 900f75fd-369c-4f59-a00c-9e48fa77febd

📥 Commits

Reviewing files that changed from the base of the PR and between c2ba750 and 8894d4c.

⛔ Files ignored due to path filters (1)

• tests/fixtures/two-audio-tracks.mp4 is excluded by !**/*.mp4

📒 Files selected for processing (2)

• src/lib/exporter/audioEncoder.ts
• src/lib/exporter/multiTrackAudio.browser.test.ts