Land Highlights studio + party/action detection on main#47
Merged
Conversation
Auto-cut highlight clips from footage with no useful transcript (party videos, action) by peak-picking a fused laughter+energy curve instead of reading the transcript. Selected via 'podcli process --profile party|action'. - saliency.py: per-video robust-z normalization (median/MAD, never global), weighted channel fusion, numpy peak-pick with min-gap NMS, reaction dilation so a brief laugh isn't drowned by a louder neighbor, reaction-first candidate generation, 8s backward expansion to capture the setup before a laugh, and quiet-point boundary snapping - profiles drive it: 'saliency' candidate source (party/action) vs 'llm' (podcast) picks the selection path in the process pipeline - deterministic on clean input; produces render-compatible clip dicts Thresholds and channel weights are starting points to be tuned on real party footage.
Register a detect_highlights task handler so the MCP/web backend can run party/ action highlight detection via the python-executor, not just the CLI. Verified end to end through the stdin dispatch.
Add detect_highlights_pooled and accept video_paths in the detect_highlights task: run detection over many videos, tag each clip with its source_file, and rank globally reaction-first. Serves the 'best N moments across a folder of party clips' use case. Verified across two clips via the backend dispatch.
Saliency profiles select on audio/visual signals, not dialogue, so transcribing a long party video is wasted work. Detect the profile before Step 1 and skip transcription entirely (no words/segments needed), relax the no-segments gate, and stop the misleading 'heuristic mode' message when clips are already chosen. Verified end to end: 'process --profile party' skips transcription, detects highlights, and renders clips including a reaction clip.
Saliency detection now takes optional segments: clip windows are built from whole sentences (never cutting mid-thought) when a transcript is available, falling back to audio-lull snapping only for true no-dialogue footage. The process flow reuses a cached transcript for saliency profiles instead of skipping it, so podcast highlights are sentence-clean. Verified on a real cached 71-min episode: clips land on segment boundaries with coherent text.
…files - Snap saliency clip boundaries to real sentences (split words on .?! punctuation) so highlights never start or end mid-thought; falls back to segments then audio lulls when word punctuation is absent - Add --no-outro to process and stop auto-appending the outro for saliency (party/action) profiles — highlights are raw moments, not branded shorts
Detect once, then iterate on moments fast. A persisted ReelSession stores the detected moments; editing one (longer/shorter/earlier/later/shift/drop/toggle) re-cuts only that moment straight from the source (~seconds) and rebuilds the reel, with no re-detection or heavy render. - services/reel.py: shared core (session persistence, edit ops, fast ffmpeg re-cut + concat) used by every surface - CLI: 'podcli reel new|show|edit|build' - MCP: 'manage_reel' task handler (new/show/edit/build) - 10 tests covering edit math, drop/toggle, and session round-trip Next: TS MCP tool definition and the web-studio reel panel.
Add the manage_reel MCP tool (server.ts) routing through a new /api/reel web endpoint to the Python reel service, so the agent can create a reel and edit moments (longer/shorter/drop/...) conversationally. Mirrors the analyze_energy tool path; add manage_reel + detect_highlights to the TaskRequest union. Verified end to end: /api/reel show and edit against a live server.
A 'Reel' page in the studio: paste a video path, detect party/action moments, then adjust each moment (+5s/-5s end, start earlier/later, disable, drop) with one click — each edit calls /api/reel and rebuilds. Routed at /reel with a sidebar link. Builds clean (tsc + vite).
YAMNet scores subtle podcast chuckles far lower than belly-laughs: across a 71-min episode the max laughter was 0.15, so the 0.15 cutoff surfaced zero reactions. Thread a reaction_threshold param (default 0.06) used consistently for candidate-picking and reaction-window classification. On the same episode this turns 0 laugh-driven clips into 6 of 12, landing on the genuinely playful moments (a quip, rapid-fire banter, the warm sign-off).
- Render reels in vertical, horizontal or square (was hardcoded 1080p landscape) - Thread moment count and min/max duration through CLI, MCP and web API - Add reel session list and delete, plus an absolute "set" trim edit - Add a generalized "auto" detection profile (new default) so the content type no longer has to be chosen by hand - Expose per-clip and reel file paths; add a path-guarded /api/reel-download - Re-cut shifted moments after a drop so the reel stops reusing stale clips
- Rename Reel to Highlights (nav, route /highlights, /reel redirects) - Drop or browse a video instead of pasting a path - Trim each moment on a video timeline with draggable in/out handles - Download the whole reel or individual clips - List, open and delete saved sessions
- set edit op: absolute bounds, start/end-only, clamping - seed_session builds moments and normalizes the format - build_reel cuts at the session format, reuses clean clips, and re-cuts shifted moments after a drop; concat includes only enabled moments - auto profile resolves as a saliency profile and fuses channels
|
Warning Review limit reached
Next review available in: 33 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (15)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PRs #44 and #45 were merged into their stacked parent branches (phase1/phase2) instead of main, so the party/action detection and the entire Highlights studio never reached main. Main currently has only the laughter channel (#43) and AssemblyAI (#46).
This branch replays exactly those commits onto main (the two already-merged commits, laughter + ONNX windowing, are dropped by rebase). It restores:
detect_highlightsbackend task + folder poolingVerified: tsc clean, full build green, 410 tests pass (only the 3 pre-existing test_ai_fallback env failures).