Skip to content

feat: sidecar WebVTT subtitle passthrough into HLS output#12

Open
svensson00 wants to merge 8 commits into
mainfrom
feat/webvtt-subtitle-passthrough
Open

feat: sidecar WebVTT subtitle passthrough into HLS output#12
svensson00 wants to merge 8 commits into
mainfrom
feat/webvtt-subtitle-passthrough

Conversation

@svensson00

@svensson00 svensson00 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Closes #10

What

Adds optional sidecar WebVTT subtitle passthrough into the HLS output. When SUBTITLE_URL is set, the encoder fetches a WebVTT source alongside the A/V input and publishes it as a segmented WebVTT rendition referenced from the HLS master playlist as an #EXT-X-MEDIA:TYPE=SUBTITLES group, available across the whole ABR ladder. Unset leaves today's video+audio only output unchanged, so this is fully backward compatible.

Issue #10 lists two possible ingest forms and states a concrete near-term need for the sidecar form. This PR implements the sidecar form as the contributed choice: a WebVTT source URL fetched alongside the A/V input. The design keeps MPEG-TS style muxing untouched, so in-band subtitle carriage can be added later on the same path without reworking it. It composes with the configurable-input proposal (SRT caller, merged as #11) and works in both RTMP-listener and SRT-caller mode.

Config

Variable Description Default
SUBTITLE_URL Sidecar WebVTT source URL (http/https). If not set, output stays video+audio only
SUBTITLE_LANGUAGE BCP-47 language tag, e.g. en und
SUBTITLE_NAME Display name, e.g. English value of SUBTITLE_LANGUAGE
SUBTITLE_DEFAULT Whether the rendition is the group default (true/1) false

The config is shaped internally as a track array so more tracks can be added later; a single track is read from the environment today.

Design choices

  • Single-carrier + master rewrite. The muxer behaviour was grounded against the exact ffmpeg the published image ships (see Contract verification). ffmpeg's hls muxer advertises the subtitle group only on the single variant that physically carries the subtitle stream in -var_stream_map, and it segfaults, reproducibly, if any additional variant references that group directly (tested for both mpegts and fmp4). So the subtitle stream is attached to one variant (crash-free) and the SUBTITLES reference is then broadened to every variant by rewriting the master playlist once ffmpeg has written it. The rewrite (rewriteMasterPlaylist) also applies the configured LANGUAGE, NAME, DEFAULT and AUTOSELECT.
  • The rewrite is a readiness gate. With subtitles configured, the channel is not declared running and the pull-push egress is not started until the master has been finalized, so a one-shot downstream consumer cannot cache the un-broadened, single-variant master for the whole session. A not-yet-complete master keeps the encoder in starting and is retried on the next monitor tick.
  • Crash-safe and atomic. The finalized flag is reset per ffmpeg attempt (after the HLS dir is purged), so a stale, already-rewritten master from a previous crashed or re-dialed attempt is never mistaken for the current one. The rewrite is written to a temp file and rename()d into place so @fastify/static and the pull-push fetcher never read a truncated master, and the master is sanity-checked as complete (subtitle rendition present plus every expected variant) before it is rewritten.
  • Bounded sidecar fetch. The subtitle input carries -rw_timeout so a stalled or unreachable subtitle server cannot pin the encoder in starting with no output.
  • Labels via the rewrite, not ffmpeg metadata. Applying LANGUAGE/NAME/DEFAULT through the master rewrite avoids ffmpeg's %v segment-naming corruption when those attributes are set on a variant-carried subtitle in -var_stream_map.

Contract verification

Grounded against the ffmpeg binary the repo's Dockerfile produces (node:18-alpine + apk add --no-cache ffmpeg), not against ffmpeg docs of another version.

  • Base image: node:18-alpine, digest sha256:8d6421d663b4c28fd3ebc498332f249011d118945588d0a35cb9bc4b8ca09d9e
  • Alpine: 3.21.3
  • ffmpeg: ffmpeg version 6.1.2, built with gcc 14.2.0 (Alpine 14.2.0), configuration includes --enable-libsrt and --enable-openssl
  • hls muxer probe (ffmpeg -h muxer=hls): Default subtitle codec: webvtt., -hls_subtitle_path, -var_stream_map present
  • Input protocols (ffmpeg -protocols): file, http, https, tls present, so http(s) sidecar URLs are supported by this build
  • -rw_timeout: documented as Timeout for IO operations (in microseconds); empirically a hung http server (accepts, sends nothing) is bounded, ffmpeg exits with Operation timed out after the configured interval rather than hanging

End-to-end probe: a two-variant ABR encode with a sidecar WebVTT input, using the exact args this code generates (-rw_timeout <us> -i <sidecar.vtt>, -map 1:0 -c:s webvtt, -var_stream_map "v:0,a:0 v:1,a:1,s:0,sgroup:subs"), exits 0 and writes media_1_vtt.m3u8 plus .vtt segments. ffmpeg's master playlist (before the rewrite step):

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-STREAM-INF:BANDWIDTH=4540800,RESOLUTION=1280x720,CODECS="avc1.f4001f,mp4a.40.2"
media_0.m3u8

#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="subtitle_1",DEFAULT=YES,URI="media_1_vtt.m3u8"
#EXT-X-STREAM-INF:BANDWIDTH=3440800,RESOLUTION=640x360,CODECS="avc1.f4001e,mp4a.40.2",SUBTITLES="subs"
media_1.m3u8

Only media_1 references the subtitle group; media_0 does not. The multi-variant crash was reproduced against this same binary when trying to make ffmpeg reference the group from both variants directly. After running that real master through rewriteMasterPlaylist, every variant references the group and the rendition carries the configured labels:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-STREAM-INF:BANDWIDTH=4540800,RESOLUTION=1280x720,CODECS="avc1.f4001f,mp4a.40.2",SUBTITLES="subs"
media_0.m3u8

#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",NAME="English",LANGUAGE="en",AUTOSELECT=YES,DEFAULT=YES,URI="media_1_vtt.m3u8"
#EXT-X-STREAM-INF:BANDWIDTH=3440800,RESOLUTION=640x360,CODECS="avc1.f4001e,mp4a.40.2",SUBTITLES="subs"
media_1.m3u8

Subtitle-exhaustion probe (A/V 12s, sidecar VTT with cues only for 0 to 4s): ffmpeg exits 0, the A/V variants continue for the full duration, and the WebVTT rendition is padded with empty segments (media_10.vtt, media_11.vtt, media_12.vtt) rather than aborting. The vtt rendition segment naming in this build is media_<variant><index>.vtt.

Tests

npm test, npm run typecheck, npm run lint pass locally (42 tests). Unit tests cover: subtitle input arg generation in both RTMP and SRT-caller mode (including the -rw_timeout bound and the SRT + subtitles arg order), output arg / var_stream_map generation, the no-subtitle byte-identical path, the master rewrite happy path (full-string golden), rewrite idempotency, single-variant masters, the mediaIndex clamp, a MEDIA line without URI (left alone), masterIsComplete gating, the disk finalize seam in a temp dir (atomic write, idempotency, refusal on an incomplete master), a monitor-loop test that the finalize gate holds in SRT mode until the master is complete, and config parsing (protocol rejection, malformed-URL message, defaults, name fallback, bool parsing).

Known limitations

  • Finite sidecar VTT on a long live stream. A sidecar WebVTT that ends before the live A/V does exhausts: from the exhaustion point on, the subtitle rendition carries empty WebVTT segments (verified above for a finite source; ffmpeg does not abort). There is no detection signal for this, an operator watching only encoder status would not see it. A live subtitle deployment should point at a live or long-enough WebVTT source. This is the open sidecar-live-lifecycle question and is left for maintainer input; it is a source-lifecycle concern, not something this passthrough can fix.
  • Single track wired from env. The internal API accepts an array of tracks, but only one track is validated and read from the environment in this PR.
  • In-band (MPEG-TS) subtitle carriage is intentionally not implemented; the path is left open for it.

Reconciliation with PR #11 (SRT caller, now merged)

PR #11 merged to main first, so this branch has been merged with main and the generateInput collision reconciled here (RESOLVED):

  • generateInput is now (rtmpPort, streamKey, inputUrl?, subtitles = []). The sidecar -rw_timeout/-i args are appended in BOTH the RTMP-listener and the SRT-caller branch, so the subtitle is always input index 1 and -map 1:0 resolves to it in either mode. A unit test asserts the SRT + subtitles arg order (SRT -i first, sidecar second).
  • The subtitlesReady finalize gate is woven into feat: configurable input with SRT caller mode (INPUT_URL) #11's restructured monitor loop, gating the same running flip (and startPullPush) in its new location.
  • subtitleMasterFinalized and the finalize attempt counter are reset in startFFmpeg, which runs after feat: configurable input with SRT caller mode (INPUT_URL) #11's cleanup() purge on both the initial dial and every re-dial. Without this, a re-dial after a briefly-connected attempt would leave the fresh master un-broadened. feat: configurable input with SRT caller mode (INPUT_URL) #11's cleanup() remains the single owner of the HLS-dir purge (its stale-index protection is preserved); this branch does not add a second purge.
  • Nothing re-added in the merge bypasses feat: configurable input with SRT caller mode (INPUT_URL) #11's redactSecrets; the subtitle URL flows through the same spawnargs redaction, and no finalize log prints the URL.
  • A monitor-loop test asserts the finalize gate holds in SRT mode: with an incomplete master the encoder stays starting, and only goes running (broadening the master) once ffmpeg has written a complete master.

The combined SRT-caller + sidecar case is verified at the argument level (unit test) and its output stage is the same pipeline proven end to end in Contract verification; a live SRT-listener container probe of the combined case was not run.

CI note

All four checks pass. The pre-existing docs/api.yaml prettier issue is excluded by the .prettierignore added in #14. A separate chore commit in this PR formats .github/workflows/sync-fork.yml (added on main with double-quoted strings): the pretty check only runs on pull requests, so that violation was never caught on the push to main and would otherwise fail the check here.

svensson00 and others added 3 commits July 9, 2026 16:39
Add optional sidecar WebVTT subtitles to the HLS output. When SUBTITLE_URL
is set, the encoder fetches the WebVTT source alongside the A/V input and
publishes it as a segmented WebVTT rendition referenced from the master
playlist as an #EXT-X-MEDIA:TYPE=SUBTITLES group. Unset leaves today's
video+audio only output unchanged, so this is fully backward compatible.

The config is shaped as a track array so more tracks can be added later,
and the sidecar input keeps the MPEG-TS style muxing untouched so in-band
subtitle carriage can be added later without reworking this path.

ffmpeg's hls muxer only advertises the subtitle group on the single
variant that carries the subtitle stream, and crashes if any other variant
references the group directly, so the stream is attached to one variant and
the SUBTITLES reference is broadened to every variant afterwards by
rewriting the master playlist once it has been written.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
svensson00 and others added 3 commits July 9, 2026 17:01
Harden the seam that broadens the subtitle group across the ABR ladder:

- Purge the hls output dir at the start of every ffmpeg run so a stale,
  already-rewritten master from a previous crashed run cannot be mistaken
  for this run's output, and only mark the master finalized after a write
  this process performed.
- Turn finalizeSubtitleMaster into a readiness gate that returns success or
  failure. The running flip and the pull-push start now wait for it, so a
  one-shot downstream consumer can no longer cache the un-broadened master
  for the whole session.
- Write the rewritten master via a temp file plus rename so readers never
  see a truncated playlist, and sanity-check the master is complete (the
  subtitle rendition plus every expected variant) before rewriting rather
  than trusting mere existence.
- Bound the sidecar subtitle input with -rw_timeout so a stalled subtitle
  server cannot pin the encoder in starting with no output.
- Surface finalize failures at warn/error level and align the readiness
  gate predicate with the rewrite predicate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move subtitle env parsing into a side-effect-free config module so it can
be unit tested without starting the server. Wrap the URL parse so a typo
yields an intentional error instead of a raw ERR_INVALID_URL, and warn when
SUBTITLE_URL is set while HLS_ONLY is false, since only the HLS output
carries subtitles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
svensson00 and others added 2 commits July 10, 2026 14:01
…passthrough

# Conflicts:
#	readme.md
#	src/encoder.test.ts
#	src/encoder.ts
#	src/server.ts
The workflow added on main is not prettier-clean (double-quoted strings),
which fails the pretty check. The check only runs on pull requests, so it
was not caught when the workflow was pushed. Format it so the check passes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WebVTT subtitle passthrough into the HLS output

1 participant