Explain Video Pipeline
Turns a concept into a finished narrated video. No digital human, no avatar — an HTML deck is the source of truth, and everything else is generated from it.
The whole point of doing it this way: an HTML deck is code. It diffs, it takes a scripted screenshot, it re-renders identically, and one file drives both the visuals and the narration text. Drag-and-drop slide tools cannot do that.
Pipeline
concept ──discuss──▶ framework.json ──▶ index.html + narration.{txt,json,md} │ ┌─────────────────────────┴──────────────────────┐ ▼ ▼ TTS (voicebox/edge/say) Playwright records audio/voice.wav the animated deck │ │ build_srt.py trim the lead-in audio/voice.srt ─────────▶ caption PNGs ────▶ overlay + mux ────▶ final/*.mp4 │ ▲ mix_bgm.sh ──────────────────────────────────────────┘
Steps 1–2 are authoring (do them with the user). Steps 3–7 are mechanical and
scripts/build_video.sh runs them.
Step 1 — Agree on the framework before writing anything
Do not open an editor yet. Talk the concept through with the user, then write
framework.json:
json{ "concept": "...", "audience": "...", "one_liner": "...", "total_pages": 8, "target_duration_sec": 94, "structure": [{ "page": 1, "role": "钩子", "point": "..." }] }
Rules that hold up: 8 pages ≈ 90–110 seconds. One point per page. Give every page a role (hook / context / why / overview / method / method / method / close), not just a title. Show the framework and get a yes before drafting.
Step 2 — Deck and narration together
Copy references/deck-template.html to presentations/<slug>/index.html and
rewrite the slides. Write the narration into the slides as
data-narration, with data-duration in seconds — that markup is the single
source, and everything downstream reads it.
Narration voice: spoken, not written. Short sentences. No academic register. One to two sentences per page. Read it out loud; if you stumble, so will the TTS.
Then emit four sibling files:
| file | what | consumed by |
|---|---|---|
index.html | the deck | recording |
narration.json | per-slide title / narration / duration_sec | timeline |
narration.txt | continuous script, one blank-line-separated paragraph per slide, same order | TTS + SRT |
narration.md | human-readable script + page/time table | the user |
narration.txt's paragraph-per-slide structure is load-bearing — build_srt.py
uses paragraph boundaries to place the pauses between slides. Pressing E in
the deck exports narration.json directly from the markup.
Let the user read narration.md and approve before spending TTS time.
Steps 3–7 — Assemble
bashS=~/.claude/skills/minli-skill/skills/explain-video/scripts bash $S/build_video.sh presentations/<slug> --bgm audio/music.mp3
--tts auto (the default) uses the local Voicebox daemon if it answers, else
edge-tts if installed, else macOS say. Nothing downstream cares which one
ran, so a first pass on say is a legitimate way to see the whole pipeline
work before setting up a real voice:
bashbash $S/build_video.sh presentations/<slug> --tts say # zero install bash $S/build_video.sh presentations/<slug> --tts edge --voice zh-CN-YunxiNeural bash $S/build_video.sh presentations/<slug> --tts voicebox --voice <profile> # cloned voice python3 $S/tts.py --engine edge --list # what voices exist
Each step skips if its output exists, so iterating is cheap:
bashbash $S/build_video.sh presentations/<slug> --only mix,compose --bgm-volume 0.12 # BGM too loud bash $S/build_video.sh presentations/<slug> --from record # deck changed bash $S/build_video.sh presentations/<slug> --only srt,compose # captions off bash $S/build_video.sh presentations/<slug> --force # rebuild all
Scripts, if you need one on its own:
tts.py— narration →audio/voice.wavon whichever engine is available.tts_voicebox.py— the Voicebox backend on its own.--listshows profiles.build_srt.py— narration + measured audio duration →audio/voice.srt.deck_capture.py timeline|slides|video— timeline scaled to real audio; static PNGs; animated webm.render_captions.py— SRT → transparent PNGs +record/burn_caps.sh.mix_bgm.sh— voice + music →audio/voice_with_bgm.wav.compose.sh— align, burn captions, mux →final/<slug>.mp4.
Use .venv/bin/python if the project has one (Playwright and Pillow live
there); build_video.sh finds it automatically.
Things that will bite you
Audio duration is the master clock. Authored data-duration values are
estimates. TTS decides the real length. Everything — slide timeline, captions,
recording — gets scaled to the measured voice.wav duration, never the other
way round. Never hardcode a duration; always ffprobe it.
Playwright's webm is longer than the sum of the waits. Browser startup
happens before the first deckAPI.go(), and that surplus sits at the head of
the file. compose.sh trims vid_dur - audio_dur off the front before burning
captions. Skip that and every caption lands late by ~2–3 seconds. This was the
"字幕和音频不同步" bug.
subtitles= may not exist. Homebrew ffmpeg is frequently built without
libass. Check with ffmpeg -filters | grep subtitles. compose.sh falls back
to Pillow-rendered transparent PNGs overlaid with
enable='between(t,start,end)' — which also gives better CJK font control.
Caption PNG chains scale fine to ~50 cues; past a few hundred, split the video.
amix normalizes by default. It divides each input's gain by the input
count, so the BGM disappears and raising volume= does nothing. Always
normalize=0 and set levels explicitly. Voice at loudnorm=I=-16, BGM at
volume=0.20, alimiter=limit=0.97 on the sum. Verify with volumedetect
rather than trusting the graph.
Synthesized sine pads sound like tinnitus. The "滋滋声" complaint. Use a real
music file. --bgm synth is only a no-network fallback.
Voicebox hangs need clearing. /generate/{id}/status is a blocking SSE
stream — poll /history/{id} with Accept: application/json instead. A crashed
worker leaves jobs stuck in generating forever and blocks new ones;
tts_voicebox.py --cancel-stuck clears them. If the daemon itself is
unresponsive, kill the orphaned voicebox-server and reopen the app.
Don't ASR your own script for subtitles. The text is already known exactly;
Whisper only adds transcription errors. What's missing is timing, and
distributing known text across the measured duration by character weight lands
within ~0.3s per cue. (Voicebox's /transcribe also failed on every model tier
in practice.)
A deck sized for the preview looks empty at 1920×1080. The "右边太空了"
problem. Two causes, both fixed in the template's body.record-mode block:
max-width on slide children strands the right third, and rem-capped
clamp() font sizes tuned for the ~1120px preview stage stay small when the
stage grows to full frame. Record mode drops the ceilings and sizes type in
vw, and reserves padding-bottom: 16vh so vertically-centred content sits
above the caption band instead of behind it. Always eyeball a real frame
(ffmpeg -ss <t> -i final.mp4 -frames:v 1 f.png) before calling it done.
Output layout
presentations/<slug>/ framework.json index.html narration.{json,txt,md} audio/ voice.wav voice.srt bgm_loop.wav voice_with_bgm.{wav,mp3} record/ timeline.json animated_capture.webm aligned.mp4 caps/ caps_meta.json burn_caps.sh captioned.mp4 final/ <slug>.mp4
Requirements
Required: ffmpeg + ffprobe; Python 3.9+ with playwright (then
python -m playwright install chromium) and pillow.
bashpip install playwright pillow && python -m playwright install chromium brew install ffmpeg # macOS; any recent ffmpeg works
For the voice, one of: Voicebox on 127.0.0.1:17493 (cloned voice, best),
pip install edge-tts (free neural voices, cross-platform), or macOS say
(already there). A CJK font is needed for Chinese captions — render_captions.py
probes for Arial Unicode.ttf, Hiragino Sans GB, or STHeiti.
Developed and run on macOS (Apple Silicon). The Python and ffmpeg steps are
portable; --tts say is the only macOS-only piece.
Reference
references/deck-template.html— working deck with the fulldeckAPIrecording contract, staggered enter animations, and[data-step]progressive highlights.
