Hyperframes Media

HyperFrames media

Use fish-audio-api for narration and media-use to ingest reviewed local assets. This package supplies workflows and code; users supply accounts, permitted voices, music, sound effects, images and video.

Requirements and scope

Node >=22.20 and FFmpeg/ffprobe must be on PATH. Read requirements before generation. The engine loads the video project's .env; existing environment values win. Fish uses its official API with FISH_API_KEY and FISH_REFERENCE_ID. Word timing uses OpenAI whisper-1 with OPENAI_API_KEY, or an explicitly configured OpenAI-compatible WHISPER_API.

The shared engine performs one POST per requested narration line and one transcription POST. Neither is automatically retried. It preflights both API configurations before creating voice audio. A whole --only tts rerun creates narration again; use the scaffolded paragraph pipeline for cached/surgical revisions. All generated audio needs listening review.

Shared engine

shCopy code
node <MEDIA_DIR>/scripts/audio.mjs --request audio_request.json --hyperframes . --out audio_meta.json --only tts
node <MEDIA_DIR>/scripts/audio.mjs --request audio_request.json --hyperframes . --out audio_meta.json --only bgm,sfx

The second command needs no API credentials. It preserves and validates any existing voice metadata. Example request (all files shown must be supplied by the user):

jsonCopy code
{
  "lang": "zh",
  "fish": {
    "transport": "official-api",
    "request": { "normalize": true, "prosody": { "speed": 1 } }
  },
  "lines": [
    { "id": "s0", "text": "这是一段待审听的旁白。", "sfx": [
      { "name": "confirmation", "path": "library/confirmation.wav", "offset_s": 0.2 }
    ] }
  ],
  "bgm": { "path": "library/underscore.wav", "source": "reviewed-local", "volume": 0.18 }
}

fish.reference_id and fish.model are optional project overrides; no personal voice is supplied. fish.request uses official API fields, such as top-level temperature, top_p, normalize and prosody. The adapter owns text, reference ID and WAV format. It rejects credential-like fields and website backend configuration.

Output freezes assets/voice/<id>.wav and returns voices[].words as {id,text,start,end}. voice_id, fish_model and fish_transport describe the resolved non-secret profile. API keys and error bodies are never persisted. A transcription failure stops delivery with the voice WAV still on disk; do not invent word timing or claim captions are synchronized.

BGM/SFX come only from explicit local paths or the user's project manifest. Missing/non-audio files produce anomalies and are skipped; they never trigger search, generation or a fallback library. Review every anomaly before final delivery. Frozen files have content hashes so repeated filenames do not overwrite unrelated sources.

Discovery and approval

In a scaffolded project, use npm run audio:discover -- --type sfx --query "confirmation". No review catalogs are included. Install media-use in .agents/skills, or set MEDIA_USE_SKILL_DIR to that skill directory; set MEDIA_CATALOG or pass --catalog for your own workspace-local catalog. Empty search results mean the user must supply or select media. Catalog membership is not approval: audition, record source/license/hash and a review note, then ingest the exact bytes through media-use.

Reference guide

API credentials do not replace the user's authorization for generation. Stay within the requested scope, preview a short sample, preserve original audio and keep listening/review decisions distinct from technical validation.