Make Npcs Talk

Make NPCs talk

The finished loop, as built for GTA V (examples/gta-v), with times measured in the game:

Copy code
player's mic ──► voice activity ──► speech-to-text ──► who? + scene ──► LLM in character ──► voice server ──► NPC speaks
   (open mic,       (end of the       (fal Nemotron,     (the game)      (Cerebras, with      (Fish Audio on     (streamed PCM,
    push-to-talk,    sentence)         0.4-0.8 s)                         persona + lines,    an H100, first     panned, subtitles,
    or typing)                                                            0.3-1.0 s)          audio ~0.4 s)      mouth, action)
                                                                │
                                                                └──► action choice (Jev, ~0.1 s, in parallel) ──► animation

From the end of the player's sentence to the NPC's voice: 1.2-2.2 s warm with push-to-talk (logged exchanges: 1.15 s, 1.20 s, 1.57 s, 2.19 s); the open mic adds the 0.7 s of silence it waits for before deciding the sentence ended. The LLM sometimes spikes to 5-8 s, so log every step's time per exchange.

The loop: build it in this order

1. Recon the game. Version, how code runs inside it (script hook, mod loader, plugin API, or your own engine), single-player or not, how other mods find nearby characters and play sounds. Read references/game-hooks.md for the engine, and knowledge/ for anyone who already did this game.
2. Pick the engines (table below). Default: Fish Audio Drama 3 on Modal for voices, Cerebras for the LLM, fal for speech-to-text.
3. Stand the voice server up first, outside the game (skill fish-voice-server): npctalk health, then npctalk say "..." --voice Name/clone. A game is the worst place to debug a GPU server.
4. Make reference voices for two or three characters (skill voice-references).
5. Vertical slice in the game: one NPC, typed text → LLM → voice → playback. Implement the four hooks below, nothing else. Log stt / llm / tts / total for every exchange.
6. Voice input (skill voice-input): push-to-talk first, then the open mic.
7. Characters (skill npc-dialogue): persona files, the character's own lines as style samples, scene facts, per-NPC history, output cleaning.
8. Actions (skill npc-actions): what they physically do, and doing what they're told.
9. Verify in the real game, with a dev mode that shows the target, the timings and the mod's state (below).
10. Show it (skill npc-showcase-clips), and if voice likeness matters, measure it (skill voice-likeness-eval) before claiming a setting is better.
11. Write it down: a field note in knowledge/games/<game>/ (see knowledge/README.md).

The four hooks a game has to provide

Everything else is game-free code. In the GTA V example about 2,450 of the mod's ~5,500 lines (dialogue, LLM, voice clients, streaming playback, speech-to-text, open mic, action picking) build and run outside the game: examples/gta-v/tools/npcvoice-check compiles exactly those files into a console app.

Keep the hooks thin. If a game exposes them, the rest of this repo's code carries over.

Choosing engines

Costs: an H100 runs about $4/hour while it's up; it scales to zero 5 minutes after the last request, and the next request waits 1-3 minutes for a cold start. The game pings the server while it runs, so it stays warm exactly while someone plays (server/keepwarm.py holds it up for a demo without the game).

Verify like you mean it

• Log every exchange with each step's time and what was heard and said. Most "it doesn't work" reports are answered by the log (in GTA V: "open mic -> nobody near" 755 times = the target lookup found nobody).
• Show the mod's state on screen. A mod switched off with a hotkey looks exactly like a broken one; say "off" somewhere visible. Show who would be talked to (a marker) in a dev mode.
• Test the network half without the game (npcvoice-check in the GTA example, npctalk say for the server).
• Measure, don't eyeball, voice likeness (skill voice-likeness-eval). Two runs of ten lines fooled us once; ten runs didn't.

Rules

• Single-player, or servers you run. Never inject into online games with anti-cheat (for GTA V: never launch GTA Online with a script hook installed).
• Voices you have the rights to. Your own, an actor who agreed, a licensed or generated voice. Cloning a game's characters for your own single-player experiment is one thing; publishing those clones is another. Never commit voice clips, game audio, or the game's dialogue (npctalk check blocks audio and transcripts).
• No secrets in the repo. Keys come from environment variables. A server URL is a secret too when its /health needs no token: anyone with it can keep your GPU awake.
• Content. NPCs swear if the game's characters do. Slurs are filtered by default and opt-in per persona (skill npc-dialogue).
• Ask first before driving the user's keyboard and mouse in a running game, installing into game folders, or publishing anything.

Full reasoning: references/safety.md. Architecture and file map: references/architecture.md.