Replicate a reference video
Credit the original. A replica is a study of someone else's work: name the author and
link the original in the experiment's brief.md, in sandbox/INDEX.md and wherever the
replica is shown (the README's gallery). Keep the reference video and its soundtrack out of
git (out/), and commit the replica's render without the original's audio (-an).
Worked example: sandbox/2026-09-24-what-do-you-love/, a replica of a video by Kevin Ngo
(@kevin_t_ngo) (a 28 s paper-cutout film, 2160²,
24 fps, animated on twos). Its whole-video difference went 29 → 22.6 (animatic) → 14.2
(detail pass), and the user's verdict was «I have to zoom in to see differences». Its
review.md tells the story round by round; segments/README.md explains how it is built.
Transcribing is copying. A tool that reads an artist's pixels (or vectors, or frames) off
the reference and replays them is a copy of their art, not a drawing by code: keep that
data in the experiment's private/ (gitignored), never commit or publish it, and say so in
the brief. What goes in the repo is the kit, the tools and the lessons (pixel-building).
Rights: a replica of someone else's film is an internal study. The reference video and
its audio stay in the experiment's out/ (gitignored), never in the repo, and the result
is never published as ours.
The detail gate (mandatory)
node engine/detail.mjs <scene.js> <reference.mp4> --every 1 splits every instant into
6×6 zones and compares colour (after a blur) and texture (high-frequency energy: grain,
dots, noise, fine lines) with the reference. It passes only at the level of the replica the
user approved: colour median ≤ 8, p90 ≤ 27 (colour compared at a 12 px blur, over a halftone pitch, so dot phase is not a colour error), ≤ 6 % of zones too clean, ≤ 12 % too busy.
Nothing is shown to the user before it passes; agents working on elements run it on their
own instants (--times) and do not report «done» with a failing gate or a list of things
still off. «Too clean» almost always means a missing texture layer (print noise, paper,
fine hatching), not a missing object.
The four principles
- Detail is the job. Timing and palette give you an animatic. The user judges the craft of every element at a glance: a dog made of five ellipses, round hands, a ring drawn as one stroke or polygonal shapes read as low effort next to a reference built piece by piece. Hold every element to the style's detail bar (paper-cutout: «Detail» in style-paper-cutout).
- Measure, never eyeball. Positions, sizes, colours, timing, speed: every number in the scene comes from a probe on the reference (below). Eyeballed values were wrong by 15–50 % every single time (the flower 17 % small, the girl 47 units too high, the writing too slow, the '?' off the page).
- Work per drawing. Hand animation changes drawing every 2 frames (1/12 s) and holds.
Author motion as tables with one row per drawing (
d = Math.floor((t - t0) * 12 + 1e-6)), not as smooth curves; ease curves look wrong next to a film animated on twos. - One element per file, compared at full resolution. Every object is its own file with a small drawer contract; each is crop-compared against the reference until the same pieces, shapes, colours, textures and layering show side by side.
Workflow
- Set up. Reference in
<exp>/out/reference.mp4; its audio extracted toout/reference-audio.wavand used asaudio: { mix: 'out/reference-audio.wav' }so the renders are judged with the real timing. Logical space = the frame (logical: [1000, 1000]for a square video: 1 unit = 2.16 px at 2160, and fractions × 1000 = units). - Shot list.
reference.mjs sheets --every 0.25to read the film,reference.mjs cutsfor the exact cut frames (cuts land on frames). Declare every cut, even 0.25 s cards, as a shot inshots(review.mjs then stops reporting them as jumps). - Palette.
reference.mjs colors <video> <s> name=x,y …for every colour, never guessed. - Animatic of the whole film first (structure, timing, rough shapes), compared with
compare --every 0.5. Then the detail pass, element by element (most of the time goes here: in the example the detail pass took several times longer than the animatic). - Measure each element. Full-resolution crops with a grid
(
ffmpeg -ss T -i ref.mp4 -frames:v 1 -vf "crop=W:H:X:Y,scale=…,drawgrid=w=50:h=50:t=1:c=black@0.25"), a strip of one shot's drawings (-vf "select='not(mod(n\,2))',scale=400:400,tile=6x1"),boxfor where a colour sits,runsfor edges and bands,track/facefor motion. - Build the element in its own file, authored in measured coordinates (reference pixels or logical units around an anchor), so that it lands 1:1 with scale 1.
- Calibrate against your own render. Render the stretch small
(
render.mjs --from a --to b --size 500 --out x.mp4) and run the same probe on it. Compare the two tables and correct factors (our flower's horizontal reach turned out to be 0.92 R, not 1.07 R as assumed). - Crop-compare (
compare --times … --crop x,y,w,h, fractions) until the side by side matches; then the whole video (compare --every 1) for the number. The number helps; your eyes decide. Look at every image you produce. - Log the round in
review.md(what the user said verbatim, what was measured, what changed, the new number) and move the lessons into the skills.
Measuring tools (engine/reference.mjs)
| Command | What it answers |
|---|---|
sheets <video> --every 0.25 | the film at a glance, labelled with seconds and frames |
cuts <video> | exact cut frames |
colors <video> <s> name=x,y … | the palette (5×5 px mean at fractions of the frame) |
box <video|png> <s> --color #hex --region x0,y0,x1,y1 | where a colour sits (2–98 % box, logical units): hair, ink of a text, a prop. Run it on your still too |
runs <video|png> <s> --row 0.6 (or --col) | colour runs along a line: paper edges, margins, sky bands, ruled lines |
track <video> --color #hex --from --to [--crop] | per frame: centre, box, area of a colour. The motion of a character (bounce, squash, reach) |
face <video> --body #hex --from --to | per drawing: centroid of the dark marks inside the body colour = where the face is, even when the limbs change |
compare <scene> <video> --times … [--crop] [--every] | side by side with a difference number per pair |
Throwaway probes are fine for one-off questions (a text's ink box, a column scan); when one proves useful twice, add it here.
Recipes that worked
- A character's performance per drawing (the flower in the montage): a table
[face x, face y, reach, vertical stretch, expression, face tilt]per drawing fromtrack+face, and a bounce pattern (big entry, squash, recover, stretch up, settle). Squash & stretch the limbs/rays only; the face never deforms. Expressions change with the drawing (open 'o' on the entry, squint on the squash, open laugh on the stretch). - Entry accents: radial dashes only on a card's first drawing (colour, centre, radii and count measured per card); a 7–8 % bigger object on the first drawing only where the reference has it (measure; not every card pops).
- Card words: the word writes itself as a fast wipe (≈1.5 letters on the first drawing, whole by 0.16 s), width/tilt/baseline measured per card.
- Objects the character sits inside (letters flying out of a book, a boat, a cup): a
back and a front layer with the character between them (
order: 'split'). - Writing on paper: the pen follows the text metrics (
penAt: width of the written prefix + a fraction of the next letter), at the measured letters per second (keys per drawing), the forearm leaning from an off-screen elbow. - Paper actions drawing by drawing: tearing along the perforation (square tabs, scraps
flying up, the lifted page's shadow clipped to the pad, the next blank page under the
rings), folding a plane (a table of five paper shapes with flaps and creases, mittens at
the corners), turning a note over in perspective (
noteFliponMotion.quad; the front turns without the answer written on the back). - Lettering to match hand-lettering: pick the closest free font, then measure: width of
each line, x-height (squash with
sy), line ratio (per-line letter spacing), stroke (felt-tip = a thinned core inside a lighter rim). - Miniatures (the heart gathering every object): redrawn chunkier with fewer pieces, not just scaled; pop-in order measured per drawing.
- The same character in several framings (girl in the interior, small in the window,
bigger when she leans in): one puppet with a scale and layers (
layer: 'body' | 'arms') so a desk can go between torso and forearms.
Pitfalls met, and their fixes
| Symptom | Cause | Fix |
|---|---|---|
| The '?' ran off the page | the font's second line is relatively wider (1.27× vs 1.21×) | per-line letter spacing keeping the reference's line ratio (WL.lineSpacings) |
| Bold, flat text next to a felt-tip reference | strokeText over fillText only thickens | erode the glyph with a destination-out stroke on an offscreen canvas, inside a lighter rim; cache per scale (getTransform()) |
| Shaded pieces painted as brown blobs | shade() returned hsla(), Paper.marker parses hex | colour helpers return hex |
| The character sat 47 units too high in every interior | anchor guessed | box of the hair on the reference and on our still; move the anchor |
| The flower 17 % too small | reach vs radius assumed | calibrate by tracking our own render |
| Drawings one frame late | ffmpeg -ss rounding lands on the next frame | seek at frame/24 − 0.004 |
| Cuts one frame off, or a sweep of cut times all one frame early | times read off compare/-ss sheets | cut times from a frame-index scan (select=between(n,a,b) or all frames piped raw, diff > 25 between neighbours): the index is the truth |
| Rings and flecks «almost» right but the gate says too clean | vector strokes, softened by the print's spread, thinner than the painted ones | measure widths on a pixel profile; paint annuli with varying width; lower spread for line-only shots |
track catches other things | the colour is shared | tighter --tol, --crop, or a colour unique to the character |
| Rays looked like blobs when short | width kept constant | short rays slimmer too |
| A caught plane drawn over or under the whole flower | one layer | rays in an arc drawn over the plane, the face over everything |
| Waves / tentacles / steam «boiled» | outline re-torn every frame | 2–4 cached replacement drawings swapped on twos |
| A hole looked like a grey ring | a ring polygon filled | PaperDetail.punch (torn rim + destination-out) |
| Cup, band, tag bulged | spline overshoots between long and short segments | cspline (centripetal) or a torn polygon for straight paper |
| Blurry sprite | cached at low resolution | sprite res ≥ 1.1 × the largest scale it is drawn at |
| Knit screamed at small size | texture alpha tuned at full size | knit alpha 0.1–0.2 on small pieces |
review.mjs flags flashes at 15–16 s | the reference's own 0.25 s colour cards | documented in review.md; a real piece would soften them |
| Subagent worktrees ended up in a commit | git add -A with .claude/worktrees/ present | .claude/worktrees/ in .gitignore |
Splitting the work across agents
A replica has dozens of elements; parallel agents (Agent tool with isolation: "worktree")
did four groups of montage objects and the exterior shots while the lead did the
choreography.
- Contract first: one file per element with a tiny drawer contract
(
Things.x(g, x, y, s, t, part?)), shared helpers in a kit file. Agents edit only their own files and their rows in the shared tables; the lead owns choreography, the character and common kits. - Brief: copy
agent-brief.mdfrom this folder and fill it in: the user's words (why detail matters), setup (ln -s <repo>/node_modules node_modulesin the worktree), the files and the contract, coordinates, tools, rules (deterministic, cut once, look at every image, commit but don't push) and the report format (values for the shared tables, remaining differences, helper candidates, one-line lessons). - Merge by file:
git checkout <agent-branch> -- <their files>, then copy their table values by hand and verify withcompare. Never merge a shared file wholesale: their version is stale. - Harvest the reports: agents found real lessons (layering by whose white border is on
top, replacement drawings, two-strip rings, miniatures redrawn chunkier). Move them into
the skills and promote their good helpers into the style kit (that is how
cspline,taper,curl,markerPath,punch,wordBars,sheetMusicandmapPaperreachedPaperDetail).
Lessons
- 2026-09-24 · what-do-you-love · Whole-frame comparisons hid the lack of craft; the user saw it at once. Crop every element at full resolution.
- 2026-09-24 · what-do-you-love · Pin a moving prop (a paper plane) by three measured points per drawing (an affine map): perspective and banking come for free, where position + angle + scale never matched.
- 2026-09-24 · what-do-you-love · Key a pose at a few drawings but correct the position every drawing with the tracked centre; straight lines between keys lose the arc.
- 2026-09-24 · what-do-you-love · Measure secondary motion before animating it: the head sway assumed for the girl in the window was a perfectly still head in the reference.
- 2026-09-24 · what-do-you-love · A growing dashed trail must anchor its dash pattern to the path's start, or the dashes crawl.
- 2026-09-24 · what-do-you-love · Check the drawing phase once (which frames start a drawing) with frame-to-frame differences decoded without seeking.
- 2026-09-24 · what-do-you-love · After the measured detail pass the user needed to zoom in to see differences; the one thing seen at a glance (the '?' off the page) came from a font metric nobody had measured. Measure text too.
- 2026-09-25 · opus5-riso · The detail gate, round 3 (six card agents, ~10 M tokens, median 18 → see review.md): the order that moved the numbers was (1) the press itself (inks re-measured on flat solids and divided by the paper, solids at full coverage, paper fibres instead of confetti), (2) timing by frame index (cuts, 2-frame repeats, per-frame camera pushes and ring radii), (3) elements from a side-by-side audit, then (4) regional tone: once elements are in, the ink each 45–90 px block gets decides the gate; fit it per block and calibrate against our own render (fit ours, move by the difference, 3–4 passes), adding tone as halftone dots, never as a flat fill.
- 2026-09-25 · opus5-riso · Separate colour from placement before working on a card: remove a smooth colour offset (at ~44 px) and re-score. What remains is placement; what disappears is tone.
- 2026-09-25 · opus5-riso · Grids, contours and blob lists read off the reference are transcription: they live in
private/<card>-data.js(gitignored, loaded optional) from the first commit, with a described fallback in the committed card. Moving them later leaves them in the git history. - 2026-09-26 · felt-cats · Replicating a dance: silhouette IoU is the wrong target. Three agents keyed the cats to 0.70–0.75 IoU and the user saw «no se parece en nada»: a rig without knees can fill a silhouette standing stiff. Measure motion (head bob, weight shift, crouch, paws, steps) with keypoints and fit a rig that can make it (IK legs with knees, paw targets), then check the side by side in motion, not stills.
- 2026-09-26 · felt-cats · A pose model (ViTPose+ AP-10K via transformers, CPU) finds eyes and front paws on upright kittens seen from the front, not hips, knees or backs. Fit the rig per drawing to the keyed mask drawn as 2D capsules (fast in node) plus those keypoints, with limits read off the frames (a seen face looks at the camera, depth and lean near a prior); let hand keys cover what the model can't see, crossfaded in and out.
- 2026-09-26 · felt-cats · A silhouette fit without limits turns figures round, tips heads back and lays bodies down where figures overlap: IoU can't tell front from back.
