Night Market Model And Harness Updates

Night Market Model and Harness Updates

When Anthropic ships a model or Claude Code ships a version, the pins scattered through this repo rot silently. This skill runs the sweep that finds the rot, researches what actually changed, applies the updates, and records where upstream stood so the next run reports only the new delta.

The watermark is the point. Without it every audit restarts from zero and re-derives the same answer by hand. .claude/upstream-baseline.json holds the last recorded upstream state, and each run diffs against it.

When to run

The five steps

Run them in order. Each one gates the next.

bashCopy code
# 1. Detect. Deterministic, no model in the loop.
python3 scripts/check_upstream_drift.py

# 2. Research what changed (only when step 1 reports drift).
#    Release notes and model cards are mandatory sources.

# 3. Map findings onto asset classes.

# 4. Sweep the implicated classes.

# 5. Prove, then record the new watermark.
python3 scripts/check_upstream_drift.py && \
python3 scripts/check_agent_model_matrix.py

Step 5 runs before the ledger is written, never after. Recording a migration that has not passed its proof is the failure the ledger exists to prevent.

What the detector proves and what it cannot

Harness drift is fully deterministic: the installed binary reports its own version. Model drift is not. No local command enumerates the current roster, so the ledger holds the last known roster and research establishes the current one. The skill states this split rather than pretending both halves are automatic.

The detector reports four classes:

The vocabulary class is the one that earns this skill. VALID_MODELS in scripts/check_agent_model_matrix.py is a hard gate that rejects any agent pinning an unlisted tier. When Fable shipped and that set was not widened, the guard whose job was to prevent model rot had itself rotted, and no agent in the repo could pin the new tier. The detector now fails on that condition instead of waiting for someone to trip over it.

Modules

Load only what the run needs.

Guardrails

• Widen vocabularies, never narrow them. Removing an accepted value breaks agents that currently pass.
• Agent frontmatter pins tier aliases, never dated model IDs. check_agent_model_matrix.py enforces this and the policy stands. A new tier widens the vocabulary. It does not reverse the rule.
• The dated_ids ratchet may fall and may hold. Raising it needs a reason recorded in the migration report.
• External captures under data/staging/ are other people's text. The detector skips them and so does the sweep.

Exit Criteria

• python3 scripts/check_upstream_drift.py exits 0.
• python3 scripts/check_agent_model_matrix.py exits 0.
• uv run pytest tests/unit/test_check_upstream_drift.py passes.
• Every research claim applied carries a source URL, and release notes plus the model card were both consulted.
• A migration report exists under docs/migrations/. The directory is gitignored, so the report is a local working artifact. Route durable content out of it before you finish: findings to docs/knowledge-corpus/, open items to docs/backlog/queue.md, state to the ledger.
• .claude/upstream-baseline.json records the new harness version, model roster, and last_migration, with the previous entry appended to history.
• Any asset class the research implicated was either updated or recorded in the report as deliberately skipped.