Evaluate Skill

Evaluate Skill

Operate Caliper: run a skill's eval, read what the results say, and trace each failure to the place that fixes it.

Prerequisites

The caliper CLI must be on PATH. This skill can be copied into an agent without the Caliper repo, so install it if missing:

bashCopy code
pipx install caliper-eval

caliper <command> --help is the authority on flags. REFERENCE.md covers the full spec format, engines, and what the results mean.

The four questions

A skill eval answers four separate questions. Each has its own measurement and its own fix location, so keep them apart: a blended number hides which half broke.

Spec shape

yamlCopy code
skills:
  - ./SKILL.md              # relative to the spec; installed, never preloaded
tasks:
  - name: What success looks like
    prompt: <what a real user would type; never names the skill>
    expect: <natural-language pass/fail criterion>
    assert: |               # optional deterministic Python check
      assert ...
    activates: [my-skill]   # frontmatter name: of the skill that should fire
  - name: Unrelated work stays unrelated
    prompt: <work no declared skill should answer>
    activates: []           # a trigger probe: no judge, cheap

When you write a spec, every task with expect: or assert: also asserts activates:, refusals included: the skill, plus any declared skill it delegates to on that task, and every prompt reads like a real user's request with the skill left unnamed.

The spec has no backend/model or judge: block. The engine is chosen at run time, independently for the skill and the judge: caliper run <spec> --model codex runs and grades on codex, and --judge-model picks a different judge. Backends are claude-code (default), codex, pi, and hermes. Each attempt runs in a fresh, empty workdir, so setup: builds fixtures there with relative paths.

Running

1. caliper validate <spec>. It never touches the network.
2. caliper run <spec> --k 1 to shake out spec and harness errors. Fix those before reading any score.
3. caliper run <spec> --k 3 --ablate <skill-name>, once (write skill:<skill-name> if an mcp: server shares the name). This is the control: the declared neighbourhood without this skill. The skill isn't installed, so editing SKILL.md can't move its number. Keep its results path (caliper list <spec-name> marks ablated runs) and re-diff against it. Re-run it only when the tasks or the declared skills change.
4. caliper run <spec> --k 3, then caliper compare <control.json> <spec-name> to see whether it earns its place. After each edit, also compare against the previous full run's path to see whether the edit held: a skill that got worse can still beat the control. A bare spec name resolves to that spec's latest run.

If a run stops with Not logged in, ask the user whether to run the login command the error names. With their consent, run it (it opens a browser for them to finish) and rerun; never run it without consent. hermes model is an interactive picker, so ask the user to run it in their own terminal instead.

Reading results

The success rate (successes / usable) is the headline. pass@k and pass^k are secondary views under --verbose. Activation is a separate scoreboard, over a different population: report it beside the success rate, never averaged into it.

Trace every failing task to where its fix belongs:

At k=3 one attempt is a 33-point swing. Before calling a change a win or a regression, re-run at k≥5. caliper compare flags any drop and never gates. A rewritten or shortened skill is safe to ship when, at k≥5, it stays within about 5% of the previous score and still beats the control.

Whose setup is measured

Runs load the user's own customizations by default (user skills, plugins, rules, settings and connectors; see REFERENCE.md for backend exceptions), which answers "does my skill work in my agent?". Isolate (--no-user-customizations, or user_customizations: false in the spec) when comparing backends or models, when the number leaves this machine (shared, published, compared with someone else's run), or when measuring the bare agent: each setup is different, so otherwise part of the delta is the setups. --ablate of the user's own skill needs no isolation, since both runs load the same setup.

Always tell the user which mode ran and what it loaded, from the report header's user customizations: line (absent means isolated), and relay any fix caliper compare suggests about it.

No eval yet?

If the user hasn't decided what to test (a skill with no .eval.yaml, or an eval whose gaps they want found), suggest grill-skill: it interviews them and writes the spec. To design tasks yourself, follow "Designing evals" in REFERENCE.md.

Done when

• every task has an observable criterion, and at least one has a deterministic assert:;
• execution tasks assert activates:, and the spec has at least one trigger probe;
• the full run beats the control on execution tasks, and holds against the previous full run. Trigger probes have no execution score: read them from the full run's activation;
• the spec passes caliper validate;
• the user has been told to commit the .eval.yaml beside SKILL.md. Saved runs under .caliper/results/ are useful for diffing over time and safe to gitignore.