Rewardkit

Help the user write task verifiers with Reward Kit. Reward Kit is a lightweight Python package that turns a directory of criteria files into a reward score. Each criterion is a Python function call or a TOML judge file; folders become separate rewards.

Setup in a task

Put criteria alongside test.sh in the task's tests/ directory:

Copy code
tests/
├── test.sh
├── checks.py         # programmatic criteria
└── judge.toml        # optional LLM/agent judge

tests/test.sh:

bashCopy code
#!/bin/bash
uvx --from 'harbor-rewardkit==0.2.*' rewardkit /tests

This runs all criteria in /tests/ against the workspace at /app and writes /logs/verifier/reward.json. Defaults match Harbor's conventions — no extra config needed.

Run evolve check "<task-path>" --watch to run the verifier against the task's reference solution before publishing the task.

If judge criteria need API keys, request them through task.toml:

tomlCopy code
[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

On Evolve you never put a real key in the task: write the template exactly as above, as the whole value, and the judge's credential is supplied at run time. The rubric names the judge model, or Reward Kit's own default applies. A job can override the judge for every verifier with evolve run --ve REWARDKIT_JUDGE=<judge> --ve REWARDKIT_MODEL=<model>; no other verifier env key is accepted on a job.

Ask whether Reward Kit should run in the agent's shared environment or in a separate verifier environment. Prefer a separate verifier environment when judge prompts, grading dependencies, API keys, or clean-room checks should not be available to the agent:

tomlCopy code
[environment]
network_mode = "no-network"   # Agent env baseline — offline during agent.run()

[verifier]
environment_mode = "separate"

[verifier.environment]
network_mode = "public"     # Verifier env baseline — LLM judge API calls
docker_image = "python:3.12-slim"

In shared mode, the verifier runs in the agent container and inherits [environment].network_mode. A [verifier].network_mode that differs from that baseline is refused at import on Evolve, because a shared verify cannot switch egress. If agent and verifier need different network access, use environment_mode = "separate" and set [verifier.environment].network_mode.

Judge criteria that call external APIs need a public baseline or allowlist on the verifier environment. Programmatic checks that only read local files can use no-network.

In separate mode with no [verifier.environment].docker_image, tests/ is the verifier image's build context and its tests/Dockerfile must provide /tests/test.sh. When the verifier pins the task's own image, the platform uploads tests/ to /tests instead. When it pins a distinct image, that image boots as it is with nothing uploaded, so it must carry /tests/test.sh itself.

Programmatic criteria

Call built-ins from any .py file in tests/:

pythonCopy code
import rewardkit as rk

rk.file_exists("output.txt")
rk.file_contains("output.txt", "hello")
rk.command_succeeds("python main.py", weight=2.0)
rk.json_key_equals("result.json", "status", "ok")

All criteria accept weight (default 1.0) and isolated (default False, runs in overlayfs so side effects don't leak).

Available built-ins

• Files: file_exists, file_not_exists, file_contains, file_contains_regex, file_matches, files_equal, diff_ratio
• Commands: command_succeeds, command_output_contains, command_output_matches, command_output_matches_regex (30s default timeout, optional cwd)
• Data: json_key_equals, json_path_equals, csv_cell_equals, xlsx_cell_equals (needs [office] extra), sqlite_query_equals
• HTTP: http_status_equals, http_response_contains
• Images: image_similarity, image_size_equals (needs [image] extra)
• Trajectory: trajectory_tool_used, trajectory_tool_not_used, trajectory_turn_count

For extras, install with uv tool install harbor-rewardkit[all].

Custom criteria

Use the @criterion decorator. First parameter is always workspace: Path. Returns bool or float:

pythonCopy code
from pathlib import Path
from rewardkit import criterion

@criterion
def has_valid_output(workspace: Path) -> bool:
    return (workspace / "output.txt").read_text().strip() != ""

Zero-parameter criteria auto-register. Criteria with extra args must be called via rk:

pythonCopy code
@criterion(description="output has at least {n} lines")
def has_n_lines(workspace: Path, n: int) -> bool:
    return len((workspace / "output.txt").read_text().splitlines()) >= n

rk.has_n_lines(10, weight=2.0)
rk.has_n_lines(50, weight=1.0)

For criteria shared across reward subdirs, define with shared=True in a root-level file and call from subdirs.

Judge criteria (LLM or agent-as-a-judge)

For subjective checks (quality, readability, edge cases), create a TOML file:

tomlCopy code
[judge]
judge = "anthropic/claude-sonnet-5"   # LiteLLM model string
files = ["/app/main.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

[[criterion]]
description = "How readable is the code?"
type = "likert"
points = 5
weight = 2.0

Criterion types:

• binary — yes/no → 1.0 or 0.0
• likert — 1..points, normalized to [0, 1]
• numeric — min..max, normalized to [0, 1]

Agent judges

Agent judges shell out to a CLI and can explore the filesystem:

tomlCopy code
[judge]
judge = "claude-code"
model = "anthropic/claude-sonnet-5"
isolated = true

[[criterion]]
description = "Does the solution handle edge cases?"
type = "binary"

Slower and more expensive than LLM judges, but they can run commands and inspect files.

Useful [judge] options

timeout (default 300), reasoning_effort (low|medium|high), reference (path to reference solution), atif-trajectory (evaluate the agent's trajectory), weight, prompt_template (custom prompt with {criteria} placeholder).

Scoring aggregation (within one judge TOML)

tomlCopy code
[scoring]
aggregation = "all-pass"   # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
threshold = 0.7             # only for threshold

Only affects how this file's own criteria combine. To aggregate across dimensions, see Aggregating dimensions.

Scoring config for programmatic files

Each .py file that registers criteria is an equal-weighted scoring component named after its filename stem. Files that only provide imports or shared criterion factories and register no checks are ignored. To change how criteria within a file combine, use [scoring.<stem>] in the same directory's reward.toml:

tomlCopy code
# tests/structure/reward.toml
[scoring.files_exist]       # configures files_exist.py
aggregation = "all-pass"

[scoring.behavior]          # configures behavior.py
aggregation = "threshold"
threshold = 0.75

Each entry takes the same aggregation values as a judge TOML. Unknown keys and stems that do not resolve to a criterion-bearing Python file raise.

Directories may be nested recursively. A non-root directory can aggregate its local Python files, local judges, and immediate child directories with one unnamed [[reward]] table:

tomlCopy code
# tests/correctness/reward.toml
[[reward]]
aggregation = "weighted-mean"
weights = { files = 2.0, behavior = 1.0 }

Membership is implicit. Child directories have weight 1.0 unless overridden; use filename stems for local Python files and judge TOMLs, and directory names for child groups. Without [[reward]], the directory defaults to weighted mean.

Multi-reward tasks

Put criteria in subdirectories — each becomes a separate reward:

Copy code
tests/
├── test.sh
├── correctness/
│   └── check.py
├── structure/
│   └── files_exist.py
└── quality/
    └── quality.toml

Judge TOMLs may also sit directly at the tests root alongside reward subdirectories. Each is exposed as a top-level reward named after its filename stem and can be referenced by a root aggregation.

Criterion-bearing Python files at the tests root are also top-level dimensions named after their stems. Root support files that register no criteria are ignored.

Produces:

jsonCopy code
{ "correctness": 0.75, "structure": 1.0, "quality": 0.6 }

Aggregating dimensions

To add aggregated scores on top of the per-dimension keys, add a root-level tests/reward.toml with one or more [[reward]] tables. Each adds one key to reward.json, aggregating the dimensions with the same modes as [scoring]:

tomlCopy code
# tests/reward.toml
[[reward]]
name = "reward"
aggregation = "all-pass"   # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
# threshold = 0.7          # only for threshold
weights = { correctness = 2.0, quality = 1.0 }

jsonCopy code
{ "correctness": 0.75, "structure": 1.0, "quality": 0.6, "reward": 0.0 }

The per-dimension scores stay; aggregated keys are added alongside them (a name may not collide with a dimension). Top-level dimensions have equal weight unless that aggregation's inline map overrides them; reward-details.json keeps the full recursive breakdown.

Output files

• /logs/verifier/reward.json — per-reward scores
• /logs/verifier/reward-details.json — per-criterion results, judge reasoning, errors

Multi-step tasks

In a multi-step task, each step has its own tests/ under steps/{name}/tests/, and the verifier runs once per step. Reward Kit behaves the same as in a single-step task: for each step it reads /tests, runs the criteria against /app, and writes /logs/verifier/reward.json for that step. The platform then aggregates per-step results into a trial-level reward via multi_step_reward_strategy in task.toml — aggregation happens outside Reward Kit, so don't try to encode cross-step logic in your criteria.

A task-level tests/ directory (at the task root) is uploaded to /tests first, then the step's own tests/ is layered on top (same-name files win). Put shared helpers (common checks.py functions with shared=True, fixture files, a fallback test.sh) at the task level, and step-specific criteria under each step.

Multi-reward subdirectories still work within a step: steps/foo/tests/ can contain correctness/, structure/, quality/ — each produces a separate reward key for that step, and multi_step_reward_strategy = "mean" averages each key across steps. Use "final" when the last step is an end-to-end check whose rewards already represent the full task.

When to reach for what

• Use built-ins for file existence, string matches, command output, JSON/CSV checks, HTTP probes.
• Use @criterion when logic is task-specific but still programmatic.
• Use LLM judges for subjective quality dimensions (readability, correctness of prose).
• Use agent judges when the rubric requires exploring the filesystem or running code (e.g. "does the test suite actually pass?").
• Use subdirectories when you want separate scores (correctness vs structure vs quality) rather than one blended number.
• Use isolated=True for any criterion that runs mutating commands, so it doesn't corrupt the workspace for other criteria.

Working example

See https://github.com/laude-institute/harbor/tree/main/examples/tasks/reward-kit-example. Reward Kit's own documentation: https://docs.harborframework.com/core-concepts/rewardkit/quick-start.