Benchmark

Benchmark — SWE-bench Lite

You are a benchmark guide. Help the user run, configure, and evaluate SWE-bench Lite benchmarks against agtx agent workflows.

Answer questions, surface the right commands, and walk through setup interactively. All commands assume the user is in the benchmark/ directory.

Prerequisites

One-Time Setup

bashCopy code
cd benchmark/swebench

# Initialize Python environment (once, or after pyproject.toml changes)
uv sync

# [Sandbox only] Build the tools image (tmux + Node.js + Claude Code)
python prebake_images.py --verbose

# [Sandbox only] Build the Linux agtx binary (Ubuntu 22.04 / glibc 2.35)
bash build_linux_binary.sh

The tools image populates the shared Docker volume agtx-swebench-tools on the first benchmark run. To force a refresh after updating Claude Code:

bashCopy code
docker volume rm agtx-swebench-tools
python prebake_images.py --force --verbose

Configuration

Config files live in swebench/configs/. Each is a standard agtx ProjectConfig TOML written to .agtx/config.toml in every cloned repo.

Minimal (no workflow):

tomlCopy code
default_agent = "claude"
workflow_plugin = "void"

Standard agtx workflow:

tomlCopy code
default_agent = "claude"
workflow_plugin = "agtx"
worktree_dir = ".agtx/worktrees"

Sandbox-optimised (agent works directly in /testbed, no worktree):

tomlCopy code
default_agent = "claude"
workflow_plugin = "agtx"
worktree_dir = ".agtx/worktrees"
skip_worktree = true

Mixed agents (different agent per phase):

tomlCopy code
default_agent = "claude"
workflow_plugin = "agtx"

[agents]
planning = "gemini"
running  = "claude"
review   = "codex"

With sandbox_init (install extra tooling inside the container before the agent starts):

tomlCopy code
default_agent = "claude"
workflow_plugin = "agtx"
skip_worktree = true

sandbox_init = [
    "curl -fsSL https://raw.githubusercontent.com/rtk-ai/rtk/refs/heads/master/install.sh | sh",
    "export PATH=$HOME/.local/bin:$PATH && rtk init -g",
]

sandbox_init commands run with HOME=/home/bench and PATH including /home/bench/.local/bin. Each config activates only the tools it explicitly installs — other configs are unaffected.

Available plugins: void, agtx, agtx-terse, gsd, spec-kit, bmad, openspec, superpowers, agent-skills

Pre-built configs for common combinations are in swebench/configs/.

Running

Sandbox mode requires a Linux x86_64 binary. Use --agtx ../target/agtx-linux-x86_64 (not ../target/release/agtx).

Always recommend sandbox mode. SWE-bench repos require specific Python versions and C extensions that aren't available on the host — agents typically fail to run pytest outside the containers. In sandbox mode each task runs inside its official SWE-bench Docker image with the repo pre-installed in a working conda env (testbed).

Single random task:

bashCopy code
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-void.toml \
  --instances 1 --verbose \
  --agtx ../target/release/agtx

Specific instance IDs:

bashCopy code
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-void.toml \
  --instance-ids sympy__sympy-20590 django__django-11099 \
  --agtx ../target/release/agtx

Sandbox mode (recommended):

bashCopy code
python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --instance-ids astropy__astropy-12907 \
  --sandbox --verbose \
  --agtx ../target/agtx-linux-x86_64

Parallel tasks:

bashCopy code
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --concurrency 4 \
  --agtx ../target/release/agtx

Full 300-task run:

bashCopy code
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --agtx ../target/release/agtx

Resume an interrupted run (pass the same --output-dir):

bashCopy code
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --output-dir swebench_output/agtx_claude_20260427_120000 \
  --agtx ../target/release/agtx

Hard mode (prose only — no code blocks or stack traces in the problem statement):

bashCopy code
uv run --project swebench \
  python swebench/benchmark.py \
  --config swebench/configs/claude-agtx.toml \
  --hard \
  --agtx ../target/release/agtx

All CLI Options

Observing a Running Benchmark

Attach to a running container:

bashCopy code
docker exec -it swebench-astropy-astropy-12907 tmux -L agtx attach -t testbed:1
# Ctrl+b 0 → agtx board   Ctrl+b 1 → agent session   Ctrl+b d → detach

Output

Results are written to ./swebench_output/{config-name}_{timestamp}/.

predictions.jsonl — SWE-bench format, one line per task:

jsonCopy code
{"instance_id": "sympy__sympy-20590", "model_name_or_path": "agtx-agtx-claude", "model_patch": "diff --git ..."}

results.json — detailed results with timing and cost:

jsonCopy code
[{
  "instance_id": "sympy__sympy-20590",
  "status": "success",
  "duration_seconds": 342.1,
  "cost_usd": 0.23,
  "cost_tokens": 54000,
  "model_patch": "diff --git ...",
  "error": null
}]

Status values: success, timeout, error, setup_error

Quick check:

bashCopy code
cat swebench_output/*/results.json | \
  python3 -c "import json,sys; r=json.load(sys.stdin); print(f'{sum(1 for x in r if x[\"status\"]==\"success\")}/{len(r)} success')"

Capturing the Agent Session Transcript

predictions.jsonl / results.json capture only the git diff and metrics — NOT what the agent said or did. To verify behavior (e.g. whether an injected skill/plugin actually changed the agent's output), read the Claude Code session transcript.

Where Claude Code writes it: ~/.claude/projects/{encoded-cwd}/{session-id}.jsonl, where {encoded-cwd} is the working directory with every / and . replaced by -. Each line is one event (type = user/assistant/…) with full message content, tool uses, and a cwd field.

In sandbox mode the transcript lives inside the container (agent runs as bench, HOME=/home/bench, cwd /testbed when skip_worktree = true) and is destroyed when the container stops:

Copy code
/home/bench/.claude/projects/-testbed/{session-id}.jsonl

Containers are removed at task end, and a single docker cp "before the run finishes" is unreliable — the container often stops first. Run a polling loop that re-copies every few seconds while the container is alive (start it right after launching the benchmark):

bashCopy code
C=swebench-astropy-astropy-12907
DEST=/tmp/transcript_capture
for i in $(seq 1 180); do
  docker ps --filter name=$C -q | grep -q . || { echo "container gone"; break; }
  docker cp $C:/home/bench/.claude/projects "$DEST" 2>/dev/null
  sleep 5
done
find "$DEST" -name '*.jsonl'

With skip_worktree = true, each agtx phase (Planning / Running / Review) is a separate Claude Code session → one .jsonl per phase. void runs as a single session.

Analysing the transcript — do NOT use a naive grep. SessionStart-hook plugins (focus, ponytail, caveman) inject their whole SKILL.md into the transcript as an attachment event, and that injected text documents the very tags you'd grep for. So grep -c 'context-commit' *.jsonl counts the injected instructions, not model output. In one run a raw grep reported 34 CSP-tag hits while the model emitted zero — all 34 were inside the injected skill. Parse the JSONL, keep only type == "assistant" events, exclude the injected attachment, and count tags in the text blocks:

bashCopy code
python3 - "$DEST"/**/-testbed/*.jsonl << 'PY'
import json, sys
TAGS = ["context-commit","context-scope","context-checkpoint","context-transient","context-revoke"]
for path in sys.argv[1:]:
    injected = turns = 0; counts = {}
    for line in open(path):
        try: d = json.loads(line)
        except Exception: continue
        if d.get("type") == "attachment" and "Always-on context discipline" in json.dumps(d):
            injected += 1
        if d.get("type") != "assistant": continue
        turns += 1
        content = d.get("message", {}).get("content", [])
        text = "".join(c.get("text","") for c in content
                        if isinstance(c, dict) and c.get("type") == "text") \
               if isinstance(content, list) else (content if isinstance(content, str) else "")
        for t in TAGS:
            n = text.count("<" + t)
            if n: counts[t] = counts.get(t, 0) + n
    print(f"{path}\n  injected SKILL.md: {injected}  assistant turns: {turns}  tags EMITTED BY MODEL: {counts or 'NONE'}")
PY

Gotchas:

• CSP tags are HTML-style (<context-commit>) → the Claude Code TUI hides them in the rendered pane; tmux capture-pane never shows them. Read the JSONL.
• void quirk: the phase prompt is pasted but not always submitted (no artifact/finish-marker), showing as "Pane stable but no finish marker". Submit manually: docker exec $C tmux -L agtx send-keys -t testbed:1 Enter.

Evaluation

After the run, copy-paste the printed commands or run manually:

bashCopy code
uv run python -m swebench.harness.run_evaluation \
  --dataset_name princeton-nlp/SWE-bench_Lite \
  --predictions_path swebench_output/claude-agtx_20260427_120000/predictions.jsonl \
  --run_id claude-agtx-1746345600

The harness runs tests in Docker containers — each task gets a fresh repo checkout with the patch applied and tested in isolation.

Report

Print a summary table with resolved status, duration, cost, and token usage:

bashCopy code
uv run python swebench/report.py \
  --results swebench_output/claude-agtx_20260427_120000/results.json \
  --logs logs/run_evaluation/claude-agtx-1746345600/

Example output:

Copy code
┌───────────────────────┬─────────────┬──────────┬───────┬────────┐
│ Instance              │ Status      │ Duration │ Cost  │ Tokens │
├───────────────────────┼─────────────┼──────────┼───────┼────────┤
│ astropy/astropy-12907 │ ✅ resolved │ 4m 57s   │ $0.95 │ 1.9M   │
├───────────────────────┼─────────────┼──────────┼───────┼────────┤
│ astropy/astropy-14182 │ ❌ failed   │ 5m 57s   │ $1.66 │ 5.3M   │
└───────────────────────┴─────────────┴──────────┴───────┴────────┘

1/2 resolved  ·  10m 54s total  ·  $2.60 total  ·  7.3M tokens

Cleanup

Non-sandbox (stale worktrees, tmux sessions, SQLite DBs):

bashCopy code
# All instances
./swebench/cleanup.sh

# Specific instance
./swebench/cleanup.sh astropy__astropy-12907

# Custom workdir
SWEBENCH_WORKDIR=/my/custom/path ./swebench/cleanup.sh

Sandbox (containers left running after a killed process):

bashCopy code
# Specific container
docker rm -f swebench-astropy-astropy-12907

# All swebench containers
docker ps -a --filter name=swebench- -q | xargs docker rm -f

# Remove tools volume (only if you need to repopulate it)
docker volume rm agtx-swebench-tools

How It Works

Use this to diagnose stuck or slow runs.

Copy code
benchmark.py
  ├── [sandbox] Pulls SWE-bench Docker image, starts container with tools volume mounted
  ├── [sandbox] Copies credentials, wires tools (tmux/node/claude via symlinks), runs sandbox_init
  ├── [non-sandbox] Clones each repo at base_commit → /tmp/swebench_repos/{instance_id}/
  ├── Writes .agtx/config.toml into each repo/container
  ├── Starts agtx TUI per task in detached tmux (tmux -L agtx)
  ├── Spawns agtx mcp-serve as subprocess (JSON-RPC 2.0 over stdio)
  ├── Drives task via MCP:
  │     create_task → move_forward (Planning)
  │     → poll planning artifact → move_forward (Running)
  │     → poll running artifact  → move_forward (Review)
  │     → poll review artifact   → git diff HEAD...{branch} → move_to_done
  ├── Snapshots tokscale before/after running phase for token counts
  └── Appends to predictions.jsonl + rewrites results.json atomically

Phase completion detection (per phase, in priority order):

1. Artifact file — if the plugin defines an artifact for that phase (e.g. .agtx/plan.md, .agtx/execute.md, .agtx/review.md for agtx/agtx-terse), polls for its existence every 5 seconds.
2. Claude finish marker — detects ✻ [Word] for Xs followed by a ❯ prompt in the pane; confirmed within 5 seconds of the marker appearing.
3. Pane stability — fallback for plugins without artifacts and no finish marker: pane content identical for 2 consecutive 5-second checks (10s stable). If stability is reached without a finish marker, a warning is emitted — the agent may be stuck, errored, or waiting for manual approval.

When troubleshooting a phase that won't advance: attach to the container's tmux (see Observing a Running Benchmark), check whether the expected artifact file exists, and look for the finish marker or a pending approval prompt in the agent pane.