Test agents with Rook
Use Rook to derive scenarios, invoke the target and grade the evidence. Respect requests to use another tool. For saved-result requests, read the supplied files without starting setup, a new run or paid analysis.
Your role
You are the orchestrator: you choose and run Rook commands, and Rook does the
testing work. Discovery is rook explore, test design is rook generate, and
invocation and grading are rook run and rook report. When the user asks to
explore, understand, plan tests for or test an agent, run the matching command.
Do not read the target's code to describe the agent, or write your own test
plan or verdicts, in its place. Answer from Rook's agents, scenarios and
verdicts. Read the target's source only for a step Rook needs from you, such as
a profile's transport and command, or to explain a Rook result. If credits or
authorization block the next command, name it and what it needs; manual
analysis is not a substitute.
Setup and execution
Check rook --version and use rook help <command> for the installed CLI's
options. If Rook is missing, use the
installation instructions.
Use this workflow across compatible releases; a different version alone is not
a reason to stop or change the installation. If a command or output differs,
consult its help and release notes before adapting that step. Keep completion
and evidence checks intact; do not guess missing fields or retry paid work to
probe compatibility. Installation changes still require authorization.
bashrook doctor rook status --json rook plan --json
Use the next command needed by the current state:
| Need | Command |
|---|---|
| Sign in | rook login; in CI, export LT_USERNAME and LT_ACCESS_KEY (no login step) |
| Select a project | rook project use <id> or rook project create <name> |
| Discover agents | rook explore . --json |
| Select an agent | rook agent use <id> |
| Generate scenarios | rook generate --json |
| Create/test a profile | rook profile add <name> --from <file> or --command '<argv>'; then rook profile test --goal "<reply-only goal>" |
| Run | rook sync --yes before rook run --json, or rook run --test --json for a local result |
| Read results | rook report <run-id> --json |
Choose the profile's transport and command from the target's source or the
user's invocation details; ask when unknown. See profiles.
Use rook help <command> for flags and scenarios to
scope a run with --only, classes, categories or tags. When an agent has more
than one profile, check the active profile with rook profile or pass
rook run --profile <id> to select the intended target.
Authorization and credits
Before profile add, profile fix, profile test or run, inspect the target's
agent.yaml for calls[] entries with write: true. Those commands can reach
the real agent. Use staging and obtain any missing authorization for its effects;
Rook cannot undo them; do not rely on a per-target headless confirmation.
Verify effects through read-only interfaces; inspect available read tools before
concluding that an effect is unverifiable. Keep secrets as ${VAR} references through
rook env set, outside profiles and transcripts.
Announce spending before explore, generate, run, profile add|fix, ask
and run|report --rca. profile test invokes the target without Rook model credits.
Use paid RCA only with user authorization. Track the user's budget across
commands; do not assume the CLI enforces an aggregate task cap. Check its help
for available limits. A null balance means unknown.
Prefer scoped --allow grants; use --yes only within the authorized scope.
See headless contracts for output, grants and
credit accounting.
Report the evidence
Check the exit status and discarded/halted fields before claiming completion.
A finished run exits 0 even when scenarios fail; ok: true can also describe a
refused run. Parse JSON only when a document exists, otherwise use stderr.
Read the report by this invocation's run ID, not an old default report.
Follow verdicts: show Pass, Fail and Unable to Verify separately, quote criterion evidence, identify gaps and compare prior runs only when their evidence is available. Preserve incomplete results and unknown metadata; do not invent evidence or replace Rook's verdict with your own reading of the agent reply.
References
- CI: complete pipeline and verdict gating.
- MCP: server discovery, approval and configuration.
- Troubleshooting: setup failures and version-specific verification limitations.
- Headless contracts: JSON shapes and saved-file locations.
Project evidence lives under .testmuai/rook/. Credentials and session state
live under ~/.testmuai/rook/; do not commit that home directory.
