Waza
"The way of technique — measure, refine, master."
A Go CLI tool for evaluating AI agent skills through structured benchmarks. Define test cases in YAML, run them against agent engines, and validate results with pluggable scoring validators.
Help
When user says "waza help" or asks how to use waza:
╔══════════════════════════════════════════════════════════════════╗ ║ WAZA - CLI Tool for Evaluating Agent Skills ║ ╠══════════════════════════════════════════════════════════════════╣ ║ ║ ║ COMMANDS: ║ ║ waza run <eval.yaml> # Run an evaluation benchmark ║ ║ waza init [directory] # Initialize a new eval suite ║ ║ waza generate <SKILL.md> # Generate eval from SKILL.md ║ ║ waza compare <r1> <r2> ... # Compare result files ║ ║ waza dev [skill-path] # Improve SKILL.md compliance ║ ║ ║ ║ RUN FLAGS: ║ ║ --context-dir, -c Fixtures directory (default: ./fixtures) ║ ║ --output, -o Save results JSON to file ║ ║ --verbose, -v Verbose output ║ ║ --task, -t Filter tasks by name (repeatable) ║ ║ --parallel, -p Run tasks in parallel ║ ║ --workers, -w Number of parallel workers ║ ║ --transcript-dir Save per-task transcripts ║ ║ ║ ║ COMPARE FLAGS: ║ ║ --format, -f Output format: table or json ║ ║ ║ ║ GENERATE FLAGS: ║ ║ --output-dir, -d Output directory for generated files ║ ║ ║ ║ DEV FLAGS: ║ ║ --target Adherence level: low|medium|high ║ ║ --max-iterations Max improvement iterations (default: 5) ║ ║ --auto Auto-apply without prompting ║ ║ ║ ║ WORKFLOW: ║ ║ 1. waza init my-eval # Scaffold eval suite ║ ║ 2. Edit eval.yaml + tasks # Define test cases ║ ║ 3. waza run eval.yaml -v # Execute benchmark ║ ║ 4. waza compare a.json b.json # Cross-model comparison ║ ║ ║ ║ FIXTURE ISOLATION: ║ ║ Each task gets a fresh temp workspace with fixtures copied ║ ║ in. Original fixtures are never modified. ║ ║ ║ ╚══════════════════════════════════════════════════════════════════╝
Commands
waza run
Run an evaluation benchmark from a YAML spec file.
bash# Run with default mock engine waza run path/to/eval.yaml --context-dir path/to/fixtures # Verbose output with results saved waza run eval.yaml -c ./fixtures -v -o results.json # Filter to specific tasks waza run eval.yaml -t "task-name-1" -t "task-name-2" # Parallel execution waza run eval.yaml --parallel --workers 4 # Save per-task transcripts waza run eval.yaml --transcript-dir ./transcripts
waza init
Initialize a new evaluation suite with a compliant directory structure.
bash# Initialize in current directory waza init # Initialize in a named directory waza init my-eval-suite
Creates: eval.yaml, tasks/ with example task, fixtures/ with example fixture.
waza generate
Generate an eval suite from an existing SKILL.md file.
bash# Generate eval from SKILL.md waza generate path/to/SKILL.md # Specify output directory waza generate SKILL.md --output-dir ./my-eval
Parses YAML frontmatter (name, description) and creates eval.yaml, starter tasks, and fixtures.
waza compare
Compare results from multiple evaluation runs side by side.
bash# Compare two result files waza compare run1.json run2.json # Compare three or more waza compare gpt4.json claude.json gemini.json # JSON output waza compare run1.json run2.json --format json
Shows per-task score deltas, pass rate differences, and aggregate statistics.
waza dev
Iteratively improve SKILL.md frontmatter compliance with automated scoring.
bash# Score current skill and suggest improvements waza dev skills/my-skill # Target high compliance level waza dev skills/my-skill --target high # Auto-apply improvements without prompts waza dev skills/my-skill --target medium --auto --max-iterations 3
Compliance Levels:
- Low (< 150 chars or no triggers) — Minimal description
- Medium (150+ chars, has triggers) — Basic trigger coverage
- Medium-High (+ anti-triggers) — Routing clarity improved
- High (+ routing markers like INVOKES/FOR SINGLE OPERATIONS) — Full compliance
Scoring Checks:
- Description length (150+ chars required, 1024 max)
- Trigger phrases (USE FOR: patterns)
- Anti-trigger phrases (DO NOT USE FOR: patterns)
- Routing clarity markers (WORKFLOW SKILL, INVOKES:, etc.)
- Token budget (500 soft limit, 5000 hard limit)
Coming Soon: Trigger accuracy tests (#36), --skip-integration (#37), --fast (#38), improvement suggestions engine (#34).
Evaluation Spec Format
yamlname: my-eval skill: my-skill version: "1.0" executor: mock # or copilot-sdk tasks: - id: task-1 name: "Describe the task" prompt: "Your prompt to the agent" expected: "Expected behavior" validators: - type: code config: language: go - type: text config: pattern: "expected pattern"
Engines
| Engine | Use | Description |
|---|---|---|
mock | Testing | Returns canned responses for validator development |
copilot-sdk | Production | Executes via Copilot CLI SDK |
Validators
| Validator | What it checks |
|---|---|
code | Code compiles / passes syntax check |
regex | Output matches regex pattern |
Configuration
| Setting | Flag | Default |
|---|---|---|
| Fixtures dir | --context-dir | ./fixtures |
| Output file | --output | (none) |
| Verbose | --verbose | false |
| Parallel | --parallel | false |
| Workers | --workers | CPU count |
| Transcript dir | --transcript-dir | (none) |
Scoring Quick Reference
Each task produces an EvaluationOutcome with:
| Field | Description |
|---|---|
score | 0.0–1.0 normalized score |
pass | Boolean pass/fail |
validator_results | Per-validator details |
duration | Execution time |
