Skill Trigger Tester
What it does
OpenClaw maps user intent to skills by matching the user's message against each skill's description: field. A weak description means your skill silently never fires. A description that's too broad means it fires when it shouldn't.
Skill Trigger Tester helps you validate the trigger quality of a skill's description before publishing. You give it:
- The description string you're testing
- A set of "should fire" prompts (true positives)
- A set of "should not fire" prompts (true negatives)
It scores precision, recall, and gives an overall trigger quality grade (A–F) plus actionable suggestions.
When to invoke
- Before publishing any new skill to ClawHub
- When a skill you expect to trigger isn't firing
- When a skill keeps firing on irrelevant prompts
- Inside
create-skillworkflow (Step 5: validation)
Scoring model
The tool uses a keyword + semantic overlap heuristic against the description field:
| Metric | Meaning |
|---|---|
| Recall | % of "should fire" prompts that would match |
| Precision | % of matches that are actually "should fire" |
| F1 | Harmonic mean of recall and precision |
Grade thresholds:
| Grade | F1 |
|---|---|
| A | ≥ 0.85 |
| B | ≥ 0.70 |
| C | ≥ 0.55 |
| D | ≥ 0.40 |
| F | < 0.40 |
How to use
bashpython3 test.py --description "Diagnoses skill discovery failures" \ --should-fire "why isn't my skill loading" \ "my skill disappeared from the registry" \ "check if my skills are healthy" \ --should-not-fire "write a skill" \ "install superpowers" \ "review my code" python3 test.py --file skill-spec.yaml # Load test cases from YAML file python3 test.py --format json # Machine-readable output
Test spec file format
yamldescription: "Diagnoses skill discovery failures — YAML parse errors, path violations" should_fire: - "why isn't my skill loading" - "my skill disappeared" - "check skill health" should_not_fire: - "write a new skill" - "install openclaw"
Procedure
Step 1 — Write your test cases
For each skill you're testing, list 3–5 prompts that should trigger it and 3–5 that should not. Be honest about edge cases.
Step 2 — Run the scorer
bashpython3 test.py --description "<your description>" \ --should-fire "..." --should-not-fire "..."
Step 3 — Interpret results
- Grade A/B: description is well-calibrated. Publish.
- Grade C: borderline — add more specific keywords to the description, or narrow the wording.
- Grade D/F: description is too vague or uses jargon the user won't say. Rewrite and retest.
Step 4 — Iterate
Try alternative descriptions and compare scores side-by-side using --compare.
Step 5 — Add test file to the skill directory
Commit the trigger-tests.yaml spec alongside the skill. Future contributors can run it to verify trigger quality hasn't regressed.
Common mistakes
- Too generic:
"Helps with skills"— will either never fire or fire on everything - Technical jargon:
"Validates SKILL.md frontmatter schema coherence"— users don't say this - Action + object only:
"Creates skills"— add when/why context - Missing synonyms: If users might say "check" or "verify" or "test", the description needs to capture the semantic range
Output example
Skill Trigger Quality — skill-doctor ───────────────────────────────────────────── Description: "Diagnoses silent skill discovery failures..." Should fire (5 prompts): 4 / 5 matched recall = 0.80 Should not fire (5 prompts): 1 / 5 matched precision = 0.80 F1 score: 0.80 Grade: B ⚠ 1 false negative: "check if skills are healthy" Suggestion: Add "healthy", "health", or "check" to the description. ⚠ 1 false positive: "check my code" Suggestion: Narrow description to avoid generic "check" overlap.
