Eval Skills

Skill Evaluator

Discover all skills in the project, score them across 6 criteria, and infer the appropriate effort level based on content analysis.

When to Use

• New project: run once to establish baseline quality
• Before committing a skill to a team repo
• After bulk-importing skills from another project
• When adding effort fields for the first time
• When a skill doesn't auto-trigger and you want to diagnose why

What Gets Audited

All SKILL.md files and flat .md files found in:

• .claude/skills/**
• ~/.claude/skills/** (if requested)
• .claude/commands/** (legacy flat files, still valid)
• Any path passed as argument: /eval-skills ./my-skills-dir

Valid Frontmatter Fields

Claude Code skills follow the agentskills.io open standard, extended with Claude Code-specific fields. Flag any field not in this table as unsupported.

agentskills.io spec fields

Claude Code extension fields

Unsupported fields (flag and remove)

tags, category, keywords, usage, args are ignored by the runtime. Remove them to avoid confusion.

String Substitutions (check for correct usage)

If a skill references substitution placeholders, verify they use the correct syntax:

Incorrect placeholders ($1 when no arguments: field is declared, ${ARGS}, %ARGUMENTS%) are passed as literal text and silently break the skill.

Scoring Criteria (16 pts per skill)

Thresholds:

• ✅ Good: >=13/16 (>=80%)
• ⚠️ Needs work: 10-12/16 (60-79%)
• ❌ Fix: <10/16 (<60%)

allowed-tools format note

Both formats are valid and parse correctly:

• Space-separated string: Read Bash Grep
• YAML list: [Read, Bash, Grep]

Flag skills that use Bash globally (without path scoping like Bash(git *)) when they only need read operations (this grants unnecessary write access).

description length note

The combined description + when_to_use text is truncated at 1,536 chars per skill in Claude's context. Flag skills where the combined text exceeds this limit; the tail (often the trigger examples) gets silently cut.

Effort Level Inference Engine

For each skill, analyze description + content and classify using these signals:

low: Mechanical execution, no design decisions

Signals:

• Verbs: commit, push, sync, scaffold, generate (template-based), format, rename, bump, wrap, convert
• No reasoning required: sequential steps, template instantiation, data fetching
• allowed-tools: Bash only, or Read only
• No sub-agents spawned
• Short workflow (<5 steps)

Examples: /commit, /release-notes, /scaffold, /sync, /format

medium: Analysis with bounded scope, categorization

Signals:

• Verbs: review, triage, analyze, categorize, suggest, evaluate (single file or bounded scope)
• Requires pattern recognition but not architectural reasoning
• allowed-tools: Read + Grep + Bash combination
• May spawn 1-2 sub-agents with predefined scope
• Produces structured output (tables, categorized lists)

Examples: /code-review (single PR), /issue-triage, /dependency-audit, /test-coverage

high: Design decisions, adversarial reasoning, cross-system analysis

Signals:

• Verbs: architect, redesign, threat-model, audit (security), orchestrate (multi-agent), score, assess trade-offs
• Requires reasoning about edge cases, attack vectors, or system-wide implications
• allowed-tools: broad access (Read + Write + Bash + external tools)
• Spawns multiple sub-agents or uses parallel execution
• Produces analysis with explicit uncertainty or trade-off sections
• Keywords: "security", "architecture", "adversarial", "pipeline", "threat", "design decision"

Examples: /security-audit, /architecture-review, /eval-agents

xhigh / max: Exhaustive, multi-agent, long-running

Signals:

• Explicit ultrathink or ultracode in skill body
• Spawns many parallel agents (3+)
• Uses Workflow tool or fan-out orchestration patterns
• Expected runtime >5 minutes
• Deep synthesis across entire codebase or multi-repo scope

Examples: /deep-research, /full-security-audit, /codebase-migration

Mismatch flag

If a skill has effort: already set but the inferred level differs, flag it:

⚠️ Effort mismatch: declared low, inferred high. Skill spawns 4 sub-agents and performs security analysis.

Execution Instructions

Step 1: Discovery

bashCopy code
# Find all SKILL.md files
find .claude/skills -name "SKILL.md" 2>/dev/null

# Find legacy flat command files
find .claude/commands -maxdepth 1 -name "*.md" ! -name "README*" 2>/dev/null

# If argument provided, use that path instead

Step 2: Parse each skill

For each skill file found:

1. Read the full file
2. Extract YAML frontmatter (between first --- and second ---)
3. Parse all recognized fields from the table above
4. Flag any unrecognized fields as potentially unsupported
5. Note presence/absence of each field
6. Read the body content for structure analysis and substitution placeholder correctness

Step 3: Score and infer

Apply the scoring criteria to each skill:

• Check frontmatter fields against the valid fields table
• Evaluate description quality: does it answer "when to use"? Is combined description + when_to_use under 1,536 chars?
• Evaluate allowed-tools scope: is Bash used without path scoping when read-only tools suffice?
• Check context: fork skills for completeness; they must have an actionable task body and an agent field (or a reasonable default applies)
• Infer effort level from content analysis
• Compare inferred vs declared effort (if set)
• Evaluate content structure

Step 4: Output

Produce a structured report:

Copy code
# Skills Audit: [project name or path]
Date: [today] | Scanned: N skills

## Summary
| Status | Count |
|--------|-------|
| ✅ Good (>=80%) | N |
| ⚠️ Needs work (60-79%) | N |
| ❌ Fix (<60%) | N |

**Effort coverage**: N/N skills have effort field set

---

## Per-Skill Results

### [skill-name] ([score]/16) [✅/⚠️/❌]

| Criterion | Score | Notes |
|-----------|-------|-------|
| name | ✅ 1/1 | ok |
| description | ⚠️ 2/3 | Missing when_to_use field |
| allowed-tools | ✅ 2/2 | Well-scoped |
| effort | ❌ 0/3 | Missing. Recommended: high |
| content structure | ⚠️ 3/5 | No examples section, unsupported field "tags" found |

**Effort inference**: `high`. Skill performs security analysis with adversarial reasoning.
  Signals: "threat", "attack surface", "vulnerability scoring" in content; spawns 4 agents

**Priority fixes** (ordered by impact):
1. Add `effort: high` to frontmatter
2. Add `when_to_use` with trigger phrases
3. Remove unsupported field `tags`
4. Add a concrete usage example section

---

After all skills: print a Fix Summary.

Fix Summary Format

At the end, print a ready-to-use patch block for all missing/mismatched effort fields:

Copy code
## Recommended effort fields (copy-paste ready)

skill-name-1: effort: low     # mechanical scaffold
skill-name-2: effort: high    # security analysis, spawns agents
skill-name-3: effort: medium  # code review, bounded scope

And a 1-line count: N skills need effort field · N mismatches · N missing allowed-tools · N unsupported fields to remove