Evals Analyze

evals-analyze

What this skill does

Provides cross-functional team elevation and closed-loop feedback following EDD Principle VIII (Close the Production Loop) by deep-analyzing trajectory failure traces and routing them to correct resolution pathways.

Output:

1. Trajectory Analysis - Full multi-turn trace analysis with tool calls and context preservation (EDD Principle V)
2. Failure Routing:
   • Specification Failures (agent logic missing/ambiguous) → subclassified (EVAL-010):
     • Mechanical (fixed, checkable pattern) → fix/extend the existing grader or add a unit test — a deterministic check, not a CDR
     • Judgment gap (missing intent, ambiguity, needs context) → automatically triggers a local call to team-levelup to propose new context rules in adlc branch drafts/cdr/ to fix agent behavior
   • Generalization Failures (grader flawed or lacks edge-case coverage) → Appends evaluator backlog items to the project backlog for ongoing monitoring.
3. Cross-Functional PR - Creates a team-ai-directives PR with insights and rule updates (EDD Principle X)

Key EDD Principles Applied:

• Principle VIII: Close Production Loop - Spec failures → fix directives; Gen failures → evaluator backlog
• Principle V: Trajectory Observability - Full multi-turn traces, not just outputs
• Principle X: Cross-Functional Observability - PMs, domain experts, and AI engineers collaborate

When to use

• After /evals-validate: Analyze failures and resolve them
• Closing a development loop: Translate evaluation failure insights into rule or evaluator fixes
• Reporting to stakeholders: Generate readable summaries for PMs and domain experts

When NOT to use

• Evals not yet executed: Run /evals-validate first to generate results in evals/results/
• Trivial tasks: Closed-loop analysis is overhead for simple features

Process

User Input

textCopy code
$ARGUMENTS

• --focus AREA — Focus analysis on specific areas (e.g., security, quality, performance)
• --dry-run — Analyze results and print report, but skip PR creation and local skill triggers

Execution Steps

Phase 1: Load Evaluation Results

• Reads results JSON from evals/results/.
• Extracts failure cases and full multi-turn conversation traces (including tool calls).

Phase 2: Failure Classification

Categorizes each failure trace:

• Specification Failure: The agent was correct relative to its context, but the rule/directive was missing, ambiguous, or incorrect. Subclassify (EVAL-010):
  • Mechanical: the gap is a fixed, checkable pattern (banned API, import shape, file-location, syntactic shape) → deterministic check territory
  • Judgment gap: the gap needs intent, context, or cross-file judgement → context rule territory
• Generalization Failure: The rule was correct, but the agent made a mistake anyway (hallucinated, missed a constraint, or grader lacked edge-case coverage).

Phase 3: Action Routing (Close the Loop)

• For Mechanical Specification Failures: Fix/extend the existing binary grader or add a unit test that enforces the pattern. Do NOT propose a CDR for a mechanically-checkable gap — pay once for the check instead of re-deriving it per session.
• For Judgment-gap Specification Failures: Automatically triggers local skill /team-levelup with the failure trace as input. This creates new rule/persona/example CDRs in adlc branch drafts/cdr/ to fix the agent's behavior.
• For Generalization Failures: Appends an evaluator backlog item to evals/results/evaluator_backlog.md detailing the needed grader edge-case updates.

Phase 4: Cross-Functional Insights & PR

• Generates a stakeholder-specific report in evals/results/team_insights.md (tailored for PMs, domain experts, and AI engineers).
• If git remote and gh CLI are available, commits rule/eval changes in team-ai-directives and opens a draft PR (uses team-levelup logic under the hood).

Verification

• Trajectory failure traces analyzed and classified
• Mechanical specification failures routed to grader/unit-test fixes (deterministic checks); judgment-gap failures routed to /team-levelup (proposes CDRs in adlc branch drafts/cdr/)
• Generalization failures written to evals/results/evaluator_backlog.md
• Stakeholder report evals/results/team_insights.md generated
• Draft PR created in team-ai-directives (if applicable)
• Final report summary presented with PR link and backlog details