Agents Skills Feedback Loop

Agent Skills — Feedback Loop

Use this skill to wire a learnings loop into another skill so it gets better with use, without rewriting SKILL.md automatically.

The loop has four moving parts:

1. learnings.md — raw, append-only, committed. Shared working memory across machines; created on first append via append_learning.py, not seeded empty.
2. learnings.consolidated.md — pruned, dated, committed. Portfolio-grade institutional memory; seeded at wiring time.
3. learnings.local.md — machine-specific notes, gitignored. Use for one-operator-on-one-machine context that should not propagate.
4. scripts/append_learning.py + scripts/consolidate.py — keep raw entries well-shaped and promote durable ones to consolidated.

The name is borrowed from the 2026 "learnings loop" pattern (MindStudio) and Anthropic's own skill-authoring guidance to ask Claude to self-reflect on what went wrong and fold it back into reusable context. The mechanism deliberately does not match MindStudio's: MindStudio's loop has the model rewrite the skill's persistent instructions directly from user corrections — that is the exact auto-rewrite this design forbids (see Anti-Patterns). This skill keeps the same "accumulate corrections across sessions" shape but routes it through append-only raw entries, human-reviewed consolidation, and an eval-gated promotion step before anything touches skill logic — mapped onto this repo's existing 4-type memory schema (see agents-memory).

Quick Reference

Workflow

1. Confirm the host skill should have a learnings loop; do not wire routers, one-off scaffolds, or stable skills with no recurring edge cases.
2. Follow references/wiring-protocol.md to seed learnings.consolidated.md, add the addendum, and verify .gitignore coverage.
3. Use scripts/append_learning.py for raw entries; do not hand-edit raw learnings.md during normal operation.
4. Use scripts/consolidate.py <skill-dir> when the raw file hits the cap, before release, or during a scheduled maintenance pass.
5. Promote durable lessons into the host skill's references/ only after human review; never auto-rewrite SKILL.md.

ASCII Flow

textCopy code
Learning-loop request
  -> Check host skill fit
     +-- router/one-off/stable surface -> do not wire
     +-- recurring edge cases          -> seed consolidated file and addendum
  -> Append raw dated bullets via script
  -> Consolidate when capped, stale, or release-bound
  -> Promote durable rules by human-reviewed reference edits

When to Use

• A skill is high-traffic and you keep teaching it the same lesson twice
• A skill's domain has emerging edge cases you want captured (e.g. tax rules, payment scheme changes, new API quirks)
• A skill failed and you asked Claude to self-reflect — the reflection needs a home

When NOT to Use

• One-off skills, internal scaffolding, or skills with stable, well-known surfaces
• Skills whose "learnings" are actually just code conventions — those belong in CLAUDE.md or coding-behavior.md, not a per-skill loop
• Cross-skill patterns — promote to references/ of the right skill or to agents-memory, not into every learnings file

Typical Scenarios

End-to-end recipes. Each maps to the machinery below — no new commands.

1. Wire a high-traffic domain skill for the first time. You keep re-teaching project-taxation the same HMRC quirks. Seed the loop, add the addendum, smoke-test:

bashCopy code
cp frameworks/shared-skills/skills/agents-skills-feedback-loop/assets/learnings.template.md \
   frameworks/shared-skills/skills/project-taxation/learnings.consolidated.md
# add the ## Learnings Loop addendum to project-taxation/SKILL.md (verbatim, see The Addendum)
python3 frameworks/shared-skills/skills/agents-skills-feedback-loop/scripts/append_learning.py \
   frameworks/shared-skills/skills/project-taxation --section "Domain Knowledge" \
   --text "MTD quarterly update deadline is 1 month + 7 days after period end."

Full procedure: references/wiring-protocol.md. Add a per-skill filter override if the domain needs a sharper bar (HMRC-anchored only, etc.).

2. A skill just failed — capture the lesson before it evaporates. The session hit a non-obvious bug. Append one dated bullet to the right section; do not edit SKILL.md:

bashCopy code
python3 .../append_learning.py frameworks/shared-skills/skills/<skill> \
   --section "Mistakes to Avoid" \
   --text "Webhook needs HMAC in the header, not the body — body-signed requests 401."

If you wired Layer 1 (scenario 5), this happens automatically at session end.

3. The raw file is full / weekly maintenance. append_learning.py refused at 150 entries, or it's your Friday cadence. Dry-run, review, consolidate:

bashCopy code
python3 .../consolidate.py frameworks/shared-skills/skills/<skill> --dry-run   # propose
python3 .../consolidate.py frameworks/shared-skills/skills/<skill>             # apply after review

Promotion criterion: an insight must have recurred ≥2× before it consolidates. Full protocol: references/consolidation-protocol.md.

4. Promote a battle-tested principle into the host skill (Layer 3). A consolidated entry has been load-bearing for 3+ cycles. It cannot enter skill logic without a discriminating regression eval:

bashCopy code
# author <slug>.json from assets/promotion-eval.template.md, then:
python3 .../promote_learning.py frameworks/shared-skills/skills/<skill> --check <slug>.json
python3 .../promote_learning.py frameworks/shared-skills/skills/<skill> --gate "<principle text>"

On a pass it prints the exact hand edit and logs to learnings.promotions.jsonl; it never edits SKILL.md. Full protocol: references/promotion-protocol.md.

5. Onboard a new laptop — close the capture loop automatically (Layer 1). Stop relying on humans to remember to append. Install the portable session-end hook (registers Claude Code SessionEnd + Codex if present):

bashCopy code
cd frameworks/shared-skills/skills/agents-skills-feedback-loop
python3 scripts/install_capture_hook.py --dry-run   # preview
python3 scripts/install_capture_hook.py             # apply
cat ~/.agents/hooks/learnings_capture.log           # capture-rate gauge

Caveat: confirm your runtime's hook event actually fires before trusting capture (see references/closed-loop-capture.md → Verifying it works). Codex hook support is newer and less stable than Claude's — treat the log as the source of truth, not the design doc.

6. Monthly drift audit across every wired skill. Catch orphaned loops, oversized files, undated entries in one pass:

bashCopy code
for skill in frameworks/shared-skills/skills/*/; do
  [ -f "$skill/learnings.consolidated.md" ] && \
    python3 .../consolidate.py "$skill" --audit
done

Anything not status=ok needs a human pass. Full checklist: references/audit-checklist.md.

Core Contract

Per-skill, the loop lives inside the skill's own directory:

Copy code
<skill>/
├── SKILL.md                       # adds the 4-line Learnings Loop addendum
├── learnings.md                   # committed, append-only, raw (shared across machines)
├── learnings.consolidated.md      # committed, pruned, dated
├── learnings.local.md             # gitignored, machine-specific, optional
└── references/...                 # consolidation can promote durable rules here

Required entry shape (enforced by append_learning.py):

• - [YYYY-MM-DD] <one-sentence atomic insight>
• One bullet, one insight. No paragraphs.
• Belongs to exactly one section: Patterns That Work / Mistakes to Avoid / Domain Knowledge / Open Questions / Consolidated Principles.

Hard limits (refuse rather than truncate):

• Raw learnings.md cap: 150 entries — exceeding triggers a forced consolidation pass.
• Consolidated file cap: 60 entries — exceeding means the skill itself needs a references/ extraction.

The Addendum

Any skill that opts in adds this block, verbatim, near the end of its SKILL.md:

markdownCopy code
## Learnings Loop

Before applying this skill on a non-trivial task, read `learnings.consolidated.md` in this directory (and `learnings.md` if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.

That's it — no other change to the host skill.

Consolidation Cadence

• Trigger: raw file hits the 150-entry cap, OR a weekly cron (recommended: Friday), OR before a release/PR that touches the skill.
• Protocol: see references/consolidation-protocol.md. Human approves promotions to learnings.consolidated.md.
• Promotion criterion: an entry must have triggered behavior change at least twice (manually noted by the operator) before it is consolidated. Single-occurrence entries age out after 90 days.

Monthly Maintenance Checklist

Run these checks on any skill with an active loop:

• python3 scripts/consolidate.py <skill-dir> --audit — confirm raw ≤150 and consolidated ≤60
• Verify every entry starts with - [YYYY-MM-DD] (no undated bullets)
• Confirm SKILL.md still carries the ## Learnings Loop addendum
• Scan for entries older than 90 days with no recurrence; flag for pruning
• Check for consolidated entries stable for 3+ cycles; run scripts/promote_learning.py --gate before promoting
• Confirm learnings.md and learnings.consolidated.md are committed, and learnings.local.md is the only gitignored learnings file
• Run the full loop audit across all wired skills (copy the for loop from references/audit-checklist.md)

Lint Commands

bashCopy code
# Audit a single wired skill (checks caps, dates, section names)
python3 frameworks/shared-skills/skills/agents-skills-feedback-loop/scripts/consolidate.py \
  frameworks/shared-skills/skills/<skill-name> --audit

# Audit all wired skills at once
for skill in frameworks/shared-skills/skills/*/; do
  if [ -f "$skill/learnings.consolidated.md" ]; then
    python3 frameworks/shared-skills/skills/agents-skills-feedback-loop/scripts/consolidate.py \
      "$skill" --audit
  fi
done

# Dry-run consolidation (proposes diffs, does not write)
python3 frameworks/shared-skills/skills/agents-skills-feedback-loop/scripts/consolidate.py \
  frameworks/shared-skills/skills/<skill-name> --dry-run

# Check Layer 3 promotion gate for a specific principle
python3 frameworks/shared-skills/skills/agents-skills-feedback-loop/scripts/promote_learning.py \
  frameworks/shared-skills/skills/<skill-name> --gate "<principle text>"

Closing the Capture Loop (Layer 1)

The base loop above is open: without a session-end hook, every append requires a human to manually call append_learning.py. In practice raw files stay near-empty. Layer 1 closes the capture half with a runtime-neutral session-end hook that detects which wired skills a session used, runs one cheap reflection pass, and appends via the existing append_learning.py — no new format, no SKILL.md rewrite.

It is machine-global and portable (any laptop, any username) and covers both Claude Code and Codex from one script:

bashCopy code
# From this skill directory, on any laptop:
python3 scripts/install_capture_hook.py --dry-run   # preview
python3 scripts/install_capture_hook.py             # apply

The installer resolves every path from $HOME at run time (nothing is hardcoded), registers Claude Code (SessionEnd) and Codex (Stop, only if Codex is present), and is idempotent. Full design, guardrails (recursion guard, blast-radius limit, fail-silent-fail-logged), and verification: references/closed-loop-capture.md. Layers 2 (scheduled consolidation) and 3 (eval-gated promotion) remain manual.

Eval-Gated Promotion (Layer 3)

Consolidation (Layer 2) ends at learnings.consolidated.md. The next step — baking a load-bearing principle into the host skill's references/ — was a pure human judgment call with no objective backstop. Layer 3 closes the promotion half with a gate, not an automator:

bashCopy code
python3 scripts/promote_learning.py <skill-dir> --gate "<principle text>"

A principle cannot be promoted unless it has a registered regression eval that is discriminating — it passes with the principle applied and fails without it. An eval that cannot fail when the principle is reverted is invalid (coding-behavior Rule 9). Grading is deterministic and zero model cost by default (capability-agnostic; PROMOTION_EVAL_CMD opt-in for model grading, which fails loud rather than silently passing if unset).

The gate never edits SKILL.md or references/ — like consolidate.py, promotion-out stays a human edit; the gate only decides whether the human is allowed to make it, prints the exact edit on a pass, and logs to learnings.promotions.jsonl. Re-run --audit before any PR touching the skill; a promoted principle's eval is now a permanent regression test. Full protocol: references/promotion-protocol.md.

Complementary Pattern: Atomic Instincts

The loop above works at skill granularity — one learnings.md per skill. It has a gap: behavioral observations that don't belong to any one skill (e.g. "always grep before editing a file this large," "this repo prefers functional style") have no natural skill home and go uncaptured. references/instinct-pattern.md documents a smaller-granularity complement — a single-trigger, single-action, confidence-scored (0.3–0.9), domain-tagged, project-or-global-scoped unit — adapted from affaan-m/ECC (commit 51a6950, MIT). It does not replace the loop above, does not adopt ECC's automatic project→global promotion (this repo keeps promotion human-gated, per ## Anti-Patterns and Rule 7 of coding-behavior.md), and is documented as a scaffold and capture-mechanism only — no seeded example entries. For hook mechanics, see agents-hooks.

What Counts as a Learning (the filter)

The filter is project-specific. The default filter lives in references/learnings-format.md under Quality Filters. Override per-skill if your domain needs a sharper bar — e.g. project-taxation should reject anything not anchored to an HMRC manual reference or a dated statute.

Judgment Calls (what a non-expert misses)

The mechanics above (caps, dates, dedup ratio) are checkable by script. The following are not, and are where most wired loops actually fail:

• A quiet loop is not a healthy loop. --audit reporting status=ok only means the files are in shape — it does not mean the loop is being used. Cross-check raw=0 or a stale oldest date against how often the host skill is actually invoked (check recent transcripts or the Layer 1 log). A high-traffic skill with zero raw entries after 60+ days almost always means the addendum stopped firing (someone edited it out, or nobody reads learnings.consolidated.md before applying the skill) — not that nothing went wrong. Silence is a signal, not an all-clear.
• consolidate.py's duplicate clustering is syntactic, not semantic. It clusters near-identical wording (SequenceMatcher ratio ≥0.82); it will not notice that "webhook needs HMAC in header" and "signature must be in the request header, not the payload" are the same lesson said twice. Treat the script's promotion proposals as a first pass, not the ground truth for "recurred ≥2×" — a human still has to read the raw file once per cycle to catch semantically-duplicate entries the ratio threshold misses, and to catch the reverse: two genuinely different lessons that happen to share vocabulary and get wrongly clustered.
• Signal vs. noise is a counterfactual test, not a vibe. Before appending, ask: if the next session had read this bullet first, would it have behaved differently? If the answer is no — it's an interesting observation, a diary entry, or something already obvious from reading the skill's SKILL.md — it's noise. Reject it even if it feels insightful; a loop that accepts "interesting" over "load-bearing" degrades into the exact ACE-paper failure mode ("context collapse" / "brevity bias" per references/evidence-base.md) where volume drowns out the few entries that matter.
• Recurring entries that never get consolidated are a maintenance failure, not a filter failure. If the same lesson keeps getting re-appended in raw form across cycles instead of being promoted to consolidated, the operator is skipping consolidation, not writing bad entries. Check the cadence (## Consolidation Cadence) before tightening the filter.
• A loop that never nominates anything for Layer 3 promotion after 6+ months is not necessarily disciplined — it may mean nobody is running the promotion-out step in consolidation-protocol.md step 5. Distinguish "genuinely no principle was load-bearing enough" from "the human-gate step is a dead letter" by checking whether learnings.consolidated.md itself is stuck at the same size cycle over cycle.
• Cross-skill leakage is easiest to miss in shared-vocabulary domains. A payments skill and a compliance skill both use words like "reconciliation" or "settlement" — a consolidated entry that reads correctly in isolation can still be the wrong skill's fact. When auditing, ask whether the entry is true because of this skill's domain or merely phrased in this skill's vocabulary.

Anti-Patterns

• Auto-rewriting SKILL.md. The loop never modifies SKILL.md. If consolidation produces a durable rule, promote it to references/ by hand. Silent self-modification of skills is the failure mode this design exists to prevent.
• One central learnings file across all skills. Forbidden. project-cosmic-tarot learnings must not leak into software-payments. See feedback_project_skills_independent.
• Learnings that duplicate CLAUDE.md or coding-behavior.md. General coding rules belong in those files, not in a per-skill loop.
• Undated entries. Without a date the entry cannot age, cannot be pruned, and cannot be weighted. Reject on append.

Compatibility

• Portable across Claude Code and Codex (no runtime-specific frontmatter).
• Scripts are Python 3.10+, stdlib only.
• Hosts skill can be any prefix family (project-*, software-*, marketing-*, etc.).

Navigation

• references/wiring-protocol.md - five-step host-skill wiring procedure
• references/learnings-format.md - entry schema, section names, and quality filters
• references/consolidation-protocol.md - deduplication, pruning, and human-review promotion path
• references/audit-checklist.md - monthly loop drift checks
• references/closed-loop-capture.md - Layer 1 auto-capture hook design, guardrails, install, and verification (Claude Code + Codex, portable)
• references/promotion-protocol.md - Layer 3 eval-gated promotion: gate rule, falsifiability requirement, human edit path
• references/evidence-base.md - graded May-2026 source provenance for the maturity claims and the Layer 1/3 designs
• references/instinct-pattern.md - atomic, confidence-scored instinct pattern (complement to the per-skill loop, adapted from affaan-m/ECC)
• scripts/append_learning.py - append raw dated entries
• scripts/consolidate.py - audit and consolidate raw entries
• scripts/promote_learning.py - Layer 3 eval gate; refuses promotion without a discriminating regression eval (never edits SKILL.md)
• scripts/bulk_wire.py - mechanical loop wiring helper
• scripts/install_capture_hook.py - idempotent installer for the Layer 1 capture hook (registers Claude SessionEnd + Codex Stop)
• assets/learnings.template.md - starter consolidated file
• assets/learnings_capture.py - runtime-neutral Layer 1 capture hook (source of truth; installed to $HOME/.agents/hooks/)
• assets/promotion-eval.template.md - Layer 3 regression-eval case schema + worked example and anti-patterns

Fact-Checking

• Verify current Claude Code, Codex, and skill-runtime behavior against official docs before making claims about automatic reflection, hooks, compaction, or skill invocation semantics.
• Treat third-party learnings-loop examples as pattern evidence only; review provenance before installing or recommending automation. Graded source provenance for the maturity and design claims is in references/evidence-base.md.
• Do not store secrets, PII, customer-specific facts, or unreleased client details in either raw or consolidated learnings files.

Related Skills

• agents-skills — for creating or auditing the host skill itself.
• agents-memory — for cross-skill, cross-session memory (the 4-type schema this loop scopes from).
• agents-hooks — the session-end-hook mechanism behind Layer 1 auto-capture (see references/closed-loop-capture.md).