Security Audit — adversarial, evidence-grounded
A full audit, not a checklist. It runs in phases, keeps a coverage ledger so
nothing is skipped silently, and — the core discipline — the agent that finds a
candidate is never the one that confirms it. Severity is likelihood × impact,
not deviation from a style rule. Output is machine-readable and reproducible.
Methodology inspired by Cloudflare's public security-audit-skill, rebuilt to reuse maggy's own pieces:
council-reviewfor adversarial validation,cpg-analysis(Joern/CodeQL) for static taint/data-flow,agent-teams/polyphonyfor isolated parallel hunters, andsecurityfor the vuln classes.
Principles (read first)
- Evidence over intuition. A finding is
confirmedonly when you can name the file:line of the boundary that fails and a concrete attack that crosses it. - Finder ≠ validator. Whoever proposes a candidate must not confirm it. A
different agent (or a
council-reviewmodel) tries to disprove it. - Impact-driven severity. Rate
likelihood × impact, not "differs from best practice". A hardcoded key in a test fixture is not critical; an auth bypass on a tenant boundary is. - Coverage is tracked, not assumed. Every input surface / trust boundary is a ledger unit with a status; a coverage critic hunts the gaps.
- No theater. Do not pad the report with generic "consider using HTTPS" advice. Report only boundary failures you can stand behind.
Phases
1. Reconnaissance → architecture.md + coverage-ledger.json
Map the target before hunting. Identify: entry points (HTTP routes, CLI, queue
consumers, webhooks), trust boundaries (authn/authz, tenant isolation, privilege
transitions), input surfaces (params, headers, files, env, deserialization),
secret handling, external calls (SSRF surface), and data stores. Write a short
architecture.md, then enumerate every surface as a unit in
coverage-ledger.json ({id, surface, boundary, status: pending}).
2. Coverage-led hunting
For each ledger unit, hunt the relevant classes. Prefer isolated sub-agents
(via agent-teams / polyphony) so one hunter's context does not bias another,
and lean on cpg-analysis for data-flow/taint where a graph beats grep. Mark each
unit hunted; a coverage critic pass re-reads the ledger and reopens units
that were skimmed. Classes to cover (depth in security + cpg-analysis):
| Class | Look for |
|---|---|
| Injection | SQL/NoSQL/OS/LDAP/template; unparameterized queries, shell=True, eval |
| AuthN / AuthZ | missing checks, IDOR, broken tenant isolation, JWT/session flaws |
| Secrets | keys in code/history/logs, client-exposed VITE_/NEXT_PUBLIC_ secrets |
| SSRF / egress | user-controlled URLs, metadata endpoints, unvalidated redirects |
| Deserialization | pickle/yaml.load/Marshal on untrusted input |
| Path / file | traversal, arbitrary write, zip-slip, unsafe temp files |
| LLM / prompt | prompt injection, tool-abuse, unbounded fan-out, data exfil via output |
| Supply chain | typosquats, unpinned deps, postinstall scripts, CI token scope |
| Cloud / IaC | over-broad IAM, public buckets, exposed admin, secrets in env |
| Client-side | XSS, DOM sinks, CSP gaps, sensitive data in localStorage |
| Resource / DoS | unbounded loops/allocations, regex catastrophic backtracking |
| Data isolation | cross-tenant reads, missing RLS, PII in logs/caches |
| Memory (native) | overflow, UAF, integer wrap (for C/C++/unsafe Rust targets) |
3. Adversarial validation (finder ≠ validator)
For each unique candidate, a different reviewer attempts to disprove it:
is the tainted input actually reachable? is there a guard upstream? is the sink
real? Route this through council-review (multiple models vote) for anything
rated high/critical. Assign a verdict: confirmed, needs_validation, or
rejected.
4. Structured output → findings.json
Emit findings.json conforming to report-schema.json (shipped in this skill).
Validate it before reporting:
bashpython3 "$(cat ~/.claude/.bootstrap-dir)/skills/security-audit/validate_findings.py" findings.json
The validator enforces the rules that keep the audit honest: unique ids, valid
enums, and that every confirmed finding has a file:line location, an attack
scenario, and a validated_by that differs from found_by.
5. Record verification
A fresh agent re-opens each confirmed finding and checks the cited file:line
still supports the claim (code may have moved). A material mismatch drops it
back to needs_validation.
6. Target-neutral report → REPORT.md
Write REPORT.md (executive summary + confirmed findings by severity),
FINDINGS-DETAIL.md (per-finding evidence + remediation), and
NEEDS-VALIDATION.md (candidates that could not be confirmed). Neutral tone: no
vendor names, no editorializing, just boundary → attack → impact → fix.
Anti-patterns
- Confirming a finding you found yourself without an independent disprove attempt.
- Severity inflation ("uses md5" rated critical with no reachable attack).
- Grep-only hunting on a data-flow bug — use
cpg-analysis. - A report that lists advice instead of reachable, evidence-backed findings.
- Running audited target code outside a sandbox. Read and analyze; never execute untrusted target code to "see what it does".
How it fits the harness
- Slots into the
baseDefinition of Done for security-critical changes as the "prove it's safe" gate, above the preventivesecurityskill. - Reuses
council-review(adversarial validation),cpg-analysis(static taint),agent-teams/polyphony(isolated parallel hunters). - Iterations are additive: a second run over the same ledger typically surfaces more, so re-run before a major release rather than trusting one pass.
