Survey Methodology

Survey Methodology Skill

The default failure mode of an autonomous-agent survey is paper-trust: read N papers, organize claims into a taxonomy, ship a prose digest. The output passes type-check (it looks like a survey) but fails verification (none of the cited numbers checked, contradictions not adjudicated, code not opened, negative space not bounded). This produces B-grade output.

Across ~240 reviews from 2024-2026, A-grade reviews share one structural discriminator:

Removing the new taxonomy from an A-grade survey leaves a contribution. Removing it from a B-grade survey leaves nothing.

Empirical A-rate by domain (with our wave-1 + wave-2 evidence base):

Cochrane / BMJ / Lancet SRs are 100% PRISMA-compliant by editorial policy but item-level adherence is asymmetric: ~75% of Cochrane abstracts use GRADE, but only ~7.5% of nominally compliant SRs across journals do full certainty + reporting-bias assessment. The PRISMA label is not the substance — verify item-by-item.

Two key empirical insights from the corpus:

1. A-grade is topic-determined, not author-determined. Surveys of open artifacts (open-source models, public conference proceedings, public datasets) admit A-grade execution. Surveys of capabilities reported by closed systems (RLHF/alignment, frontier-model agents, healthcare LLMs, industry-disclosed tools like Aletheia) are structurally trapped at B because the survey author cannot independently re-execute cited results.

2. Disagreement-handling is a near-universal blind spot. 0/31 CS surveys, ~12/30 biology reviews and ~9/30 physics reviews fence-sit on contradictions. Even A-grade work routinely fails this dimension. It is the cleanest novelty axis the agent can exploit.

When to use this skill

Trigger when RESEARCH.md uses: survey, review, overview, landscape, state of the art, comparative analysis, taxonomy, benchmark of benchmarks, perspective.

Skip for primary-research projects (single experiment + paper) — those use the standard experiment / experiment_reviewer pattern directly.

Step 1 — Pick the review type explicitly

Default-narrative is the modal mistake. An autonomous agent has no editorial-gatekeeping defense, so it inherits all narrative-review failure modes (cherry-picking, confirmation bias, irreproducibility) without the defenses. Default to PRISMA-ScR-grade documentation at minimum.

Choose one and commit it in notes/scope.md before any literature load:

For AI-scientist comparison surveys, SOTA-landscape surveys, or "compare N systems' capabilities" projects: Audit / benchmark survey is the default. The ai_scientist_2026 survey was halfway there with 1 audit + 11 paper-trust = B-grade.

Step 2 — Pick a verification floor explicitly

A-grade requires at least one of the following floors. Declare which in notes/scope.md before writing. Mixing is allowed but each floor must be cleared completely.

Anything below all relevant floors is B by default. This includes "comprehensive survey", "perspective", and "tutorial" formats lacking any audit/measurement/SLR/anchor/re-derivation/re-analysis component.

Step 3 — Pre-commit scope in notes/scope.md

Write before any literature load. Required fields:

markdownCopy code
# Scope — frozen at <ISO timestamp>

## Review type
<one of the 9 types from Step 1>

## Verification floor
<one or more from Step 2; for each, name an anchor exemplar>

## Topic-ceiling honesty check
- Are the artifacts I'm reviewing open? (Open-weight models / open
  datasets / public proceedings / accessible source code = audit possible)
- Or are they closed? (Frontier-model evals / industry-disclosed tools /
  closed-weight benchmarks = structurally B-capped)
- If closed: state explicitly that the highest achievable grade is B and
  describe why; do not pretend audit is possible

## Question
<PICO/PECO/PICOS or analog. For ML: "Population: <system class>;
Intervention: <capability under test>; Comparator: <baseline>; Outcome:
<measurable>; Study design: <eligible source types>".>

## Inclusion criteria (each with a yes/no test)
- <criterion 1>
...

## Exclusion criteria (each with a yes/no test)
- <criterion 1>
...

## Information sources
- <database / venue / repo registry>, dates: <inception> → <cutoff>
- <full search query / URL pattern>

## Bounded N
- Target corpus size: ~N
- Selection rule if more than N qualify: <recency / citation rank /
  representative sampling rule>

## What is OUT of scope
- <explicit anti-list — what readers might expect but won't get and why>

Amendments after this point go in notes/scope_amendments.md with timestamp + reason. Brain must not silently rewrite scope to match what the search returned.

Step 4 — Plan experiment types per chosen floor

The notes/plan.md for a survey project must include the experiment types below, matched to the chosen floor.

For Audit / benchmark survey + Counting floor

For every system that claims a measurable capability:

1. audit_<system> — clone repo, read source, verify the README's capability claims against the actual implementation. Output: claim_verification table with claim, paper_says, code_does, verdict ∈ {Confirmed, Partial, Refuted, Not_inspectable}. Spawn one per open-source system. Closed-source: verdict: Not_inspectable + reason: closed-source.

2. benchmark_sample_<system> — for every reported benchmark number (e.g. "82% on SWE-Bench Verified"), run a sample (≥10-30 instances) on the same benchmark with the same model and check the reported number holds. Output: claim, paper_reports, sample_observed, sample_n, verdict. If running infeasible: verdict: Not_runnable with reason.

3. code_repo_inspect_<system> — separate from audit: "does the README's pip install resolve? does the example script run? are cited capabilities reachable from the documented entrypoint?" Cheapest verification, most-skipped.

For Counting / SLR floor (large corpus)

4. bounded_corpus_extract_<source> — pull all papers from a named source (proceedings/repository) within a date range; build a public lexicon for classification; release lexicon + classifications. Pattern: VLM-26K (release on GitHub).

5. screen_dual_<batch> — Cochrane two-reviewer pattern. Spawn tool_impl + tool_review blind on the same candidate batch (the blind impl/test split). Disagreements escalate to brain. Track inter-rater agreement; flag <0.7 kappa.

For Systematic review + GRADE/RoB

6. bias_assess_<study> — apply design-matched tool: RoB 2 (RCTs), ROBINS-I (non-randomized), AMSTAR 2 (included SRs in umbrella), Newcastle-Ottawa (observational). Output: per-domain signaling-question table with one of {Low / Some concerns / High}.

7. grade_certainty_per_claim — every load-bearing claim carries one of {High / Moderate / Low / Very_low} with downgrade reasons:

   • Risk of bias
   • Inconsistency (across studies / sources)
   • Indirectness (population / intervention / outcome mismatch)
   • Imprecision (sample size; for ML, run-to-run variance)
   • Publication bias (for ML: "what failures aren't being reported?")

   Untagged claims do not enter the report.

For Anchor-experiment floor

8. anchor_experiment_<claim> — pick ≥1 sub-claim from the surveyed literature; set up a controlled test; report results to standalone- empirical-paper depth (data, code, hardware identified). Materially stronger than narrative.

For Re-derivation floor (math / theoretical)

9. unified_object_<topic> — identify the single object (frame / estimator / equation / category) from which the prior literature should follow; derive it; show ≥10 named methods drop out as instances; provide ≥1 new short proof of a known result.

For Dataset re-analysis floor (earth-science / observational)

10. dataset_reprocess_<observation> — reprocess a named observational dataset (ERA5, CERES, AERONET, etc.); document provenance, version, processing-pipeline hash in notes/datasets.md; ship new figures from reprocessing.

Universal across all floors

11. cross_paper_reconcile_<metric> — for any metric reported by ≥2 primary sources, build a table comparing the values. If they disagree, adjudicate (cite which paper's setup is more rigorous, or call it a genuine open question). GRADE inconsistency / Cochrane I² analog. Don't average and move on. This is the universal blind spot — 0/31 CS surveys do it.

12. excluded_but_relevant — PRISMA item 16b. Maintain notes/excluded.md: studies/systems/papers that almost qualified, with reason. Single most-skipped item; clearest signal of confirmation bias if missing.

13. disagreement_resolution_log — per the universal blind spot, write notes/adjudication.md: every cross-paper or cross-source disagreement, the resolution policy applied, the verdict. Mirrors Copernicus open-review model where adjudication becomes a public artifact (the only mode that produces externally auditable disagreement records).

Step 5 — Citation tier policy

Apply citation downgrades systematically. Untagged citations are inadmissible for quantitative claims.

Every cited source in notes/literature.md carries a tier: field. Brain uses tier to gate quantitative claim insertion.

Step 6 — Cross-cutting requirements

These hold regardless of review type / floor.

6a. Citation budget

Hard cap: 25 cites per 1000 words of report body (Nature Reviews norm: ~150 cites over ~6000 words). Drive-by citation clusters ([3, 4, 5, 6] without claim anchor) count as 1 cite for the budget but trigger a quality flag if used >2× per section.

6b. Claim-first, not author-first

Every paragraph leads with a claim about the phenomenon, not "Smith et al. (2023) showed". This is skills/review/'s anti-stacking rule; surfaced here because the citation grammar is diagnostic of rigor — claim-first forces the writer to commit to what is true; author-first lets them launder it through attribution. Validated across all 4 wave-1 domains.

6c. Adjudicate disagreement, don't fence-sit

When primary sources contradict, the report must contain one of:

• A ruling with reasoning ("Janus 2025: the SK-vs-RSB controversy is of no consequence for off-equilibrium dynamics because…").
• An honest "currently unresolved" with the named experiment that would resolve it ("Smith 2025: octupolar correlation requires a measurement with direct sensitivity to octupolar moments below 0.05K, not yet performed").
• An escalate_authority_bound call if the disagreement requires modifying RESEARCH.md scope.

Banned: "some authors find X, others find Y; further work is needed."

6d. Negative space must be concrete

Every Open Problems / Outlook item is bound to a specific missing measurement, method, system, or experiment. Generic "more research needed" is rewritten or dropped. Pattern (validated across physics, math, and engineering): each open question names the observable / method that would close it.

6e. PRISMA-label verification

When citing an upstream SR, agent must independently verify:

• GRADE-tagged outcome table is actually present (not just the PRISMA badge — only ~7.5% of nominally compliant SRs have it)
• Funnel plot or Egger statistic for publication bias is present
• PRISMA flow diagram shows actual screening counts

PRISMA name-drop without screening log is worse than not invoking PRISMA — it's badge fraud. Flag explicitly in notes/excluded.md when an upstream review fails the verification.

6f. search_current_as_of separate from publication year

Every cited SR / review carries search_current_as_of: <date>. Cochrane reviews routinely cite as canonical with searches 5+ years stale. If search_current_as_of is older than 36 months from your work, surface as a freshness flag in the cited claim's context.

6g. Data provenance for re-analysis figures

If your survey reprocesses observational data, notes/datasets.md required:

markdownCopy code
## <dataset name>
- Source: <URL / DOI>
- Version: <version string>
- Download date: <ISO>
- Processing pipeline: <script path / hash>
- Citation: <DOI of dataset paper>

Without this, your "re-analysis figure" is unauditable — equivalent to PRISMA-name-without-substance for empirical sciences.

Step 7 — Finish-gate checklist

Before brain calls finish(), verify each item below. Adapt to chosen review type per Step 1's relaxations.

1. ☐ notes/scope.md exists, written before any literature load; amendments in notes/scope_amendments.md
2. ☐ Review type declared and committed
3. ☐ Verification floor declared and cleared (anchor exemplar matched in your output)
4. ☐ Topic-ceiling honesty check completed (B-cap acknowledged if applicable)
5. ☐ Question stated in PICO/PECO/PICOS form
6. ☐ Eligibility criteria explicit; measured-but-excluded distinguished from not-measured
7. ☐ Information sources documented with dates and full search strategy
8. ☐ Selection process documented: screener count, dual-screening policy, conflict resolution
9. ☐ Risk-of-bias / quality appraisal at study level (or declared-skipped for scoping reviews)
10. ☐ Synthesis plan pre-specified
11. ☐ notes/adjudication.md exists with disagreement resolution log (universal blind spot — non-empty for any survey covering ≥10 sources)
12. ☐ Certainty-of-evidence rating per load-bearing claim (GRADE / tier)
13. ☐ Flow diagram of identified → screened → eligible → included counts
14. ☐ notes/excluded.md exists with reason per excluded source
15. ☐ Citation budget ≤25/1000 words
16. ☐ All citations carry tier: field per Step 5
17. ☐ notes/datasets.md exists if any re-analysis figure is in report
18. ☐ Open problems concrete (each bound to named missing measurement / method / system)
19. ☐ Funding source / COI / retraction status checked for each cited primary work where applicable
20. ☐ search_current_as_of date noted for each cited SR / review

Anti-patterns — symptoms of B-grade output

Brain should flag in self-review and rewrite. Frequencies validated across the 240-review corpus.

Top 12 anti-patterns by frequency

1. "We comprehensively review..." without a corpus boundary (27/31 CS surveys, ~80% across domains). Replace with bounded N + named selection rule.
2. "We focus on" / "we prioritize" replacing inclusion criteria (22/31 CS surveys). Unfalsifiable scope; replace with explicit yes/no tests.
3. Performance numbers passed through without provenance flagging (20/31 CS surveys). Every quoted score must carry (self-reported / re-run / leaderboard) + setup.
4. Taxonomy figure as primary deliverable (26/31 CS surveys). Removing the figure should leave a contribution; if not, B-grade.
5. Application section without deployment outcome data (18/31 CS surveys). Cut sections that lack measured outcomes.
6. GitHub awesome list as audit substitute (14/31 CS surveys). Curated link list = organize; extracted structured data + reproducible code = audit.
7. PRISMA name-dropped without screening log (1 explicit, ~3 implicit per CS wave). Worse than not invoking PRISMA.
8. "First comprehensive survey of X" for non-disjoint X (6/31 CS surveys). Essentially a noise word in 2024-2026; drop or replace with specific differentiator.
9. Disagreement-handling silently absent (31/31 CS surveys, 12/30 biology). The universal blind spot. Cleanest novelty axis if addressed.
10. "Open questions for human decision" populated with brain's own punts (e.g. "Should we re-weight the matrix?"). Section is for authority-bound questions only. Use escalate_authority_bound tool instead; don't use the section as catch-all.
11. Re-summarizing already-known foundational systems in detail when RESEARCH.md says reader knows them.
12. "Some authors find X, others Y; further work needed" (~10/30 biology, ~9/30 physics). Banned per 6c.

Three failure-mode-specific anti-patterns

13. Bibliometric counting without releasing count code (Renewable Sustainable Energy Reviews modal pattern). The count is the contribution; if it's not reproducible, it's decoration.
14. Dimensional/normalization mixing (Renewable Sustainable Energy Reviews on LCOE; ML surveys on accuracy across benchmark variants). Numbers from different conditions tabulated without reconciliation.
15. Citation laundering through "expert opinion" — quoting Nature Reviews / NEJM Review Articles for quantitative claims as if they were primary research. Per Step 5: these are expert-opinion-tier and require corroboration.

Genre-aware grading

A-grade in math is different from A-grade in CS is different from A-grade in biology. Grade within genre, not uniformly:

• Math A-grade: re-derivation in unified notation OR new short proof OR theoretical-essay with sharp normative argument (Bull AMS allows the essay genre)
• CS/ML A-grade: bounded corpus + author-run measurements OR released extracted-feature data + screening log
• Biology A-grade (rare): original quantitative cross-study re-analysis (e.g. Chang & Prakash dimensionless cellular-acceleration number) OR explicit competing-model adjudication with mechanistic predictions per dataset
• Earth/environment A-grade: dataset reprocessing pipeline documented + new figures from reprocessing
• Engineering A-grade: unified formal architecture subsuming prior approaches OR harmonized device/algorithm spec sheets across primary studies

The rubric varies but the discriminator is invariant: independent verification work present, beyond paper-trust narration.

Mapping to Sisyphus architecture

• Cochrane "two independent reviewers" pattern → already mapped to tool_impl + tool_review blind split (the fix for self-circular impl/test). Survey-time application: screen_dual_<batch> experiment type.
• PRISMA artifact ledger → data/experiments/E*/runs/run_*/results.json
  • notes/experiments.md. Each PRISMA artifact (scope.md, search log, screening table, excluded.md, bias matrix, GRADE summary, adjudication log) is a discrete file, not a prose section.
• GRADE certainty per claim / tier system → tag every load-bearing assertion in notes/literature.md with certainty: {High|Moderate|Low|Very_low}
  • tier: per Step 5. Brain reads when drafting; untagged claims inadmissible in report.
• Authority-bound concern → escalate_authority_bound tool. Use when scope adjudication requires modifying RESEARCH.md. Do not dump punted decisions into a generic "Open questions for human decision" section.
• notes/datasets.md (new artifact for narrative-with-re-analysis type) → tracks observational dataset provenance, versions, processing hashes; required before any re-analysis figure renders.
• notes/adjudication.md (new artifact, Copernicus-inspired) → public disagreement-resolution log; every cross-source contradiction
  • verdict + resolver-agent + evidence path.
• compile_latex hook → can verify presence of finish-gate artifacts (scope.md, datasets.md, adjudication.md, citation count vs body length, tier:-tagging on every cite) and emit warnings to brain pre-finish.

Worked example — what the AI Scientist 2026 survey should have done

The actual run produced 1 audit (E1: AI-Researcher) + 4 synthesis experiments (E2-E5). All 11 other systems were paper-trust. Cost $32, graded B (with 1 element of A-grade in E1).

Under this skill, plan-time decisions:

1. Type: Audit / benchmark survey
2. Floor declared: Counting + Measurement (open-source systems admit both)
3. Topic-ceiling honesty: closed-source tier (Kosmos / Aletheia / Anthropic W2S / Claude Code internals / DeepMind tools) is structurally B-capped — declare in scope.md
4. Plan:

Copy code
E1-E9: audit_<system> for each open-source: AI-Researcher, RepoAgent,
       LocAgent, RepoAudit, CompileAgent, AIDE, Agent-Lab,
       AutoResearchClaw, OpenHands. Closed-source: documented
       Not_inspectable.

E10-E13: benchmark_sample_<system> for each reported benchmark number:
       - Claude Code on 30-instance SWE-Bench-Verified sample
       - AIDE on 10-instance MLE-Bench-Lite sample
       - AI-Researcher on 5 Scientist-Bench tasks
       - LocAgent file Acc@5 on 30-instance SWE-Bench-Lite sample

E14: cross_paper_reconcile of "GitHub research capability" claims
     across systems (the capability matrix becomes a verified-by-audit
     artifact, not a paper-trust digest).

E15: code_repo_inspect for each of the 9 open-source systems.

E16: disagreement_resolution_log compilation across E1-E15 — the
     universal blind spot, addressed.

Synthesis experiment (E17): the architectural-comparison + failure-mode
     + SOTA chapters, drawing only on E1-E16 evidence.

Estimated cost: $200-400. Estimated grade: A-.

The discriminator isn't writing skill (the existing skills/review/ already covers prose). It's plan-time experiment-type vocabulary. This skill is brain's vocabulary expansion.

References

Detailed evidence and quoted excerpts:

• research/interdisciplinary_protocols.md — PRISMA 2020, PRISMA-ScR, Cochrane Handbook, GRADE, RoB 2, AMSTAR 2, Arksey-O'Malley scoping, Cochrane Rapid Reviews, Greenhalgh narrative-review counterpoint.
• research/bio_med.md — Nature Reviews family editorial framing; Cochrane CDSR; eClinicalMedicine SR worked examples; the 8 things narrative reviews skip.
• research/cs_ml_math.md — BetterBench (NeurIPS 2024), Leak-Cheat-Repeat (EACL 2024), Berkeley RDI benchmark exploits (2025), Acta Numerica unified-derivation pattern, modal LLM-agent surveys as B-grade.
• research/physics_chem.md — RMP Janus collab spin-glass review, Sherrington-Kirkpatrick NRP, Annual Review of Cond Matter exemplars, Chem Soc Rev, J Mater Chem B editorial guidance.
• research/wave2/physics_broad.md — 30 reviews from Living Reviews / RMP / NRP / Annu Rev Nuc Part Sci / ARA&A / Annu Rev Fluid Mech / Reports on Progress in Physics. 70% A-rate (Living Reviews norm).
• research/wave2/chem_materials.md — 30 reviews from Chem Rev / Chem Soc Rev / Nat Rev Chem / Nat Rev Mater / Annu Rev Phys Chem / Annu Rev Mater Res. 60% A-rate; cross-paper benchmark tables in 57%.
• research/wave2/biology_broad.md — 30 reviews from Nature Reviews Immunology / Microbiology / Neuroscience / MCB / Annu Rev Imm / Biochem / Cell-Dev / Neuro / Microbiol / Plant Bio / Trends. 2/30 strict A — the discriminator is hardest to clear in this venue.
• research/wave2/cs_ml_broad.md — 31 surveys from arXiv (cs.LG, cs.CL, cs.CV, cs.AI, cs.CR, cs.SE, cs.IR). 4/31 strict A (~13%). Critical finding: A-grade is topic-determined (open vs closed artifacts).
• research/wave2/math_stats.md — 28 surveys from Acta Numerica / Bull AMS / SIAM Review / Probab Surv / Annu Rev Stat / Stat Surv / FnT TCS. 24/28 A-grade — Acta Numerica norm of unification holds. Genre-aware grading required (essays ≠ unification surveys).
• research/wave2/earth_med.md — 15 earth-environment + 15 clinical medicine reviews. Earth introduces a 3rd methodology mode (narrative-with-embedded-re-analysis); Cochrane is the verbatim template for autonomous-agent SR work. PRISMA adherence item-asymmetric (~7.5% on hard items).
• research/wave2/social_eng.md — 15 economics + 15 engineering reviews. 22/30 A or A- — JEL norm of author re-estimation; AR Econ "stylized-fact tables"; Renewable Sustainable Energy Reviews confirmed B-modal venue.

External references (for brain to fetch on demand):

• PRISMA 2020 statement (Page et al.), https://pmc.ncbi.nlm.nih.gov/articles/PMC8007028/
• Cochrane Handbook ch. 8 (RoB 2), https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08
• PRISMA-ScR (Tricco 2018), https://pubmed.ncbi.nlm.nih.gov/30178033/
• GRADE Working Group, https://www.gradeworkinggroup.org/
• BetterBench (Reuel et al. NeurIPS 2024), https://arxiv.org/abs/2411.12990
• Leak-Cheat-Repeat (Balloccu et al. EACL 2024), https://arxiv.org/abs/2402.03927
• Berkeley RDI benchmark exploits, https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/
• VLM-26K bibliometrics, https://arxiv.org/abs/2510.09586
• Lu et al. SLM survey, https://arxiv.org/abs/2409.15790
• OCL-SLR with public artifacts, https://arxiv.org/abs/2501.04897
• Greenhalgh "Time to challenge…", https://pmc.ncbi.nlm.nih.gov/articles/PMC6001568/
• Acta Numerica norm — see Vol 33 (2024), particularly Blanes-Casas-Murua splitting methods (arXiv:2401.01722) for the unified-Lie-series template
• JEL author re-estimation norm — Banerjee-Hanna-Olken-Sverdlin Lisker social protection (DOI 10.1257/jel.20241646) with openICPSR replication