AAAI Reproducibility
Use this when a draft needs to survive AAAI review on rigor, not just novelty. AAAI-27 requires the reproducibility checklist to be uploaded separately from the main PDF, in its own field on the submission form (AAAI-26 carried it inside the PDF after the references) — so it is a document a reviewer opens on its own, and it has to agree with the paper and supplement rather than read as an afterthought. AAAI-27 also states that reviewers assess reproducibility from what was actually submitted, and that material promised "after acceptance or publication" is not evidence it exists.
Reproducibility audit
- Map each central claim to submitted evidence: theorem, table, figure, ablation, appendix item, checklist answer, or code/data artifact.
- Record seeds, splits, preprocessing, hyperparameters, model selection, early stopping, prompt selection, and hardware.
- Report variance or uncertainty when stochasticity affects conclusions.
- Document dataset licenses, access constraints, sensitive data, human-subjects issues, and annotation procedures.
- Separate training compute, inference compute, and experiment search cost.
- Check the reproducibility checklist for contradictions with the main text and supplement.
Common AAAI weaknesses
- Checklist says code/data are available but supplement lacks runnable commands.
- Main results rely on one seed, one benchmark, or one prompt family.
- Baselines are weaker than current open-source or widely cited systems.
- Evaluation uses closed data or APIs with no reproducibility substitute.
- Human evaluation omits annotator instructions or quality control.
Checklist-to-evidence consistency grid
AAAI places the reproducibility checklist after the references, and reviewers cross-check each "yes" against the paper and supplement. A "yes" with no backing artifact reads worse than an honest "no", because it signals the checklist was filled in carelessly.
| Checklist answer | Must be backed by | Phase-1 risk if unbacked |
|---|---|---|
| code available | runnable scripts in the ZIP | "claimed but absent" |
| seeds reported | seed list and variance | "single-run cherry-pick" |
| compute disclosed | train vs. inference vs. search cost | "hidden tuning budget" |
| data accessible | license and access path | "irreproducible by anyone" |
Claim-evidence ledger
Create a row for every claim that appears in the abstract, introduction, or conclusion. The ledger should be short enough to audit before submission and concrete enough that a Phase-1 reviewer can see that each headline claim is checkable.
| Ledger field | What to record | Common failure |
|---|---|---|
| Claim text | exact sentence or paraphrase from the paper | claim becomes stronger than the evidence |
| Evidence artifact | theorem, table, figure, appendix, code command, data sheet, or log path | evidence exists but is not submitted |
| Reproducibility inputs | seeds, splits, prompts, preprocessing, hardware, hyperparameters, and model versions | rerun cannot recreate the result |
| Variance and controls | confidence interval, standard deviation, multiple seeds, ablation, or matched-compute baseline | single lucky run drives the claim |
| Checklist answer | the checklist item whose answer depends on this artifact | checklist contradicts the supplement |
| Reviewer risk | what a skeptical reviewer would challenge first | rebuttal cannot fix missing evidence |
For each row, choose one of three actions: keep the claim because the artifact is present, weaken the claim to match the evidence, or add the missing artifact before submission. Do not leave a row in "promise later" state.
Artifact dry-run
Before upload, run the artifact as if the reviewer has no private context:
- Unzip the submitted package into a clean directory.
- Read only the included README, not local lab notes.
- Run the smallest command that regenerates one headline table or figure.
- Check that expected runtime, hardware, random seeds, data download/access, and license constraints are stated before the command.
- Confirm that output files have deterministic names and map back to paper tables.
- Mark any non-runnable or restricted component as such in both the README and checklist.
The dry-run can be small; it does not need to reproduce every experiment. Its purpose is to prove that the submitted artifact is not merely decorative and that the checklist answers are honest.
Reviewer-pushback patterns
- "Checklist says code available but I see only figures." Fix: ship scripts and a one-line driver before the deadline; do not promise the repository in rebuttal.
- "Results may be seed-dependent." Fix: report multiple seeds with spread, and set the checklist seed answer to match the supplement exactly.
- "Closed API, not reproducible." Fix: add an open substitute model or release prompts and outputs so the claim is checkable.
Worked vignette
A vision-language paper checks "code and data available" but the ZIP holds only PDFs of plots. Audit
verdict: reproducibility grade "fragile", with a checklist conflict between the "yes" and the missing
scripts. The smallest fix is a reproduce.sh that regenerates one headline table from seeds plus a
dataset license note, after which the checklist answer becomes truthful and Phase-1 defensible.
Output format
text[Reproducibility grade] strong / adequate / fragile / not reviewable [Checklist conflicts] <answers that contradict paper/supplement> [Evidence gaps] <claims without submitted verification> [Compute/data disclosure] complete / incomplete [Priority fixes] <smallest changes before submission>
