Analyze Paper Recovery

Analyze Paper Recovery

Use this skill after recover-paper-result has produced recovery/recovery_result.json. The goal is to decide whether the generated skills are good enough or whether specific module skills need refinement.

Write all analysis artifacts in English.

Inputs

• paper_profile.md
• module_plan.json
• modules/*.md
• generated skill validation JSON files
• environment/runtime_handoff.json
• environment/logs/command_log.json
• recovery/source_manifest.json
• recovery/recovery_result.json
• recovery/experiment_validation.json

Outputs

• analysis/analysis_report.json
• analysis/feedback.md

Workflow

1. Confirm the recovery respected the source boundary. If the source manifest includes the original repo, return refine.
2. Confirm recovery/recovery_result.json.paper_target matches module_plan.json.fast_recovery_target for dataset, metric, and target value. If the recovery target drifted, return refine until the plan or recovery is corrected.
3. Compare recovered metrics against the target declared in module_plan.json.fast_recovery_target.
4. Inspect recovery_result.json.mechanism_checks when present. Metric success is not sufficient if the recovery skipped required searches, injected unsupported facts, or failed to exercise a core module contract.
5. Separate full-result gaps from proxy-result gaps:
   • Full result: compare directly to the paper metric.
   • Proxy result: evaluate whether the proxy exercised the method contracts and whether the proxy metric passed its declared threshold.
6. Inspect generated skill validation logs for failures or weak coverage.
7. Inspect recovery/experiment_validation.json or run recover-paper-result/scripts/validate_recovery_experiment.py. Do not accept a run whose experiment gate fails.
8. Produce a decision:
   • accept: skills are usable and recovery/proxy evidence is sufficient.
   • refine: skills need another cycle.
9. When refining, assign feedback to concrete module ids and explain what to change.

Decision Heuristics

Accept if all are true:

• Source boundary is valid.
• The declared module-plan target and the recovered-result target metadata match.
• All module skill validations passed.
• Recovery produced a numeric metric.
• recovery/experiment_validation.json reports ok: true.
• Recovery has executable command evidence, not only hand-written result JSON.
• The generated skill invocation log proves each claimed core module was called, imported, or cross-checked.
• Mechanism checks pass when the recovery harness records them.
• For full reproduction, the primary metric gap is within the configured tolerance.
• For proxy reproduction, the proxy is justified and exercises the core paper mechanism.

Refine if any are true:

• A module skill test failed.
• The run reports missing packages, model cache, dataset, benchmark files, or search credentials as a blocker, but environment/logs/command_log.json does not show the allowed setup attempts first: isolated environment creation/reuse under the DisCo-private env root, targeted package installs, permitted model/dataset downloads, configured network/VPN or mirror retries, or a user credential/permission request when credentials are genuinely required. Treat this as refine of environment preparation, not a terminal blocker.
• environment/runtime_handoff.json points to a private recovery Python, but the recovery experiment command log or source manifest shows the experiment used the host/shared conda Python instead.
• The recovery result targets a different dataset, metric, or paper value than the module plan.
• A proxy recovery omits mechanism_checks; proxy acceptance requires explicit evidence that the paper mechanism was exercised.
• A nontrivial recovery run lacks environment/runtime_handoff.json, or recovery ignores a handoff blocker by claiming a full runtime succeeded without matching package/model evidence.
• Environment preparation lacks environment/logs/command_log.json, or that file does not contain a command list for subprocess probes, installs, downloads, clone attempts, and version checks.
• recovery/experiment_plan.md, recovery/logs/experiment_command_log.json, recovery/logs/generated_skill_invocations.json, or recovery/experiment_validation.json is missing.
• The experiment command log has no successful command, no command strings, or cannot be tied to the produced recovery result.
• A benchmark/runtime source was obtained, but the recovery item is only described as benchmark-style and does not record concrete resource-file provenance when resource-derived construction was feasible.
• A recovery reuses a benchmark checkout from another attempt or workspace path without recording the fresh-fetch blocker, reused path, commit/version, concrete files used, and a current-attempt snapshot or copy of those resource files. Reusing previous attempt generated data, prompts, outputs, training logs, or recovery results as evidence should return refine.
• A reduced data-generation/training recovery lacks recovery/logs/generated_data_item.json or a discoverable primary training trace with loss/metric before and after, parameter or optimizer-change evidence, and fallback/full-runtime booleans. Prefer params_before and params_after in the trace; parameters_before and parameters_after may appear as readable aliases.
• The recovery claims to exercise a generated loss, scoring, retrieval, or prompting module but bypasses that module with duplicate harness logic and provides no cross-check or equivalence log.
• A trajectory-derived skill puts failed or invalid actions into golden_workflow instead of keeping them in mistake_analysis, unless the paper requires retaining failed attempts in the ideal plan.
• A prompt-separation test rejects ordinary task/environment words instead of privileged-only skill content; refine the test contract rather than weakening the prompt-separation invariant.
• The recovery metric passes but the mechanism check shows missing required searches, unsupported Reason-in-Documents facts, or no live search events.
• Recovery skipped a core method component.
• The result relies on original repo code.
• The metric gap is large and unexplained.
• The raw answer is semantically correct but the extracted/evaluated answer fails due to a missing answer-formatting or canonicalization rule.
• A retrieval/document-refinement module injects final-answer text or other downstream-control text that is outside its output contract.
• The run used a mutable environment change that was not authorized or not recorded.
• The artifacts are not sufficient for a future agent run to repeat the experiment.

Scripts

• scripts/compare_recovery.py: compare recovery metrics and produce analysis_report.json.