Simulation Failure Triage

Simulation Failure Triage

Goal

Classify common simulation failure signatures and return immediate actions, retry ladders, and stop conditions.

Requirements

• Python 3.10+
• No external dependencies
• Works on Linux, macOS, and Windows

Inputs to Gather

Decision Guidance

• First preserve evidence: logs, inputs, executable version, and scheduler output.
• Separate setup errors from numerical instability and physical model issues.
• Retry with a single controlled change.
• Stop retrying when the result becomes scientifically meaningless or a required model input is missing.

Script Outputs

scripts/failure_triage.py emits:

• likely_causes
• immediate_actions
• retry_ladder
• stop_conditions
• evidence

Workflow

bashCopy code
python3 skills/robustness/simulation-failure-triage/scripts/failure_triage.py \
  --code LAMMPS \
  --stage runtime \
  --symptoms nan,pressure-blowup \
  --recent-change "increased timestep" \
  --json

Error Handling

Invalid stages or oversized log files stop with exit code 2. Unknown symptoms are retained as custom evidence.

Limitations

This skill gives first-response triage. It does not guarantee that a failed simulation can be repaired.

Verification checklist

• Recorded the preserved-evidence set BEFORE any rerun: copied logs, input decks, scheduler output, and the executable/version string (per the first immediate_actions item) so the original failure is reproducible.
• Quoted the FIRST warning/error from the run, not just the final crash line, and confirmed evidence.log_excerpt (first 500 chars, original casing) captures it; re-run with --log-file/--log-text if the excerpt misses the root signature.
• Logged every likely_causes entry with its symptom, category, and first_action, and verified no real signature landed in the category: custom bucket because its keyword was absent from LOG_HINTS/PATTERNS.
• Confirmed the failure category is correct rather than mislabeled: a segfault/SIGSEGV/signal 11/core-dumped maps to crash (not corrupted-output), and an OOM/bad_alloc/oom-kill maps to out-of-memory (not incomplete-run).
• Applied the retry_ladder strictly one change at a time, recording the single parameter changed and its effect at each rung (including the memory rung when memory-bound).
• Checked every stop_conditions entry and halted the retry loop instead of stacking arbitrary stabilizing tweaks when any condition is met (e.g. unverifiable potential/pseudopotential, results dependent on arbitrary stabilizers).
• After any numerical recovery, handed off to the simulation-validator skill for conservation / physical-bounds / "can I trust these results" checks rather than declaring success here.

Common pitfalls & rationalizations

Security

Input Validation

• --stage is checked against a fixed allowlist (setup, runtime, postprocess, unknown); any other value exits with code 2.
• --code must be non-empty and at most 100 characters; an empty or oversized value exits with code 2.
• --symptoms are split on commas and capped at 50 entries, each at most 100 characters; exceeding either limit exits with code 2.
• Unknown symptoms are not rejected; they are retained as custom evidence rather than silently dropped.
• There are no numeric arguments, so no finite/positive checks apply.
• All input-validation failures (and log read errors) are caught and exit with code 2 and a stderr message.

File Access

• The only file read is the optional --log-file path; when omitted, the script reads no files.
• Before reading, the log file's size is checked via stat; if it exceeds the 10 MB cap (MAX_LOG_SIZE) the script raises and exits with code 2. Both file content and --log-text are then truncated to 10 MB.
• The script writes no files; all results go to stdout (JSON with --json, otherwise plain text).
• Path-sandboxing caveat: the log path is not restricted to a working directory, so the script can read any file the invoking process has permission to read; only pass trusted paths.

Tool Restrictions

• allowed-tools is Read, Bash, Write, Grep, Glob.
• Bash runs only the bundled scripts/failure_triage.py.
• Read opens the skill's own files and a user-provided log to inspect failure evidence; Grep/Glob locate log files, inputs, and prior output; Write records triage notes or preserved evidence at the user's direction. The bundled script itself does not write files.

Safety Measures

• No eval/exec and no dynamic code execution; log text is treated as data and never run.
• The script spawns no subprocesses and runs no external solvers.
• Output is structured JSON (--json), making results machine-checkable.
• DoS caps bound resource use: 10 MB log size/text limit, 50 symptoms, and 100-character limits on code and each symptom.

References

• See references/failure_patterns.md for common failure signatures and retry ladders.

Version History

• 1.1.3: Add Verification checklist (evidence-based checks tied to failure_triage.py outputs — preserved-evidence set, first-error excerpt, category-mislabel guards, one-change retry ladder, stop conditions, simulation-validator handoff) and Common pitfalls & rationalizations table covering last-line-only triage, segfault-as-I/O, OOM-vs-walltime, stacked stabilizing changes, and "it ran so it's valid".
• 1.1.1: Strengthen evals to be discriminating — each case now pins the exact failure_triage.py JSON output via script_checks (specific category/first_action strings, log-hint symptom inference, fixed retry-ladder/stop-condition text, recent-change immediate action), so the suite measures the skill's actual output rather than generic triage knowledge.
• 1.1.0: Add dedicated out-of-memory and crash (segfault) classifications; segfaults and OOM are no longer mislabeled as I/O or interrupted-execution. Preserve original-case log excerpt. Add input-validation caps (symptoms, code length). Sharpen description to defer deep validation to simulation-validator. Fix missing-potential eval token.
• 1.0.0: Initial cross-code simulation failure triage skill.