Experiment Log

Experiment Log

When to Use This Skill

Use when:

• starting a modeling experiment that should be comparable later;
• the user asks “what did we try?” or needs a decision history;
• promoting/rejecting candidates under a locked validation design;
• recording failures, deviations, and why a model was or was not shipped.

Do not use this skill as a substitute for:

• locking metrics and folds → validation-design;
• writing the public-facing results narrative → results-reporting;
• the durable model contract → model-card.

Outcome

Create one immutable record per executed experiment so another analyst can reconstruct what was tried, compare it with its declared baseline, audit its validity, and understand why it was kept, discarded, or queued for follow-up. Failed runs are evidence and belong in the history.

An experiment log is trial history. A validation charter defines evaluation; a model card freezes a promoted model contract; a results report explains an evaluation to readers.

Read the log schema while creating or auditing a record and the decision rules before assigning the final decision.

Create before execution

Record before fitting:

• unique ID, UTC timestamp, operator, falsifiable hypothesis, expected direction;
• sport, competition, row grain, target, population, and decision time T;
• immutable data snapshot/query and its source/retrieval metadata;
• feature-set reference with availability and transformation rules;
• baseline/candidate configuration, validation charter, and locked primary metric;
• code version, environment/lock reference, random seeds, and exact command;
• predeclared success, integrity, calibration, stability, and complexity limits.

Do not rewrite these fields after observing results. Corrections and deviations are appended with timestamp, author, reason, and effect on validity.

Schema

textCopy code
experiment_id: YYYYMMDD-<slug>-<nn>
created_at_utc:
operator:
hypothesis / expected direction:
sport / competition / grain / target / eligible population:
prediction_timestamp_rule:
data_sources / immutable_snapshot / data_window:
feature_set_ref / baseline_refs:
validation_charter_ref / primary_metric / success_rule:
model_family / config_ref / code_version / environment_ref:
random_seeds / commands:
status: planned | running | completed | failed | invalidated
fold_metrics / metrics_primary / metrics_secondary:
calibration / slice / stability results:
leakage_audit_status / failures / deviations:
results_summary:
decision: keep | discard | follow-up | invalid
decision_reason / next_actions:
artifacts / checksums / notes:

Workflow

1. Search existing logs, assign a new ID, and create the record before the run.
2. Freeze the falsifiable hypothesis, primary metric, baselines, and success rule.
3. Link immutable data, feature, validation, config, code, and environment artifacts.
4. Record exact noninteractive commands or notebook cell/version identifiers.
5. Execute baseline and candidate on identical eligible rows and folds.
6. Append fold-level metrics, runtime, warnings, and output artifact checksums.
7. Record calibration, leakage, slice, and failure diagnostics where applicable.
8. Mark deviations and invalid runs; never delete or convert them into successes.
9. Decide from the predeclared rule, then record one concrete next action if needed.
10. Promote to a model card only after keep and all integrity gates pass.

Comparison discipline

Candidate comparisons are valid only when target, T, population, data snapshot, rows, folds, metrics, and baseline definitions match. If they differ, log the difference and do not attribute the metric change solely to the model.

Store per-fold results rather than averages only:

textCopy code
fold | train_period | test_period | n | baseline_primary | candidate_primary |
gap | secondary_metrics | calibration | warnings

Include uncertainty or dispersion appropriate to the design. Never promote a candidate because one favorable fold offsets repeated failures hidden by a mean.

Decision rules

Post-hoc metrics must be labeled post-hoc and cannot silently drive the primary decision. A changed metric, population, or hypothesis requires a new ID.

Failure and deviation handling

Record nonzero exits, exceptions, timeouts, warnings, empty folds, missing artifacts, convergence failures, seed instability, and manual intervention. Include the last valid stage and whether partial outputs are trustworthy.

When leakage or a validation violation is found later, append an invalidation entry to every affected experiment and downstream artifact. Do not overwrite the original record or reuse its metrics after repair; the repaired run gets a new experiment ID linked to the invalidated one.

Anti-patterns

Standalone helper

bashCopy code
python /path/to/experiment-log/scripts/new_experiment.py \
  --slug home-form-logit --sport nfl
python /path/to/experiment-log/scripts/new_experiment.py \
  --slug elo-sensitivity --out-dir data/experiments

The helper creates a timestamped, user-owned Markdown artifact and validates the slug. Creation is exclusive: concurrent invocations retry the sequence and never overwrite or reuse an existing ID. The stub contains every field in the required schema, but it is only a planned record; complete the pre-run fields before execution and append results/deviations without rewriting the frozen contract.

Worked example

textCopy code
experiment_id: 20260824-home-form-logit-01
hypothesis: Shifted 5-game form improves test log-loss over constant train rate.
sport / grain / target: nfl / team-game / won
prediction_timestamp_rule: scheduled kickoff
immutable_snapshot: snapshots/nfl_team_game_2019_2025.parquet (sha256: ...)
feature_set_ref: feature-cards/team-form-v3.md
validation_charter_ref: charters/season-wf-v2.md
primary_metric: log-loss
success_rule: beat baseline on mean and majority of outer folds; audit CLEAN
commands: [exact invocation]
status: completed
fold_metrics: artifacts/20260824-home-form-logit-01-folds.json
leakage_audit_status: CLEAN
decision: keep | discard | follow-up | invalid
decision_reason: fill after evaluation

Review checklist and integrity rules

• ID existed before results; hypothesis is falsifiable.
• Data, code, config, feature, charter, environment, seeds, and commands resolve.
• Baseline/candidate share rows, folds, and metrics or differences are disclosed.
• Fold-level results and failed checks are present.
• Leakage status and decision follow the predeclared rule.
• Referenced artifacts are immutable and checksummed where practical.

Never omit failed runs, alter the primary metric silently, overwrite referenced artifacts, or promote without a baseline and timing audit. Use log_schema.md for required field semantics and decision_rules.md for promotion logic.