Anti Slop Analytics

Anti-Slop Analytics

Outcome

Produce an item-by-item keep, fix, or kill review. Every retained figure or table must make a defensible claim and disclose its population, period, denominator, comparison baseline, uncertainty, and reproduction pointer.

Pretty is optional. Honest is required. Presentation cannot rescue a weak design, a losing model, or an unverifiable calculation.

When to use this skill

Use it when reviewing:

• exploratory or diagnostic plots;
• model-comparison and calibration figures;
• dashboards, notebooks, reports, papers, or public posts;
• requests to make results look cleaner, stronger, or more impressive;
• any visual whose design may conceal a weak baseline or unstable result.

If no visual or table exists, create the analytical artifact first. Use sports-visualization for figure construction, validation-design for the evaluation itself, and results-reporting for the full narrative. This skill audits presentation; it does not repair the underlying study by relabeling it.

Inputs and evidence

Request or inspect:

• the figure, table, notebook, or report;
• the one claim each item is intended to support;
• the underlying user-owned data or metrics artifact when available;
• the decision context and audience;
• the plotting command, script, or notebook cell;
• relevant filters, exclusions, and fold definitions.

If evidence is missing, audit what is visible and mark unverifiable properties as unknown. Never infer provenance, sample size, uncertainty, or held-out status from visual polish.

Audit workflow

1. Inventory every figure and table with a filename or stable identifier.
2. Write the single decision-relevant claim each item is meant to support.
3. Verify sport, competition, grain, period, population, sample size, exclusions, and denominator.
4. Confirm that the plotted quantity matches the title, caption, and claim.
5. Check that a relevant null, naive model, or incumbent baseline is visible.
6. Inspect axes, transformations, scales, bins, smoothing, color, ordering, facets, and annotations.
7. Separate training, in-sample, held-out, and forward-looking evidence.
8. Inspect statistical honesty: uncertainty, multiplicity, selection, missing segments, and sensitivity to the chosen time window.
9. Trace the item to a user-owned data file, metrics file, notebook, or command.
10. Assign keep, fix, or kill; prescribe the smallest honest replacement.

Failure catalog

For additional examples, read references/slop_catalog.md when classifying a suspect item and references/replacements.md when choosing a redraw.

Verdict rubric

If a decorative chart can be fixed only by changing its core claim, kill it and redraw from the underlying evidence.

Sports-specific checks

• A doubled team-game panel has two rows per contest. Do not report its overall win rate as home advantage; filter home rows or return to game grain.
• Team and player rankings often have unequal schedules and sample sizes. Show opportunity, exposure, or shrinkage rather than raw order alone.
• Season averages can hide rule, schedule, roster, or measurement changes. Label eras and test whether the conclusion survives them.
• Probabilities require calibration evidence. Discrimination alone does not show that a displayed 70% forecast occurs about 70% of the time.
• Rolling form and rating lines must be as-of series. Centered windows or post-event updates cannot support a pre-event claim.
• Fold charts must use identical held-out events for candidate and baseline. A bar comparison is invalid if populations differ.
• Ties, overtime, postseason games, and canceled events need visible treatment when they change denominators or target definitions.

Replacement defaults

Caption repair template

Use a factual caption that answers:

textCopy code
What: <metric or quantity and units>
Who/where: <sport, competition, population, grain>
When: <period>
Evidence: n=<denominator>; <baseline or null>
Uncertainty: <interval, fold spread, or explicitly unknown>
Interpretation: <bounded claim>
Source: <artifact and reproduction pointer>

Avoid conclusion-first titles such as “Dominant Model.” Prefer a measurable title such as “Held-out log-loss by season versus constant baseline.”

Hard constraints and integrity rules

1. Never truncate an axis to manufacture an effect without conspicuous, defensible disclosure; default to an honest comparison scale.
2. Never drop losing folds, teams, segments, or seasons silently.
3. Never present in-sample curves as validation.
4. Every key item needs period, denominator, metric definition, sport, and grain.
5. If uncertainty is unknown, say unknown; do not invent error bars.
6. A baseline is required whenever a comparative model claim is made.
7. A reproduction path is required for an item used in a public claim.
8. Never imply causality from observational association.
9. Never label probabilities calibrated without a calibration check.
10. Never hide missing data, filters, exclusions, or changed definitions.

Anti-patterns

• dashboard makeup on a weak model;
• annotation spam that tells viewers what to feel;
• color used as the argument;
• one magical chart carrying the entire claim;
• a UI skin covering a missing method;
• hiding the constant baseline because it makes the result look weaker;
• dense visuals whose only purpose is to signal sophistication;
• rebuilding a chart from rounded report numbers when source metrics exist.

Worked audit

Input: A chart titled “Dominant Model” shows cumulative accuracy from one season. The y-axis begins at 0.55, there is no base-rate line, no sample size, and no underlying metrics file.

Verdict: kill.

Reasons: axis distortion, cherry-picked period, baseline erasure, metric laundering, overclaiming title, and missing provenance.

Replacement: Show held-out log-loss by season for the candidate and locked baseline on identical events; add fold sample sizes and a factual title. If the source metrics cannot be recovered, report the evidence as unverifiable rather than recreating precise values from the screenshot.

When a suitable metrics JSON already contains a folds array, an honest comparison can be drawn with the standalone visualization helper:

bashCopy code
python <path-to-sports-visualization>/scripts/plot_walkforward_metrics.py \
  --json data/metrics.json \
  --metric logistic_log_loss \
  --baseline constant_log_loss \
  --out data/walkforward_log_loss.png

Then audit that exported artifact and its underlying JSON.

Helper

Run the bundled helper by absolute path or from this skill directory:

bashCopy code
python <path-to-anti-slop-analytics>/scripts/figure_audit_template.py \
  --title "NBA walk-forward log-loss" \
  --claim "Candidate improves held-out log-loss versus the locked baseline" \
  --out data/figure_audit.md

The helper writes a user-owned Markdown checklist and requires no project package. It creates an audit scaffold; it does not decide the verdict.

Output contract

Return:

1. an executive verdict;
2. one audit row per item with keep, fix, or kill;
3. the named failure patterns and evidence for each verdict;
4. exact replacement instructions, including captions, for every fix;
5. a list of missing or unverifiable evidence;
6. the remaining misread risk after repair;
7. artifact paths or commands needed to reproduce every retained item.

Resources

• references/slop_catalog.md — read when a visual needs a named failure classification.
• references/replacements.md — read when selecting the smallest honest redraw.
• scripts/figure_audit_template.py — portable audit-template writer.