Slop Eval

Slop Eval

Evaluate a design the way skill-evaluation evaluates a skill: every finding cites concrete evidence, every axis gets a 0–100 score, arithmetic runs through a script, and the output is a structured report — never a vibe check.

The tell catalog lives in references/tells.md; read it before sweeping. The positive rubric (signature formula, cohesion checks, slop→premium pairs, and the "Adding Soul" guide) lives in references/premium-markers.md; read it before scoring Axes 7–8 and when writing fix prescriptions. The design context guide lives in references/contexts.md; read it to adjust priorities and tolerances based on the type of design being evaluated.

Source

• The pols.dev anti-slop design law — the tell catalog, absolute rules, and signature formula are distilled from it.
• Method modeled on skill-evaluation (cite-or-cut, weighted axes, scripted scoring, failure-mode diagnosis).

Parameters

Write the report in the language the user is speaking; keep tell IDs and names in English so they stay greppable against the catalog.

Evaluation modes

Standard mode (default)

Single evaluation of a design. Produces a Slop Report with scores, tells, section ledger, and prioritized fixes.

Comparison mode

Side-by-side evaluation of two different designs (e.g., competitor analysis, A/B variants). Add --compare pointing to another target or existing report.

Tracking mode

Evaluate the same design over time to measure improvement. Use when:

• Running weekly/sprint design reviews
• Measuring progress after a redesign
• Validating that fixes actually moved the score

Usage:

bashCopy code
# First evaluation — establishes baseline
slop-eval --target https://site.com --output ./reports/baseline.md

# Later evaluation — tracks evolution
slop-eval --target https://site.com --output ./reports/week-2.md \
  --compare ./reports/baseline.md

Tracking mode adds to the report:

• Score progression table with trends (✅ improved / ⚠️ regressed)
• Tells resolved (what got fixed)
• New tells (what got introduced)
• Regressions (axes/sections that got worse)
• Section ledger evolution
• Velocity metrics (tells resolved per week, score improvement rate)
• Recommendations for next iteration

See references/output-template.md for the full tracking output format.

Evidence channels

What you can verify depends on what you were given. Never score a check you could not observe — mark it Unverifiable and exclude it (like N/A in skill-evaluation).

With code, grep before you stare: fonts.googleapis|next/font, lucide-react, linear-gradient, box-shadow, border-radius: *9999, backdrop-filter, opacity: *0, initial={{ *opacity: *0, overflow: *hidden, clip-path, position: *fixed. Each hit is a lead, not a verdict — confirm against the catalog entry before recording it.

Evidence acquisition SOP

Route by what the target is; always end with an evidence inventory (what was captured, what is Unverifiable) — it feeds the report header.

Live URL — the richest channel; prefer it whenever reachable. Use whatever browser automation this session has (a browser MCP such as Playwright or Chrome DevTools, or npx playwright screenshot as the no-MCP fallback) and capture, saving every artifact to the scratchpad so findings can cite file + region:

1. Load at desktop (1440×900) and mobile (390×844); wait for network idle.
2. Full-page screenshot of both viewports immediately after load, before any scrolling — sections sitting at opacity:0 waiting for a scroll reveal show up blank here (M1 evidence).
3. Scroll pass top to bottom, then a second full-page capture; diff the two mentally for reveal-gated content, seams (C11, X13), and fold ownership (L16).
4. Interaction pass: hover the primary CTA, one card, one nav link (M2–M4); click every tab, accordion, toggle, and button (M8); Tab through the page and confirm a visible focus ring (X14).
5. Zoom crops at 2x of: anything near a clipped edge (X2), circled/tiled numbers and icons (X1), pricing columns side by side (X3), button labels (X5).
6. Pull the rendered sources for the code-channel greps: font names from the network panel or <link>/@font-face, computed hex values from the stylesheets.

No browser automation available → fetch the HTML/CSS (curl) and run the code channel on it, ask the user for full-page desktop + mobile prints, and mark every visual-only and interaction check Unverifiable until the prints arrive. Never score a visual check from raw HTML.

Screenshots — Read each image. If only partial crops were provided, ask for full-page desktop + mobile before sweeping (a hero-only print cannot support L11, L15, or the cohesion axis). All interaction checks (M1, M8, X14, hover tells) are Unverifiable.

Code path — run the greps, read every file they hit, plus the layout/ page components and global styles. If the project runs locally, start its dev server and continue under the Live URL SOP — code plus a live render is the only combination that can verify everything.

Figma export — treat as Screenshots for visual tells; additionally fonts, hex values, and spacing are exact from the file. Motion and interaction axes are Unverifiable (score NA for Axis 5 unless prototypes were shared).

Axes and weights

Axis 7 carries the heaviest weight on purpose: the law's deepest rule is that dodging the tell list is still slop — a page with zero tells and no signature is unfinished work wearing restraint as an alibi.

Scoring

Axes 1–6 (tell-counted). Count confirmed tells on the axis by severity, then: score = max(0, 100 − 30·critical − 15·major − 5·minor). Run scripts/score.py axis CRIT MAJOR MINOR — don't do it by hand. One tell, one count: a pattern repeated across sections is still one tell (note the repetition in the evidence; repetition may upgrade minor → major where the catalog says so).

Axis 7 (Signature). Score each of the 7 formula elements 0 (absent), 50 (attempted, weak), or 100 (strong) per the rubric in premium-markers.md; the axis is their mean.

Axis 8 (Cohesion). Same 0/50/100 on the 4 cohesion checks; mean.

Compounding rule. Three or more major layout tells on one page cap Axis 4 at 40 — a page assembled from known skeletons is slop no matter how clean each block is.

Gates (pass as --cap to the overall run):

• Signature gate: Axis 7 < 40 caps the overall at 59 (grade C max). No amount of clean spacing rescues a page with no signature.
• Absolute-rule gate: any confirmed critical tell caps the overall at 69 (no grade A with broken execution).

Overall & Slop Index.

Copy code
overall    = sum(axis_score × weight) / sum(weight)   # capped by gates
Slop Index = 100 − overall

Run scripts/score.py overall 1:80:2 2:65:2 ... [--cap 59] [--cap 69]. Unverifiable axes score NA and drop out of both sums. --fail-below N exits non-zero for CI gating, e.g. gating a PR on its preview deploy:

yamlCopy code
# .github/workflows/slop-gate.yml (step excerpt)
- name: Slop gate
  run: |
    # run slop-eval against $PREVIEW_URL, export each axis score, then:
    python3 skills/slop-eval/scripts/score.py overall \
      1:$A1:2 2:$A2:2 3:$A3:1 4:$A4:2 5:$A5:1 6:$A6:2 7:$A7:3 8:$A8:2 \
      --fail-below 40

Grade scale

Absolute rules check

Six execution laws, each pass/fail/unverifiable, reported in their own table. Any fail is a critical tell (counts on its axis AND triggers the absolute-rule gate):

1. Content visible by default — nothing gated on an entrance animation (opacity:0 + reveal) (M1)
2. Clear the cut — no text/control sliced by clip, notch, overflow, or fixed height (X2, X11)
3. Parallel alignment — comparable columns share baselines; buttons anchored (X3)
4. Real centering — everything meant to be centered is, mathematically and optically (X1)
5. Legible contrast — every text clears its background by a real value gap (X5)
6. Controls work — every interactive-looking control responds (M8)

Workflow

1. Gather evidence — route the target through the Evidence acquisition SOP above. Done when the evidence inventory states what was captured and what is Unverifiable.
2. Read references/tells.md — the catalog you sweep against.
3. Sweep axes 1–6 — walk the catalog group by group. Cite-or-cut: a tell is only recorded with concrete evidence (hex value, font name, file:line, or screenshot region); no evidence, no tell. Check each candidate against its premium-pair note — the crafted version of a pattern is not the tell. Done when every catalog group has been swept and every recorded tell carries a citation.
4. Run the absolute rules check — all six, pass/fail/unverifiable with evidence.
5. Score Axes 7–8 — read references/premium-markers.md, score the 7 signature elements and 4 cohesion checks with one-line justifications each. Done when all 11 items carry a score and a justification.
6. Compute — score.py axis per tell-counted axis, then score.py overall with weights and any triggered --cap. Never hand-compute.
7. Write the report — read references/output-template.md and emit exactly that structure to output, ending with the 3–5 prioritized fixes that would move the score most (biggest weighted deltas first; a missing signature usually outranks any single tell).

Gotchas

• The brief overrides the law. If the user or brand explicitly directed a choice (a color, a layout, an effect), it is not a tell — the law itself says the user's word wins 100%. Ask for the brief when the design clearly follows one; note excluded tells in the report with proper justification tags (see Exclusion system below).
• Context flips a tell. Mono on real data is correct; a populated, real-feeling product window is a signature, not the fake-window tell; a tight micro-grid with texture is premium, a full-page graph paper is slop. Always check the premium pair before recording.
• Don't reward the clean miss. Zero tells with a weak signature is the most common failure of designs that tried to avoid slop. The signature gate exists for this — apply it without mercy.
• Severity discipline. Critical is reserved for broken (the six absolute rules). A blue-purple gradient is loud but not broken: major.
• One-axis bleed. Some tells could sit on two axes (cut-off glow is color and execution). The catalog assigns each tell to exactly one axis — count it only there.
• Portfolio tells. L19 (recycling your own house style) needs prior work from the same author to verify; without it, mark Unverifiable rather than guessing.

Exclusion system

Every excluded tell MUST have a justification tag. A tell without a tag counts — no exceptions. This creates an audit trail and prevents lazy exclusions.

Justification tags

Valid vs invalid exclusions

Valid exclusions:

markdownCopy code
| C1 | Blue→purple gradient | `// BRIEF: brand guidelines v2.3 specify #6366f1→#8b5cf6` |
| K14 | Countdown timer | `// DESIGN DECISION: real sale ends 2026-12-31, verified in CMS` |
| T4 | Mono as house voice | `// CONTEXT: SaaS product with code-heavy documentation` |
| K25 | Glass effect | `// PREMIUM PAIR: proper backdrop blur, chromatic dispersion, directional light` |

Invalid exclusions (tell still counts):

markdownCopy code
| C1 | Blue→purple gradient | "we liked it" | ❌ Not a justification
| K9 | Default CTA pair | "it's our style" | ❌ Too vague
| L1 | Default hero stack | "approved by team" | ❌ Who? When? Why?
| K6 | Kitchen-sink card | "industry standard" | ❌ Slop IS the industry standard

Exclusion limits

• >5 exclusions → Review each one. Mass exclusions suggest the brief wasn't followed or the evaluator is being too lenient.
• >10 exclusions → Something is wrong. Either the brief allows nearly everything (in which case, why evaluate?) or exclusions are being used to inflate the score.
• Excluding signature elements → Almost never valid. If S1–S7 are excluded, the design has no signature by definition.

Exclusion documentation in report

In the Excluded tells table, format as:

markdownCopy code
## Excluded tells

| ID | Tell | Exclusion reason |
|----|------|------------------|
| C1 | Blue→purple gradient | `// BRIEF: brand guidelines v2.3 specify #6366f1→#8b5cf6` |
| K14 | Countdown timer | `// DESIGN DECISION: real sale ends 2026-12-31, verified in CMS` |

**Exclusion summary:** 2 tells excluded (1 BRIEF, 1 DESIGN DECISION)

Challenging exclusions

When reviewing someone else's slop report, check exclusions first:

1. Is the tag present? No tag = tell counts.
2. Is the tag appropriate? // BRIEF: needs an actual brief reference.
3. Is the reasoning concrete? Vague reasoning = tell counts.
4. Is the exclusion count reasonable? >5 warrants scrutiny.

Evaluation with Jev (Optional)

When the harness has access to TypeSafe Jev, the subjective scoring steps (Axes 7-8) and Quality Checklist verification can use Jev for calibrated assessment.

Where Jev is used:

• Axis 7 (Signature) — 7 Score questions (S1-S7) with 0/50/100 rubric
• Axis 8 (Cohesion) — 4 Score questions (H1-H4) with 0/50/100 rubric
• Quality Checklist — 27 Noul questions for binary verification

Where Jev is NOT used:

• Axes 1-6 — Tell detection is factual (cite-or-cut); score.py handles arithmetic
• Gates & Caps — Deterministic rules applied by score.py

Discovery Protocol

Copy code
1. MCP Tool `jev_eval` configured in harness → use it
2. Model `typesafe/jev-latest` via OpenRouter → request it
3. Auxiliary slot (Hermes/Devin/Codex) with Jev → delegate
4. Fallback → inline scoring via current LLM using premium-markers.md rubric

Integration Files

Full documentation: See references/jev-integration.md for discovery details, harness-specific instructions, and request/response structures.

Quality checklist

Final gate before delivering. Run through every item — a single failure means the report is not ready. This is the self-evaluation rubric; treat it as a hard gate, not a suggestion.

Pre-sweep checks

• Evidence inventory complete — documented what was captured (code, screenshots, live URL) and what is Unverifiable
• Brief documented — if provided, summarized in report header; if not provided, noted as "no brief"
• Design context identified — what type of design is this? (landing page, SaaS dashboard, editorial, e-commerce). Read references/contexts.md to adjust priorities and tolerances
• All reference files read — tells.md, premium-markers.md, and contexts.md loaded before starting the sweep

During-sweep checks

• Cite-or-cut enforced — every recorded tell has ID + severity + concrete citation (hex value, font name, file:line, or screenshot region)
• Premium pair checked — before recording any tell, verified it's not the crafted premium version of the pattern
• Portability test applied — for borderline cases, asked: "Could this element be moved to another site without alteration?" If yes → tell. If no (it's specific to this brand) → not a tell
• Defense test applied — for borderline cases, asked: "Could the designer defend this choice with concrete reasoning if asked?" If no → tell. Slop cannot be defended; deliberate choices can.
• Section attribution — every tell assigned to a specific section (Hero, Features, Pricing, Footer, etc.) for the Section Ledger
• Severity discipline — critical reserved for absolute-rule violations only; no severity inflation

Exclusion checks

• Exclusions documented — every excluded tell has a // BRIEF: or // DESIGN DECISION: justification
• Exclusions are genuine — "we liked it" or "it looked good" are NOT valid exclusion reasons. Only explicit brief direction or documented design decisions with concrete reasoning qualify.
• Exclusion count reasonable — if >5 tells excluded, double-check each one. Mass exclusions suggest the brief wasn't followed, not that the tells don't apply.

Post-sweep checks

• All unverifiable checks marked — not silently passed or skipped
• All 6 absolute rules reported — pass/fail/unverifiable with evidence
• All 11 signature/cohesion items scored — 0/50/100 with one-line justification each
• Section Ledger complete — every major section has a verdict (CLEAN/SUSPICIOUS/INFLATED/CRITICAL) with tell count and action
• Gates applied correctly:
  • Signature gate: if Axis 7 < 40, overall capped at 59
  • Absolute-rule gate: if any crit, overall capped at 69
  • Compounding cap: if ≥3 major layout tells, Axis 4 capped at 40
• Math from script only — all scoring via score.py, never hand-computed

Report checks

• Template followed exactly — structure matches output-template.md
• Fixes ranked by weighted impact — signature issues (3x weight) typically outrank single tells
• Language correct — report in user's language, tell IDs in English

Final self-audit

Before delivering, ask yourself:

• "What still looks like obvious slop that I didn't flag?" — if something visually screams slop but isn't in your findings, either find the tell that covers it or note it as a gap in the catalog.
• "Did I over-correct?" — a sparse report on a clearly-slop design suggests missed tells. A bloated report on a premium design suggests false positives.
• "Would I trust this report if someone else wrote it?" — read the report as if reviewing a colleague's work. Does every claim hold up?