Fact Check

Claim–Source Fact Checker

This skill extends citation-check (which verifies a citation exists, resolves, and is formatted) to the next question: does the cited source actually support the claim it is attached to? It checks claims against the project's per-source Markdown knowledge base — one Markdown file per cited source, each converted or summarized to sources/md/<author>-<year>-<slug>.md by whatever intake workflow the project uses. This is local-source verification, not open-web fact-checking: it audits whether a manuscript's sentences are backed by the documents the author has actually read and filed. Run once a draft has citations and a populated knowledge base.

Instructions

1. Pre-flight gate — orient, then refuse if the knowledge base is not ready

Orient. Identify the inputs:

• Manuscript source: LaTeX, Markdown, Word/PDF text, or pasted prose with citations or footnotes.
• Bibliography: .bib, CSL JSON, or reference list — used to resolve cite keys to author-year-title so source files can be found.
• Knowledge base: the project's per-source Markdown directory. Search in order: sources/md/, knowledge_base/md/, sources/, knowledge_base/, then any chapter-local */sources/. Also read any index or crosswalk file (sources/inventory.md, notes/source_map.md, *source*crosswalk*, *source*matrix*) that maps bib keys or titles to files.
• Scope: whole manuscript, one section, the literature review/theory sections, or a supplied list of claims.

Pre-flight gate. Claims are verified against local Markdown source files, not from memory or the open web. A fact-check run against an absent or half-built knowledge base gives false reassurance, which is worse than no check. Return a Pre-flight: NOT READY notice instead of a report when any of the following holds:

1. No knowledge base. No per-source Markdown corpus exists: sources/md/, knowledge_base/md/, or an equivalent is missing or empty.
2. Unconverted sources (not cleaned up). Raw documents — .pdf, .docx, .epub, or scanned images — sit in the sources area (a drop-zone, sources/, sources/pdf/, */source_pdfs/) without a corresponding .md. The base is not clean: every source must be converted to Markdown first (or, if long, to a Markdown summary).
3. Coverage too low. The available Markdown does not cover the cited works in scope. Compute coverage = matched source files ÷ cited non-background works; below roughly two-thirds, treat as not ready.

Do not fact-check around the gap or fill missing sources from memory or the web. The user can fix the knowledge base and rerun, or explicitly rescope the run to the covered subset; in that case proceed with the subset and state the rescoping in the report.

Remediation to print on refusal:

• Build a per-source Markdown knowledge base — one file per cited source, e.g. produced by scripts/convert-sources.sh in a repo laid out by the research-repo skill, or by your own source-intake workflow. Each raw source becomes sources/md/<author>-<year>-<slug>.md, summarized if long.
• Populate any cited work that has no file yet: locate the PDF, then run it through the same intake step.
• Re-run fact-check once sources/md/ covers the cited set and no raw files remain unconverted.
• The knowledge-base layout and intake conventions are documented in the Open Science Skills repo (github.com/scdenney/open-science-skills) and in research-repo.

2. Run citation-check first

Before judging support, run the citation-check skill on the same inputs and consume its report. A claim cannot be evaluated against a source that is missing, fabricated, or mis-keyed. Carry its findings forward: any LIKELY FABRICATED, DOI RESOLVES TO DIFFERENT WORK, or in-text/reference parity break makes the dependent claim UNVERIFIABLE — citation problem, not a clean support check. Cite its result rather than repeating its checks.

3. Build the claim inventory

For each in-text citation, parenthetical, narrative cite, or footnote that attaches a source to a substantive statement, record:

• the claim — the exact sentence or clause the citation backs;
• the cite key(s) / author-year;
• the claim type — empirical result, descriptive fact, definition/concept, method precedent, theoretical position, or direct quotation;
• the asserted strength — the verb does work: "shows" / "demonstrates" / "proves" claim more than "suggests" / "is consistent with" / "see".

Skip pure background or "see also" citations unless they carry a checkable assertion. For multi-source claims, check each source's contribution separately — a claim can be supported by one cited work and not another.

4. Locate each source's Markdown file

Resolve bib key to author + year + title, then find the matching knowledge-base file:

• Match on author(s)-year plus a title slug. Naming varies across projects (hyphen-separated or underscore_separated; -and- between co-authors; first-author-only vs. all authors). Be tolerant of diacritics, name particles (de, van, von), and short vs. full titles.
• Prefer an explicit crosswalk/inventory entry when one exists over fuzzy filename matching.
• These files are frequently summaries or PDF→Markdown conversions, not full text. Note which: a faithful conversion can adjudicate fine-grained claims; a condensed summary may legitimately omit the detail a claim rests on.
• Record one of FOUND (with path), AMBIGUOUS (more than one candidate — list them, do not guess), or NOT IN KB.

5. Verify claim support

Read the located source Markdown and judge whether it supports the specific claim. Classify each:

• SUPPORTED — the source states or directly entails the claim. Quote the supporting passage.
• PARTIALLY SUPPORTED — the source supports a weaker or narrower version. Name the gap (scope, population, magnitude, hedging).
• UNSUPPORTED — the source does not contain the claim; the passage that should support it is absent.
• CONTRADICTED — the source reports the opposite or a materially different result (wrong direction, wrong sign, wrong magnitude).
• MISATTRIBUTED — the claim is real but belongs to a different work than the one cited.
• SOURCE INSUFFICIENT — the Markdown is a summary that does not carry enough detail to adjudicate; name what to check in the original PDF.
• NOT IN KB / UNVERIFIABLE — citation problem — carried from steps 2 and 4.

For every SUPPORTED, PARTIALLY SUPPORTED, or CONTRADICTED verdict, include a verbatim quote from the source (with section/page if the file carries them). Never assert support from memory, topical overlap, or plausibility.

Actively watch for:

• Overclaiming — source hedges ("may", "in this sample"), manuscript asserts a general law.
• Direction errors — source finds a null or opposite effect from what the sentence claims.
• Scope creep — a single-country, single-period, or single-population source cited for a universal claim.
• Citation stuffing — a source piled onto a claim it does not actually address.
• Stale quantitative values — numbers, point estimates, or sample sizes that do not match the source.
• Definitional drift — a concept attributed to a source that defines or uses it differently.

6. Calibrate the verdicts

• A faithful summary that simply omits a detail is SOURCE INSUFFICIENT, not UNSUPPORTED.
• Background and framing citations are not evidentiary claims; do not demand data from them.
• Theory/position citations are supported when the source advances that position, even with no data.
• A missing source Markdown is an infrastructure gap (NOT IN KB → convert and file the source), not evidence of fabrication. Keep "the source doesn't support the claim" strictly separate from "I couldn't find the source."

Output

If the pre-flight gate failed, return only this notice:

Copy code
# Fact-Check — Pre-flight: NOT READY

Reason: <no knowledge base | N unconverted sources | coverage X% below bar>
Knowledge base searched: <dirs>
Cited works with no Markdown file: <cite keys / author-year>
Raw files needing conversion: <paths>

## To proceed
1. Convert and file the raw files / missing sources listed above into the Markdown knowledge base
2. Confirm each cited work has sources/md/<author>-<year>-<slug>.md
3. Re-run /oss:fact-check
Best practices: github.com/scdenney/open-science-skills · the research-repo skill

Otherwise, produce a Fact-Check Report:

Copy code
# Fact-Check Report

Scope: <files/sections checked>
Knowledge base: <dir(s) searched> · sources matched: <n>/<total cited>
citation-check: <ran? key carry-forward findings>
Summary: <N contradicted, N unsupported, N partial, N supported, N not-in-KB, N unverifiable>

## Contradicted / Unsupported (blocking)
| Location | Claim (short) | Cite key | Verdict | Source passage or "absent" | Suggested fix |

## Partially supported / Overclaim (recommended)
| Location | Claim | Cite key | Gap | Source passage | Suggested rewording |

## Source not in knowledge base
| Cite key | Author-year | Why unresolved | Action (convert and file / locate file) |

## Supported (spot-check log)
| Location | Claim | Cite key | Source passage |

Severity:

• Blocking: a decisive or headline claim is CONTRADICTED or UNSUPPORTED; a reported quantitative result does not match the source; a citation-check failure that invalidates the claim.
• Recommended: overclaiming or partial support fixable by rewording or rescoping; misattribution where the correct source is known.
• Minor: a background citation that could be tightened; a summary-only source worth re-checking against its PDF before submission.

Optional Jev verification (--verify)

/oss:fact-check --verify <task> opts into experimental Jev verification; /oss:fact-check --verify jev <task> is identical. Plain /oss:fact-check is unchanged and makes no Jev call. This is an advisory second look at one already-adjudicated claim/source pair — never a fact source, never a replacement for this workflow, and never a replacement for the report labels. An audit that finds errors is already complete without it.

Run the pre-flight gate (step 1) before any Jev action, then run citation-check first as usual and carry its findings forward. Refer to that report rather than repeating provider checks or sending duplicate claim-support evidence. If source identity is still genuinely ambiguous, request /oss:citation-check --verify for that entry, reusing the citation inventory and fetched records. A lead may manually reuse a prior result only for the same complete evaluation input and model/contract version; nothing here caches automatically.

Only on opt-in, read ${JEV_VERIFIER_HOME:-$HOME/.local/share/oss-experiments/jev-verifier/current}/FACT-CHECK.md and follow it; that shared runtime file is authoritative for the state manifest, the verify.py call, and the disclosure you owe the user before a live call. Installation and credentials are covered by ${JEV_VERIFIER_HOME:-$HOME/.local/share/oss-experiments/jev-verifier/current}/README.md in the same bundle; if that bundle is not present, the runtime is not installed and verification is blocked. Do not call Jev for a failed pre-flight, an absent knowledge base, invalid or unchecked citation integrity, or insufficient or unchecked evidence. Record your own report label and verbatim source passage first, either way. A missing runtime, an unresolvable TYPESAFE_API_KEY, or a failed call means the requested check is blocked, never verified and never mocked.