Research Repo

Research Repository Scaffold & Audit

Scope and organizing principle

This skill sets up a new research repository, or audits an existing one, around a single organizing principle: the source library is the spine of the project, and everything else grows from it.

sources/og/ holds the original PDFs and documents — archival, gitignored (size + copyright), rarely read directly. sources/md/ holds their Markdown conversions, tracked in git: this is the LLM-readable knowledge base, and sources are always read from here, not from the PDFs. sources/unprocessed/ is the drop zone where new PDFs land until they are named, converted, and filed. sources/inbox/ sits beside it for notes, links, screenshots and loose files that feed the knowledge base without going through intake: agents read them as context and build on them, but process-source never converts or moves them; a file enters the corpus only when the author moves it into unprocessed/. sources/references.bib keys the corpus to the manuscript — one entry per source, author+year resolvable to its md/ file, so a \cite key maps to a document actually read. A conversion script (scripts/convert-sources.sh, OpenDataLoader PDF) and a process-source intake command keep the three in sync. Get that spine wrong — PDFs scattered, no Markdown, a bib that does not match what was read — and every downstream skill (literature-review, citation-check, fact-check, paper-tex) is working on sand.

Everything else — data/, analysis scripts/, manuscript/ or paper/, review/ + codebook/, figures/, tables/, replication/ — grows outward from that spine, conditioned on what kind of project this is.

Instructions

Step 1. Resolve the target and decide the mode

Use $ARGUMENTS as the repo path if given; otherwise use the current working directory. Then decide the mode from what is there:

• Scaffold mode — the directory is empty, is not yet a git repo, or has no sources/ and no manuscript. You will build the spine and the outward folders.
• Audit mode — the directory already has research content (a sources/ tree, a manuscript, analysis scripts, a bib). You will read what exists, compare it against the convention, and report present / partial / missing. Overwrite an existing file only with the user's confirmation; offer to create only what is missing.

If a repo is half-built (e.g. PDFs exist but no md/, or an og/ with no convert script), that is audit mode with scaffolding gaps — report the gaps and offer to fill them.

Step 2. Identify the project archetype

The bibliography is the spine of every project; the outward folders — and whether a PDF corpus is even appropriate — depend on what the project is. Detect the archetype from concrete signals in the directory, or ask one question if it is genuinely ambiguous.

Most projects have a literature corpus and head to a paper; when in doubt, scaffold the spine plus data/, scripts/, and the manuscript folder, and let the project grow the rest. But some repos legitimately keep no PDF corpus — a theory paper, or a short empirical paper with a hand-maintained .bib. There the spine is the bibliography alone: audit the .bib wherever it lives (often paper/references.bib) and offer — never impose — a sources/og/sources/md tree.

Step 3. Scaffold the sources spine

For any project that reads and archives a literature, this is the core. (For a corpus-free repo — the lightweight/theory archetype in Step 2 — scaffold only the bibliography and the manuscript folder, then skip to Step 4.) Create:

textCopy code
<repo>/
├── sources/
│   ├── og/                 # original PDFs/docs — gitignored
│   ├── md/                 # Markdown conversions — tracked
│   ├── unprocessed/        # drop zone for new PDFs
│   ├── inbox/              # notes read as context, never converted
│   ├── references.bib      # bibliography (one entry per source)
│   └── README.md           # the convention, written down
├── scripts/
│   └── convert-sources.sh  # PDF/docx → Markdown (OpenDataLoader PDF + anydoc)
├── .claude/
│   └── commands/
│       └── process-source.md   # per-PDF intake command
├── CLAUDE.md               # project conventions for coding agents
├── AGENTS.md -> CLAUDE.md  # symlink so Codex/others read the same file
├── .gitignore
└── .venv/                  # python env for opendataloader-pdf (created in setup)

Make sure the repo is under version control first; the tracked/gitignored split (og/ ignored, md/ tracked) only takes effect once git exists:

bashCopy code
git rev-parse --git-dir >/dev/null 2>&1 || git init

Write the templates from the Templates section below. A newly written file is mode 0644, so make the conversion script executable right after writing it; otherwise the smoke test and every ./scripts/convert-sources.sh instruction in the scaffold fail with "permission denied":

bashCopy code
chmod +x scripts/convert-sources.sh

Then set up the conversion environment, checking the toolchain first: convert-sources.sh runs under set -euo pipefail and aborts opaquely if the Java backend is missing.

bashCopy code
command -v python3 && python3 --version
command -v java && java -version          # OpenDataLoader PDF needs Java 11+; if absent, stop and tell the user to install it
python3 -m venv .venv
.venv/bin/pip install --upgrade pip opendataloader-pdf

Link AGENTS.md to CLAUDE.md so a second agent family (e.g. Codex) reads the same root instructions:

bashCopy code
test -e AGENTS.md || ln -s CLAUDE.md AGENTS.md

Finally, smoke-test the pipeline — on an empty og/ it should print Nothing new to convert.:

bashCopy code
./scripts/convert-sources.sh

Step 4. Build outward (archetype-conditioned)

Create only the outward folders the archetype calls for (Step 2). Leave them empty with a one-line purpose in the README — the project fills them. Common folders and what they hold:

• data/ — datasets. Raw inputs and derived analysis-ready files. Large/binary data is gitignored (see template); document restricted data rather than committing it.
• scripts/ — analysis and utility code (the convert script already lives here).
• review/ + codebook/ (review projects) — the inventory CSV (one row per source), per-source prose annotations, and the coding protocol that governs them. This is where the sources/md/ corpus gets turned into structured evidence.
• manuscript/ or paper/ — the draft. Cites sources/references.bib. Hand this to paper-tex to typeset.
• figures/, tables/ — generated outputs. Pair with the figures / tables skills.
• replication/ — the public reproducibility package. Leave it empty here; when the paper is ready, replication-package scaffolds it.
• logs/, meetings/ — session logs and notes; usually local-only (gitignored).

Step 5. Document the intake pipeline

The spine only stays trustworthy if every new source flows through the same pipeline. Write it into sources/README.md and the process-source command, and state it to the user:

Copy code
drop in sources/unprocessed/  →  identify (title/authors/year/venue)
  →  rename to author-year-slug, move to sources/og/
  →  ./scripts/convert-sources.sh  (→ sources/md/<name>.md)
  →  add a BibTeX entry to sources/references.bib
  →  (review projects: add inventory row + annotation)

The per-PDF mechanics belong to process-source, which the scaffolded process-source.md command points at. This skill guarantees the pipeline exists and is wired; process-source runs it.

Step 6. The BibTeX contract

The bibliography — sources/references.bib, or wherever the project keeps it (e.g. paper/references.bib) — is the contract between the manuscript and the knowledge base: a \cite key is only trustworthy if it resolves to a source you have actually read and filed in sources/md/. Keep it honest:

• One entry per source, added at intake (Step 5), never in a batch at the end.
• Citekey follows the project's own key style (e.g. hainmueller_hopkins_yamamoto_2014 or hainmueller-etal-2014-conjoint). Do not impose a scheme on an existing project — but keep every key author+year resolvable to its md/ filename, because citation-check and fact-check map keys to source files by author and year. The filename uses author-year-slug; the bib key can differ in punctuation but must point at the same work.
• sources/missing.bib (optional, recommended) — a second bib for works that are cited but have no PDF in og/: paywalled articles with no preprint mirror, books, dissertations, authoritative web resources (a standard, a DOI registry). Record why each is missing and how to acquire it, so "cited but unfiled" stays visible instead of silently absent. A needs_updates.md-style flag file serves the same purpose if the project already keeps one.

Step 7. Audit checklist (audit mode)

Read the existing repo and report each item as present / partial / missing. If the repo keeps no PDF corpus by design (the lightweight/theory archetype), mark the PDF-corpus items n/a for this archetype, not missing, and audit the bibliography wherever it actually lives — discover it with find . -name '*.bib' -not -path './.venv/*' -not -path './.git/*'.

Spine

• sources/og/, sources/md/, sources/unprocessed/, sources/inbox/ all exist.
• sources/og/ is gitignored; sources/md/ is tracked (the LLM-readable corpus must be in git).
• Every og/*.pdf has a matching md/*.md (no unconverted sources). List the orphans.
• Every md/*.md has a references.bib entry, and every bib entry resolves to a source (flag bib entries with no file and files with no bib — drift in either direction).
• Filenames follow author-year-slug (lowercase, hyphens, ≤3 authors then firstauthor-etal).
• sources/references.bib exists; keys are author+year resolvable to filenames.
• sources/README.md documents the convention.

Detection recipes (run from the repo root; approximate starting points, not gospel):

bashCopy code
# Orphan PDFs — in og/ but never converted to md/
comm -23 <(cd sources/og && ls *.pdf 2>/dev/null | sed 's/\.pdf$//' | sort) \
         <(cd sources/md && ls *.md  2>/dev/null | sed 's/\.md$//'  | sort)

# Bib keys — compare against md/ stems by author+year to catch drift in either
# direction (a source with no entry, an entry with no source). Keys may use
# underscores where filenames use hyphens.
grep -oE '^@[a-zA-Z]+\{[^,]+' sources/references.bib | sed 's/^@[a-zA-Z]*{//' | sort

# Filenames that violate author-year-slug (lowercase-hyphen, four-digit year, slug)
ls sources/og | grep -vE '^[a-z0-9]+(-[a-z0-9]+)*-(19|20)[0-9]{2}-[a-z0-9-]+\.(pdf|docx)$'

Pipeline

• scripts/convert-sources.sh exists and points at this repo's sources/og → sources/md.
• .venv/ with opendataloader-pdf is set up (or setup is documented); Java 11+ available.
• A process-source command (or the global skill) is available for intake.

Repo conventions

• CLAUDE.md documents the project and its conventions; AGENTS.md is a symlink to it (or a copy).
• .gitignore excludes sources/og/, .venv/, large data, secrets, and OS/editor cruft.
• Outward folders match the archetype and have a stated purpose (README).
• No secrets, answer keys, or restricted data in the tracked path.

Step 8. Report

Output a short report:

1. Mode (scaffold / audit) and detected archetype.
2. The tree created (scaffold) or the present/partial/missing diff (audit).
3. For audits: orphan PDFs, bib drift, naming violations, and pipeline gaps, each with a one-line fix.
4. The next three actions (typically: set up .venv and run convert-sources.sh; drop the first PDFs in unprocessed/ and run process-source; fill in CLAUDE.md placeholders).

Do not commit unless asked.

Templates

sources/README.md

markdownCopy code
# Sources

The source library is the spine of this project. Read sources from `md/`, never the PDFs.

## Structure

- `og/` — original PDFs and documents (gitignored; not pushed)
- `md/` — LLM-readable Markdown conversions (tracked in git)
  - `<name>_images/` folders hold figures/tables extracted from the PDF
- `unprocessed/` — drop zone for new PDFs awaiting intake
- `inbox/` — notes, links, screenshots and loose files that feed the knowledge base without going through intake; read as context, never converted. Move a file to `unprocessed/` when it should enter the corpus
- `references.bib` — bibliography (one entry per source)
- `missing.bib` — cited works with no PDF available (paywalled, books, web resources)

## Naming convention

`author-year-slug`, lowercase with hyphens:

- Up to three authors → use all three; more → `firstauthor-etal`
- Then year, then a 2–4 word content slug
- Example: `hainmueller-hopkins-yamamoto-2014-causal-inference-conjoint`

The PDF in `og/` carries this name through to `md/`.

## Adding a source

1. Drop the PDF (or `.docx`) into `sources/unprocessed/`
2. Run the `process-source` command (or do it by hand): rename to `author-year-slug`,
   move to `og/`, run `./scripts/convert-sources.sh`, add a `references.bib` entry
3. The convert script only touches files without a matching `.md` — re-running is safe
4. Commit the new `md/<name>.md` (and its `_images/`); `og/` stays local

## Requirements

- Python venv at `.venv/` with `opendataloader-pdf` installed
- Java 11+ (OpenDataLoader PDF backend) · `poppler-utils` for `pdfinfo`/`pdftotext` (the image-only guard)
- Node (for `npx @firecrawl/anydoc`, which handles `.docx`/`.pptx`/`.xlsx`/`.odt`/`.rtf`/`.epub`), or `pandoc` as the fallback

```bash
python3 -m venv .venv
.venv/bin/pip install --upgrade pip opendataloader-pdf
```

## Why Markdown?

PDFs are awkward for LLMs. The Markdown versions preserve text, headings, and
references in a form agents can read directly, enabling source-grounded analysis
and writing. OpenDataLoader PDF: https://github.com/opendataloader-project/opendataloader-pdf

scripts/convert-sources.sh

bashCopy code
#!/usr/bin/env bash
# Convert new PDFs/docx in sources/og/ to Markdown in sources/md/.
# Skips files that already have a corresponding .md.
# Usage: ./scripts/convert-sources.sh          # incremental (new files only)
#        ./scripts/convert-sources.sh --all     # reconvert everything
set -euo pipefail

PROJECT_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
SOURCES="$PROJECT_ROOT/sources/og"
OUTPUT="$PROJECT_ROOT/sources/md"
VENV="$PROJECT_ROOT/.venv"

FORCE=false
[[ "${1:-}" == "--all" ]] && FORCE=true

mkdir -p "$OUTPUT"
source "$VENV/bin/activate"

converted=0
needs_ocr=0

# Expand typographic ligatures. OpenDataLoader PDF leaves them raw, which makes the
# Markdown un-greppable ("significant" never matches "significant"). Deliberately not
# full NFKC, which would also flatten superscript footnote markers into body digits.
fix_ligatures() {
    sed -i'' -e 's/ff/ff/g; s/fi/fi/g; s/fl/fl/g; s/ffi/ffi/g; s/ffl/ffl/g; s/ſt/ft/g; s/st/st/g' "$1"
}

# Image-only PDFs need OCR. Without this guard opendataloader-pdf exits 0 and writes
# a few KB of noise, which then reads downstream as a successful conversion.
is_image_only() {
    command -v pdftotext >/dev/null 2>&1 && command -v pdfinfo >/dev/null 2>&1 || return 1
    local pages chars
    pages="$(pdfinfo "$1" 2>/dev/null | awk '/^Pages:/{print $2}')"
    [[ -n "$pages" && "$pages" -gt 0 ]] || return 1
    chars="$(pdftotext -q "$1" - 2>/dev/null | wc -c)"
    [[ $((chars / pages)) -lt 300 ]]
}

# PDFs via OpenDataLoader PDF
for pdf in "$SOURCES"/*.pdf; do
    [ -f "$pdf" ] || continue
    base="$(basename "$pdf" .pdf)"
    if [[ "$FORCE" == false && -f "$OUTPUT/$base.md" ]]; then
        continue
    fi
    if is_image_only "$pdf"; then
        echo "  NEEDS OCR (image-only): $base.pdf"
        needs_ocr=$((needs_ocr + 1))
        continue
    fi
    echo "Converting $base.pdf..."
    opendataloader-pdf "$pdf" --format markdown --output-dir "$OUTPUT/"
    [ -f "$OUTPUT/$base.md" ] && fix_ligatures "$OUTPUT/$base.md"
    converted=$((converted + 1))
done

# Office and e-book formats via anydoc, falling back to pandoc without Node
for doc in "$SOURCES"/*.docx "$SOURCES"/*.doc "$SOURCES"/*.odt "$SOURCES"/*.rtf \
           "$SOURCES"/*.pptx "$SOURCES"/*.xlsx "$SOURCES"/*.epub; do
    [ -f "$doc" ] || continue
    base="${doc##*/}"; base="${base%.*}"
    if [[ "$FORCE" == false && -f "$OUTPUT/$base.md" ]]; then
        continue
    fi
    echo "Converting $(basename "$doc")..."
    if command -v npx >/dev/null 2>&1; then
        npx -y @firecrawl/anydoc "$doc" -o "$OUTPUT/$base.md"
    else
        pandoc "$doc" -t markdown -o "$OUTPUT/$base.md"
    fi
    converted=$((converted + 1))
done

if [[ $converted -eq 0 && $needs_ocr -eq 0 ]]; then
    echo "Nothing new to convert."
else
    echo "Converted $converted file(s), $needs_ocr need OCR. Markdown in $OUTPUT/"
fi

Files reported as NEEDS OCR are image-only scans; route them through vlm-ocr rather than re-running this script. For large corpora, a parallel variant (xargs -P N over opendataloader-pdf) speeds bulk conversion; the incremental script above is the default and is safe to re-run.

.claude/commands/process-source.md

markdownCopy code
---
description: Process new papers from sources/unprocessed/ through the full intake pipeline
---

# Process unprocessed sources

Check `sources/unprocessed/` for new PDFs/docs. For each one:

1. **Identify** — read it; determine title, authors, year, venue.
2. **Rename & move** — to `sources/og/` using `author-year-slug` (see `sources/README.md`).
3. **Convert** — run `./scripts/convert-sources.sh` to generate `sources/md/<name>.md`.
4. **BibTeX** — add an entry to `sources/references.bib` (author+year resolvable to the filename).
5. **(Review projects)** — classify per `codebook/` and add an inventory row + annotation.
6. **Clean up** — remove the file from `sources/unprocessed/`.

Flag anything needing human review (paywalled, poor OCR, data-integrity concerns).
If a cited work has no obtainable PDF, record it in `sources/missing.bib` instead.

Prefer this project's conventions; defer to the global `process-source` skill for the generic mechanics.

CLAUDE.md

markdownCopy code
# <Project name>

<One paragraph: what this project is, who it is for, and its current phase.>

## Repository structure

The source library is the spine. See `sources/README.md` for the intake convention.

- `sources/md/` — LLM-readable Markdown corpus. **Always read sources from here, not the PDFs.**
- `sources/unprocessed/` — drop zone for new PDFs.
- `sources/inbox/` — notes and loose files that feed the knowledge base; read them as context when working on the wiki, but never convert or move them.
- `sources/references.bib` — bibliography; the citation contract for the manuscript.
- `scripts/` — conversion and analysis code.
- <archetype folders: review/ + codebook/  |  data/ + paper/ + figures/ + tables/ + replication/>

## Conventions

- Source naming: `author-year-slug` (lowercase, hyphens).
- New sources flow through the intake pipeline (`/process-source`), never dropped straight into `md/`.
- Read from `sources/md/`; cite from `sources/references.bib`.
- Large data, original PDFs, `.venv/`, and secrets are gitignored.

.gitignore

textCopy code
# Original PDFs and large binaries — keep local, don't push
sources/og/
*.pdf
*.xlsx
*.dta
*.7z
*.zip

# Large data files (keep code, not data)
data/**/*.csv
data/**/*.tsv
data/**/*.png

# Python environment
.venv/

# Sensitive — never commit
.env
.env.*
*.pem
*.key
*ANSWER_KEY*
*do_not_share*

# Local-only working areas
logs/
.claude/settings.local.json

# macOS / R / editors
.DS_Store
.Rhistory
.vscode/
.idea/

Adjust the data rules to the project: if a dataset is public and small, track it; if it is large or restricted, gitignore it and document it in the README. The default keeps original PDFs out of git (copyright + size) while keeping the md/ corpus tracked.

When to reach for this skill vs. siblings

• research-repo (this) — create or audit the working repository's structure, anchored on sources/. Use at project start, or when a repo has grown messy.
• process-source (a global skill, not part of this plugin) — run the per-PDF intake into the structure this skill creates. Use every time a new paper arrives.
• replication-package — scaffold or audit the public reproducibility package built from the finished paper, near submission. This skill creates the working repo; that one creates the archive.
• literature-review / citation-check / fact-check — consumers of the sources/md/ + references.bib knowledge base this skill establishes.