Source Intake
The core problem
Research materials accumulate in chaotic ways: some arrive as standalone PDFs, some inside zip archives, some as image folders, some as web pages saved to disk. Processing them without first understanding what you have creates a false baseline — you think you've ingested everything when the most important items are still locked inside a zip file.
Rule zero: expand all archives before claiming anything is ingested.
Object types
Before assigning an ID to anything, classify it:
| Type | Description |
|---|---|
source_pdf | A PDF that can be processed directly |
derived_pdf | A PDF you generate from images, scans, or web pages |
image_group | A folder of images that form a logical unit (scanned pages, figures) |
dataset | Structured data files (CSV, XLSX, JSON, GIS) |
reference_file | Rights, permissions, licenses, or metadata files |
archive | ZIP, RAR, or other compressed container — expand first, then classify contents |
ambiguous | Needs a decision before it can be registered |
Workflow
1. Inventory before touching
Before doing anything else, list what you have and where it came from. Keep a raw inventory file. Do not rename or move anything in the raw layer.
2. Expand all archives
Expand every archive into a staging area. Note the parent archive ID for each extracted file so you can trace it back later.
Update these manifests after expansion:
expanded_inventory— what came out of which archivepending_pdf_candidates— PDFs waiting for registrationpending_image_groups— image folders waiting for a handling decision
3. Check for duplicates
Before assigning a new ID, check whether this item is already registered. Match on: DOI or URL first → title + year → manual review.
4. Assign stable IDs
Once an item is confirmed unique and classified, assign a stable ID. Recommended format:
- Raw objects:
RAW-####(source as received, untouched) - Processed source objects:
SRC-####(stable, normalized copy)
Keep both IDs in your manifest so the chain from raw to processed is always traceable.
5. Normalize
Create one clean, stable copy of each source object:
- PDF: copy with a stable filename into your normalized folder
- Image group: document the group strategy before doing anything else (will this become a derived PDF, or stay as individual images?)
- Dataset: register and note the format; do not force into PDF
- Web page: preserve the URL and access date; create a PDF capture if needed
6. Record derivatives separately
If you generate page images, OCR text, or other derivatives from a source object, record them in a separate manifest — not in the raw inventory. Keep the chain: raw → normalized → derivative.
Rules
- Never rename or overwrite files in the raw layer. All transformations happen in a separate staging or working layer.
- Do not register a source object before checking for duplicates.
- Do not force image groups into PDF form by default — decide the handling strategy first.
- Do not force datasets, license files, or ambiguous objects into PDF form.
- Manifests must make parent–child relationships explicit (which archive did this PDF come from?).
- Record the source before generating any derivatives.
- If a source is ambiguous, mark it
status: pending_decisionrather than guessing.
Manifest minimum fields
For each registered source object, record at minimum:
- stable ID (
SRC-####) - object type (from the type table above)
- original raw path
- normalized/working path
- parent raw ID (if from an archive)
- language
- status
- notes
References
- Read
references/naming-conventions.mdfor ID formats, filename patterns, and status vocabulary. - Read
references/archive-first.mdfor why archive expansion must come before registration, and common mistakes when skipping it.


