Golden Dataset

Golden Dataset

Comprehensive patterns for building, managing, and validating golden datasets for AI/ML evaluation. Each category has individual rule files in rules/ loaded on-demand.

Quick Reference

Total: 6 rules across 4 categories. House thresholds and scars: references/ork-delta.md.

Curation

Content collection, multi-agent annotation, and diversity analysis for golden datasets.

Difficulty ladder, coverage floors, and duplicate thresholds: references/ork-delta.md.

Management

Versioning, storage, and CI/CD automation for golden datasets.

CI automation for backups is upstream's job; see "Upstream coverage" below.

Validation

Quality scoring, drift detection, and regression testing for golden datasets.

Schema validation and duplicate detection are upstream's job (see "Upstream coverage" below); the house thresholds they must enforce live in references/ork-delta.md.

Add Workflow

Structured workflow for adding new documents to the golden dataset.

Quick Start Example

pythonCopy code
async def validate_before_add(document: dict, source_url_map: dict) -> dict:
    """Pre-addition validation for golden dataset entries."""
    errors = []

    # 1. URL contract check
    if "placeholder" in document.get("source_url", ""):
        errors.append("URL must be canonical, not a placeholder")

    # 2. Content quality
    if len(document.get("title", "")) < 10:
        errors.append("Title too short (min 10 chars)")

    # 3. Tag requirements
    if len(document.get("tags", [])) < 2:
        errors.append("At least 2 domain tags required")

    return {"valid": len(errors) == 0, "errors": errors}

Key Decisions

Common Mistakes

1. Using placeholder URLs instead of canonical source URLs
2. Skipping embedding regeneration after restore
3. Not validating referential integrity between documents and queries
4. Over-indexing on articles (neglecting tutorials, research papers)
5. Missing difficulty distribution balance in test queries
6. Not running verification after backup/restore operations
7. Testing restore procedures in production instead of staging
8. Committing SQL dumps instead of JSON (not version-control friendly)

Running a dataset as an experiment

Curating a dataset is half the job; the other half is running something against it and scoring the result. Both Langfuse SDKs ship a runner, and their shapes differ.

Python (SDK 4.x): see monitoring-observability/references/experiments-api.md.

JS/TS (SDK 5.x): @langfuse/client exposes the runner directly on a fetched dataset.

typescriptCopy code
import { LangfuseClient } from "@langfuse/client";

const langfuse = new LangfuseClient();
const dataset = await langfuse.dataset.get("my-evaluation-dataset");

const result = await dataset.runExperiment({
  name: "Retrieval quality",
  task: myTask,               // (params) => Promise<any>
  evaluators: [myEvaluator],  // per-item: (params) => Promise<Evaluation | Evaluation[]>
});

A per-item Evaluator cannot see the other items, so anything comparative belongs in a RunEvaluator. createEvaluatorFromAutoevals wraps an autoevals scorer instead of hand-writing one, and RegressionError is thrown when a run regresses against a configured baseline — catch it to fail CI on a quality drop rather than only on an exception.

Full JS surface: monitoring-observability/references/langfuse-js-v5.md.

Evaluations

See test-cases.json for 9 test cases across all categories.

Upstream coverage (do not restate)

House thresholds these must enforce: references/ork-delta.md.

Related Skills

• ork:rag-retrieval - Retrieval evaluation using golden dataset
• ork:monitoring-observability - Langfuse tracing patterns for curation workflows
• ork:testing-llm - Evaluation harnesses that consume golden datasets
• ork:testing-unit - Unit testing patterns and strategies

Capability Details

curation

Keywords: golden dataset, curation, content collection, annotation, quality criteria

Solves:

• Classify document content types for golden dataset
• Run multi-agent quality analysis pipelines
• Generate test queries for new documents

management

Keywords: golden dataset, backup, restore, versioning, disaster recovery

Solves:

• Backup and restore golden datasets with JSON
• Regenerate embeddings after restore
• Automate backups with CI/CD

validation

Keywords: golden dataset, validation, schema, duplicate detection, quality metrics

Solves:

• Validate entries against document schema
• Detect duplicate or near-duplicate entries
• Analyze dataset coverage and distribution gaps