03 Ref Hallucination Arena

Reference Hallucination Arena Skill

Evaluate how accurately LLMs recommend real academic references using the OpenJudge RefArenaPipeline:

1. Load queries — from JSON/JSONL dataset
2. Collect responses — BibTeX-formatted references from target models
3. Extract references — parse BibTeX entries from model output
4. Verify references — cross-check against Crossref / PubMed / arXiv / DBLP
5. Score & rank — compute verification rate, per-field accuracy, discipline breakdown
6. Generate report — Markdown report + visualization charts

Prerequisites

bashCopy code
# Install OpenJudge
pip install py-openjudge

# Extra dependency for ref_hallucination_arena (chart generation)
pip install matplotlib

Gather from user before running

Quick start

CLI

bashCopy code
# Run evaluation with config file
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Resume from checkpoint (default behavior)
python -m cookbooks.ref_hallucination_arena --config config.yaml --save

# Start fresh, ignore checkpoint
python -m cookbooks.ref_hallucination_arena --config config.yaml --fresh --save

# Override output directory
python -m cookbooks.ref_hallucination_arena --config config.yaml \
  --output_dir ./my_results --save

Python API

pythonCopy code
import asyncio
from cookbooks.ref_hallucination_arena.pipeline import RefArenaPipeline

async def main():
    pipeline = RefArenaPipeline.from_config("config.yaml")
    result = await pipeline.evaluate()

    for rank, (model, score) in enumerate(result.rankings, 1):
        print(f"{rank}. {model}: {score:.1%}")

asyncio.run(main())

CLI options

Minimal config file

yamlCopy code
task:
  description: "Evaluate LLM reference recommendation capabilities"

dataset:
  path: "./data/queries.json"

target_endpoints:
  model_a:
    base_url: "https://api.openai.com/v1"
    api_key: "${OPENAI_API_KEY}"
    model: "gpt-4"
    system_prompt: "You are an academic literature recommendation expert. Recommend {num_refs} real papers in BibTeX format. Only recommend papers you are confident actually exist."

  model_b:
    base_url: "https://dashscope.aliyuncs.com/compatible-mode/v1"
    api_key: "${DASHSCOPE_API_KEY}"
    model: "qwen3-max"
    system_prompt: "You are an academic literature recommendation expert. Recommend {num_refs} real papers in BibTeX format. Only recommend papers you are confident actually exist."

Full config reference

task

dataset

target_endpoints.<name>

verification

evaluation

output

report

Dataset format

Each query in the JSON/JSONL dataset:

jsonCopy code
{
  "query": "Please recommend papers on Transformer architectures for NLP.",
  "discipline": "computer_science",
  "num_refs": 5,
  "language": "en",
  "year_constraint": {"min_year": 2020}
}

Official dataset: OpenJudge/ref-hallucination-arena

Interpreting results

Overall accuracy (verification rate):

• > 75% — Excellent: model rarely hallucinates references
• 60–75% — Good: most references are real, some fabrication
• 40–60% — Fair: significant hallucination, use with caution
• < 40% — Poor: model frequently fabricates references

Per-field accuracy:

• title_accuracy — % of titles matching real papers
• author_accuracy — % of correct author lists
• year_accuracy — % of correct publication years
• doi_accuracy — % of valid DOIs

Verification status:

• VERIFIED — title + author + year all exactly match a real paper
• SUSPECT — partial match (e.g. title matches but authors differ)
• NOT_FOUND — no match in any database
• ERROR — API timeout or network failure

Ranking order: overall accuracy → year compliance rate → avg confidence → completeness

Output files

Copy code
evaluation_results/ref_hallucination_arena/
├── evaluation_report.md          # Detailed Markdown report
├── evaluation_results.json       # Rankings, per-field accuracy, scores
├── verification_chart.png        # Per-field accuracy bar chart
├── discipline_chart.png          # Per-discipline accuracy chart
├── queries.json                  # Loaded evaluation queries
├── responses.json                # Raw model responses
├── extracted_refs.json           # Extracted BibTeX references
├── verification_results.json     # Per-reference verification details
└── checkpoint.json               # Pipeline checkpoint for resume

API key by model

Additional resources

• Full config examples: cookbooks/ref_hallucination_arena/examples/
• Documentation: docs/validating_graders/ref_hallucination_arena.md
• Official dataset: HuggingFace
• Leaderboard: openjudge.me/leaderboard