analyze-leaderboard-dynamics
Purpose
Analyze leaderboard score distributions, compression or saturation, anomalies, selective reporting, and regime shifts.
Input contract
yamlrequired: [leaderboard_records, metric_schema, time_or_version_field] optional: [evaluation_protocols, source_quality, method_metadata] constraints: [scores require task, metric, condition, source, and observation provenance]
Procedure
- Normalize scores and evaluation conditions into comparable groups.
- Inspect distributions, rank movement, compression, and missingness.
- Identify anomalies, selective reporting, and regime changes.
- State whether additional evidence is continuing, near-saturated, or saturated.
Output contract
yamlproduces: [leaderboard_distribution, regime_segments, anomaly_report, saturation_assessment] delta_fields: [findings, evidence_updates, uncertainties, decisions]
Quality gates
- Every reported regime or anomaly must cite the records and comparison basis.
- Do not compare scores across incompatible protocols without an explicit limitation.
Failure and counterexamples
Do not treat rank order as progress when metric definitions or evaluation conditions changed; preserve ties and missing observations.
Provenance map
resolved: leaderboard-dynamics-analysis


