leaderboard-dynamics-analysis

$npx mdskill add yogsoth-ai/de-anthropocentric-research-engine/leaderboard-dynamics-analysis

Analyzes leaderboard score distributions and dynamics for insights

  • Identifies patterns of model performance over time on benchmark leaderboards.
  • Detects changes in the gap between top-performing models (score compression).
  • Identifies selective reporting practices by analyzing data selection criteria.
  • Provides a health assessment with statistical evidence and anomalies detection.

SKILL.md

.github/skills/leaderboard-dynamics-analysisView on GitHub ↗
---
name: leaderboard-dynamics-analysis
description: Analyze leaderboard score distributions, compression, selective reporting
execution: subagent
prompt: ./prompt.md
input: leaderboard_data (model, score, date)
used-by: benchmark-archaeology
---

# Leaderboard Dynamics Analysis SOP

Analyze the dynamics of benchmark leaderboards: score distributions, compression over time, selective reporting patterns, and statistical artifacts.

## Input

- **leaderboard_data**: Structured data with (model_name, score, date, optional: model_size, paper_reference)

## Procedure

1. Compute score distribution statistics over time
2. Detect score compression (shrinking gap between top models)
3. Identify selective reporting patterns (cherry-picking, subset selection)
4. Analyze relationship between model size and score
5. Detect anomalies (withdrawn results, unreplicated claims)

## Output

Leaderboard health assessment with statistical evidence.

More from yogsoth-ai/de-anthropocentric-research-engine

SkillDescription
abductive-hypothesis-generationStrategy: 面对异常的最佳解释推理
ablation-brainstormRemove components one by one, observe system changes to reveal hidden dependencies and generate ideas from structural gaps.
ablation-component-mappingMap system architecture to ablatable units for ablation studies
ablation-designDesign ablation studies to isolate component contributions in ML systems
ablation-executionRemove components one by one from a system, record the response/impact of each removal.
abp-vulnerability-classificationClassify assumptions on 2 axes — load-bearing (how much conclusion depends on it) × vulnerable (how likely to be false). Focuses attention on High-Load × High-Vulnerable quadrant.
abstraction-extractionExtract abstract principles from concrete domain cases. Strips domain-specific details to reveal transferable mechanisms.
abstraction-ladderPerform bisociation at multiple abstraction levels
abstraction-ladderingMove between concrete and abstract framings — 3 levels up (Why?) and 3 levels down (How?) to find the most productive research level.
abstraction-to-designAbstract biological principle to design principle. Bridge from biology to engineering.