reproducibility-checklist-audit

Name: reproducibility-checklist-audit
Author: yogsoth-ai/de-anthropocentric-research-engine

$npx mdskill add yogsoth-ai/de-anthropocentric-research-engine/reproducibility-checklist-audit

Evaluates a paper's completeness against ML Reproducibility Checklist

Identifies missing information in the paper content related to model architecture, training procedure, hyperparameters, and data preprocessing.
Deploys an automated tool that parses through the provided markdown text for the paper.
Analyzes the parsed data to determine if all required elements are present or need additional details based on the checklist.
Generates a structured report with a reproducibility score indicating completeness of information.

SKILL.md

.github/skills/reproducibility-checklist-auditView on GitHub ↗

---
name: reproducibility-checklist-audit
description: Assess paper completeness against ML Reproducibility Checklist
execution: subagent
prompt: ./prompt.md
input: paper_content
used-by: baseline-establishment
---

# Reproducibility Checklist Audit


## Purpose

Evaluate a paper against the standard ML Reproducibility Checklist (as used by NeurIPS, ICML, ICLR). Produces a structured assessment of what information is present, what is missing, and an overall reproducibility score.

## Input Schema

| Field | Type | Description |
|-------|------|-------------|
| paper_content | string | Full paper text (markdown format) |

## Output Schema

```json
{
  "paper_title": "string",
  "checklist": {
    "model_architecture": {"present": true, "details": "string"},
    "training_procedure": {"present": true, "details": "string"},
    "hyperparameters": {"present": true, "details": "string"},
    "hyperparameter_search": {"present": false, "details": "string"},
    "datasets": {"present": true, "details": "string"},
    "data_preprocessing": {"present": true, "details": "string"},
    "evaluation_metrics": {"present": true, "details": "string"},
    "error_bars_or_confidence": {"present": false, "details": "string"},
    "number_of_runs": {"present": false, "details": "string"},
    "compute_resources": {"present": false, "details": "string"},
    "code_availability": {"present": true, "details": "string"},
    "random_seeds": {"present": false, "details": "string"}
  },
  "overall_score": 0.0,
  "critical_gaps": ["string"],
  "reproducibility_risk": "low|medium|high|critical"
}
```

More from yogsoth-ai/de-anthropocentric-research-engine

Skill	Description
abductive-hypothesis-generation	Strategy: 面对异常的最佳解释推理
ablation-brainstorm	Remove components one by one, observe system changes to reveal hidden dependencies and generate ideas from structural gaps.
ablation-component-mapping	Map system architecture to ablatable units for ablation studies
ablation-design	Design ablation studies to isolate component contributions in ML systems
ablation-execution	Remove components one by one from a system, record the response/impact of each removal.
abp-vulnerability-classification	Classify assumptions on 2 axes — load-bearing (how much conclusion depends on it) × vulnerable (how likely to be false). Focuses attention on High-Load × High-Vulnerable quadrant.
abstraction-extraction	Extract abstract principles from concrete domain cases. Strips domain-specific details to reveal transferable mechanisms.
abstraction-ladder	Perform bisociation at multiple abstraction levels
abstraction-laddering	Move between concrete and abstract framings — 3 levels up (Why?) and 3 levels down (How?) to find the most productive research level.
abstraction-to-design	Abstract biological principle to design principle. Bridge from biology to engineering.