evaluation-rubrics

---
name: evaluation-rubrics
description: Designs structured scoring tools with explicit criteria, performance scales, and descriptors for consistent, transparent quality assessment. Use when need quality criteria and scoring scales to evaluate work consistently, compare alternatives objectively, set acceptance thresholds, reduce subjective bias, or when user mentions rubric, scoring criteria, quality standards, evaluation framework, inter-rater reliability, or grading/assessing work.
---
# Evaluation Rubrics

## Table of Contents
- [Workflow](#workflow)
- [Common Patterns](#common-patterns)
- [Guardrails](#guardrails)
- [Quick Reference](#quick-reference)

## Example

**Scenario**: Evaluating technical blog posts (1-5 scale)

| Criterion | 1 (Poor) | 3 (Adequate) | 5 (Excellent) |
|-----------|----------|--------------|---------------|
| **Technical Accuracy** | Multiple factual errors, misleading | Mostly correct, minor inaccuracies | Fully accurate, technically rigorous |
| **Clarity** | Confusing, jargon-heavy, poor structure | Clear to experts, some structure | Accessible to target audience, well-organized |
| **Practical Value** | No actionable guidance, theoretical only | Some examples, limited applicability | Concrete examples, immediately applicable |
| **Originality** | Rehashes common knowledge, no new insight | Some fresh perspective, builds on existing | Novel approach, advances understanding |

**Scoring**: Post A [4, 5, 3, 2] = 3.5 avg. Post B [5, 4, 5, 4] = 4.5 avg.
**Feedback for Post A**: "Strong clarity (5) and good accuracy (4), but needs more practical examples (3) and offers less original insight (2)."

## Workflow

Copy this checklist and track your progress:

```
Rubric Development Progress:
- [ ] Step 1: Define purpose and scope
- [ ] Step 2: Identify evaluation criteria
- [ ] Step 3: Design the scale
- [ ] Step 4: Write performance descriptors
- [ ] Step 5: Test and calibrate
- [ ] Step 6: Use and iterate
```

**Step 1: Define purpose and scope**

Clarify what you're evaluating, who evaluates, who uses results, what decisions depend on scores. See [resources/template.md](resources/template.md#purpose-definition-template) for scoping questions.

**Step 2: Identify evaluation criteria**

Brainstorm quality dimensions, prioritize most important/observable, balance coverage vs. simplicity (4-8 criteria typical). See [resources/template.md](resources/template.md#criteria-identification-template) for brainstorming framework.

**Step 3: Design the scale**

Choose number of levels (1-5, 1-4, 1-10), scale type (numeric, qualitative), anchors (what does each level mean?). See [resources/methodology.md](resources/methodology.md#scale-design-principles) for scale selection guidance.

**Step 4: Write performance descriptors**

For each criterion × level, write observable description of what that performance looks like. See [resources/template.md](resources/template.md#descriptor-writing-template) for writing guidelines.

**Step 5: Test and calibrate**

Have multiple reviewers score sample work, compare scores, discuss discrepancies, refine rubric. See [resources/methodology.md](resources/methodology.md#calibration-techniques) for inter-rater reliability testing.

**Step 6: Use and iterate**

Apply rubric, collect feedback from evaluators and evaluatees, revise criteria/descriptors as needed. Validate using [resources/evaluators/rubric_evaluation_rubrics.json](resources/evaluators/rubric_evaluation_rubrics.json). **Minimum standard**: Average score ≥ 3.5.

## Common Patterns

**Pattern 1: Analytic Rubric (Most Common)**
- **Structure**: Multiple criteria (rows), multiple levels (columns), descriptor for each cell
- **Use case**: Detailed feedback needed, want to see performance across dimensions, diagnostic assessment
- **Pros**: Specific feedback, identifies strengths/weaknesses by criterion, high reliability
- **Cons**: Time-consuming to create and use, can feel reductive
- **Example**: Code review rubric (Correctness, Efficiency, Readability, Maintainability × 1-5 scale)

**Pattern 2: Holistic Rubric**
- **Structure**: Single overall score, descriptors integrate multiple criteria
- **Use case**: Quick overall judgment, summative assessment, criteria hard to separate
- **Pros**: Fast, intuitive, captures gestalt quality
- **Cons**: Less actionable feedback, lower reliability, can't diagnose specific weaknesses
- **Example**: Essay holistic scoring (1=poor essay, 3=adequate essay, 5=excellent essay with detailed descriptors)

**Pattern 3: Single-Point Rubric**
- **Structure**: Criteria listed with only "meets standard" descriptor, space to note above/below
- **Use case**: Growth mindset feedback, encourage self-assessment, less punitive feel
- **Pros**: Emphasizes improvement not deficit, simpler to create, encourages dialogue
- **Cons**: Less precision, requires written feedback to supplement
- **Example**: Design critique (list criteria like "Visual hierarchy", "Accessibility", note "+Clear focal point, -Poor contrast")

**Pattern 4: Checklist (Binary)**
- **Structure**: List of yes/no items, must-haves for acceptance
- **Use case**: Compliance checks, minimum quality gates, pass/fail decisions
- **Pros**: Very clear, objective, easy to use
- **Cons**: No gradations, misses quality beyond basics, can feel rigid
- **Example**: Pull request checklist (Tests pass? Code linted? Documentation updated? Security review?)

**Pattern 5: Standards-Based Rubric**
- **Structure**: Criteria tied to learning objectives/competencies, levels = degree of mastery
- **Use case**: Educational assessment, skill certification, training evaluation, criterion-referenced
- **Pros**: Aligned to standards, shows progress toward mastery, diagnostic
- **Cons**: Requires clear standards, can be complex to design
- **Example**: Data science skills (Proficiency in: Data cleaning, Modeling, Visualization, Communication × Novice/Competent/Expert)

## Guardrails

1. **Criteria should be observable and measurable**: Not "good attitude" (subjective), but "arrives on time, volunteers for tasks, helps teammates" (observable). Test: Can two independent reviewers score this criterion consistently?

2. **Descriptors should distinguish levels clearly**: Each level needs concrete differences from adjacent levels. Avoid "5=very good, 4=good, 3=okay". Better: "5=zero bugs, meets all requirements, 4=1-2 minor bugs, meets 90% requirements."

3. **Use appropriate scale granularity**: 1-3 is too coarse, 1-10 is too fine. Sweet spot: 1-4 (forced choice, no middle) or 1-5 (allows neutral middle). Match granularity to actual observable differences.

4. **Balance comprehensiveness with simplicity**: Aim for 4-8 criteria covering essential quality dimensions. If >10 criteria, consider grouping or prioritizing.

5. **Calibrate for inter-rater reliability**: Have multiple reviewers score same work, measure agreement (Kappa, ICC). If <70% agreement, refine descriptors.

6. **Provide examples at each level**: Include concrete examples of work at each level (anchor papers, reference designs, code samples) to calibrate reviewers.

7. **Share rubric before evaluation**: If evaluatees see the rubric only after being scored, it is grading not guidance. Share upfront so people know expectations and can self-assess.

8. **Weight criteria appropriately**: If "Security" matters more than "Code style", weight it (Security x3, Style x1). Or use thresholds (score >=4 on Security to pass, regardless of other scores).

**Common pitfalls:**

- ❌ **Subjective language**: "Shows effort", "creative", "professional" - not observable without concrete descriptors
- ❌ **Overlapping criteria**: "Clarity" and "Organization" often conflated - define boundaries clearly
- ❌ **Hidden expectations**: Rubric doesn't mention X, but evaluators penalize for missing X - document all criteria
- ❌ **Central tendency bias**: Reviewers avoid extremes (always score 3/5) - use even-number scales (1-4) to force choice
- ❌ **Halo effect**: High score on one criterion biases other scores up - score each criterion independently before looking at others
- ❌ **Rubric drift**: Descriptors erode over time, reviewers interpret differently - periodic re-calibration required

## Quick Reference

**Key resources:**

- **[resources/template.md](resources/template.md)**: Purpose definition, criteria brainstorming, scale selection, descriptor templates, rubric formats
- **[resources/methodology.md](resources/methodology.md)**: Scale design principles, descriptor writing techniques, inter-rater reliability testing, bias mitigation
- **[resources/evaluators/rubric_evaluation_rubrics.json](resources/evaluators/rubric_evaluation_rubrics.json)**: Quality criteria for rubric design (criteria clarity, scale appropriateness, descriptor specificity)

**Scale Selection Guide**:

| Scale | Use When | Pros | Cons |
|-------|----------|------|------|
| **1-3** | Need quick categorization, clear tiers | Fast, forces clear decision | Too coarse, less feedback |
| **1-4** | Want forced choice (no middle) | Avoids central tendency, clear differentiation | No neutral option, feels binary |
| **1-5** | General purpose, most common | Allows neutral, familiar, good granularity | Central tendency bias (everyone gets 3) |
| **1-10** | Need fine gradations, large sample | Maximum differentiation, statistical analysis | False precision, hard to distinguish adjacent levels |
| **Qualitative** (Novice/Proficient/Expert) | Educational, skill development | Intuitive, growth-oriented | Less quantitative, harder to aggregate |
| **Binary** (Yes/No, Pass/Fail) | Compliance, gatekeeping | Objective, simple | No gradations, misses quality differences |

**Criteria Types**:

- **Product criteria**: Evaluate the artifact itself (correctness, clarity, completeness, aesthetics, performance)
- **Process criteria**: How work was done (methodology followed, collaboration, iteration, time management)
- **Impact criteria**: Outcomes/effects (user satisfaction, business value, learning achieved)
- **Meta criteria**: Quality of quality (documentation, testability, maintainability, scalability)

**Inter-Rater Reliability Benchmarks**:

- **<50% agreement**: Rubric unreliable, needs major revision
- **50-70% agreement**: Marginal, refine descriptors and calibrate reviewers
- **70-85% agreement**: Good, acceptable for most uses
- **>85% agreement**: Excellent, highly reliable scoring

**Typical Rubric Development Time**:

- **Simple rubric** (3-5 criteria, 1-4 scale, known domain): 2-4 hours
- **Standard rubric** (5-7 criteria, 1-5 scale, some complexity): 6-10 hours + calibration session
- **Complex rubric** (8+ criteria, multiple scales, novel domain): 15-25 hours + multiple calibration rounds

**When to escalate beyond rubrics**:

- High-stakes decisions (hiring, admissions, awards) → Add structured interviews, portfolios, multi-method assessment
- Subjective/creative work (art, poetry, design) → Supplement rubric with critique, discourse, expert judgment
- Complex holistic judgment (leadership, cultural fit) → Rubrics help but don't capture everything, use thoughtfully
→ Rubrics are tools not replacements for human judgment. Use to structure thinking, not mechanize decisions.

**Inputs required:**

- **Artifact type** (what are we evaluating? essays, code, designs, proposals?)
- **Criteria** (quality dimensions to assess, 4-8 most common)
- **Scale** (1-5 default, or specify 1-4, 1-10, qualitative labels)

**Outputs produced:**

- `evaluation-rubrics.md`: Purpose, criteria definitions, scale with descriptors, usage instructions, weighting/thresholds, calibration notes