alterlab-ssci-measurement-gate

$npx mdskill add AlterLab-IEU/AlterLab-Academic-Skills/alterlab-ssci-measurement-gate

Gates measurement quality before trusting a scale or instrument.

  • Ensures constructs are defined and operationalized before use.
  • Depends on alterlab-survey-design, alterlab-sem-psychometrics, and alterlab-statistical-analysis.
  • Checks reliability via McDonald omega, not Cronbach alpha alone.
  • Returns a pass/fail verdict with specific quality criteria.

SKILL.md

.github/skills/alterlab-ssci-measurement-gateView on GitHub ↗
---
name: alterlab-ssci-measurement-gate
description: "Gates measurement quality before a scale or instrument is trusted — checks that each construct is defined and operationalized, that reliability is evidenced with McDonald omega (not Cronbach alpha alone, which assumes tau-equivalence and is only a lower bound), that reliability is not confused with validity (content, criterion, convergent/discriminant construct validity), and that measurement invariance is tested before comparing groups. Use when asking whether a scale or survey instrument is valid, reporting a Cronbach alpha, building or adopting a multi-item measure, or comparing a latent construct across groups. For designing the questionnaire items prefer alterlab-survey-design; for running the confirmatory factor analysis prefer alterlab-sem-psychometrics; to execute basic statistics prefer alterlab-statistical-analysis. Part of the AlterLab Academic Skills suite."
license: MIT
allowed-tools: Read Bash(python:*)
compatibility: No API key required. A discipline-enforcing measurement-quality skill; reliability/CFA execution is handed to sibling skills that run locally via `uv run python`.
metadata:
    skill-author: AlterLab
    version: "1.0.0"
    depends_on: "alterlab-survey-design (item design), alterlab-sem-psychometrics (CFA/invariance), alterlab-statistical-analysis"
---

# Measurement Gate — Reliability Is Not Validity

**Skill type: DISCIPLINE-ENFORCING.** Before any hypothesis test consumes a scale, this gate
checks that the scale measures what it claims, reliably, and comparably across groups. It does
not fit factor models itself — it *requires* the right evidence and routes execution to the
psychometrics skill.

## The Core Rule

```
A RELIABLE MEASURE OF THE WRONG CONSTRUCT IS STILL WRONG. RELIABILITY ≠ VALIDITY.
```

High internal consistency only means the items move together; it says nothing about whether
they capture the intended construct. Evidence for *validity* (content, criterion, construct) is
separate and mandatory. And the usual reliability number — Cronbach alpha — rests on an
assumption (tau-equivalence: equal item loadings) that real scales rarely meet, so alpha is a
**lower bound** that can under- or (with correlated errors) over-state reliability.

## When to Use This Skill

- "Is my scale / survey instrument valid?"
- "My Cronbach alpha is 0.78 — is the scale good to use?" (← alpha alone is not enough)
- "I built a new 6-item measure of burnout; how do I show it works?"
- "I want to compare a latent trait (e.g. trust) across countries/groups."

### Does NOT Trigger

| The request is really about… | Route to | Why not this skill |
|---|---|---|
| Writing / wording the questionnaire items, response scales | `alterlab-survey-design` | Instrument construction, upstream of validation. |
| Running the CFA / SEM / IRT / invariance model | `alterlab-sem-psychometrics` | This gate requires CFA; that skill fits it. |
| Basic descriptive/inferential stats execution | `alterlab-statistical-analysis` | Computation, not measurement discipline. |
| Which statistical test to run on a measured outcome | `alterlab-test-selection-guard` | Test choice, not measurement quality. |
| Sampling frame / how many respondents | `alterlab-ssci-sampling-gate` | Sampling, not measurement. |

## Reliability: report alpha AND omega

Cronbach's alpha assumes **tau-equivalence** (all items load equally on one factor) and is a
**lower bound** on reliability under that model. **McDonald's omega** relaxes tau-equivalence by
weighting items by their factor loadings and should be reported alongside alpha (McNeish, 2018,
*Psychological Methods*, "Thanks coefficient alpha, we'll take it from here"). In practice the
alpha-vs-omega gap for well-constructed unidimensional scales is often small (Warne, 2025 —
around 4.5% underestimate), so the discipline is: **report both**, plus item-total correlations
and a dimensionality check — not to fetishize a single number, but to show the estimate is not
resting on an unexamined assumption.

Rules this gate enforces:

1. **Alpha alone is insufficient.** Ask for omega; if only alpha is available, state the
   tau-equivalence caveat explicitly.
2. **Reliability presupposes dimensionality.** A high alpha on a multidimensional set is
   meaningless — check the factor structure (CFA, `alterlab-sem-psychometrics`) first.
3. **Item diagnostics.** Item-total correlations and (for two halves) Spearman-Brown; drop or
   revise items that do not cohere — but pre-specify such decisions, do not fish.

## Validity: four kinds, none optional

| Validity | Question | Evidence |
|----------|----------|----------|
| **Content** | Do items cover the construct's domain? | expert review, blueprint against the definition |
| **Criterion** | Does it predict/correlate with a gold standard? | concurrent & predictive correlations |
| **Convergent** | Does it correlate with related measures? | AVE, correlations with kindred scales |
| **Discriminant** | Is it distinct from other constructs? | heterotrait-monotrait ratio, cross-loadings |

Construct validity is established with a **confirmatory factor analysis** — route the fit to
`alterlab-sem-psychometrics` (CFI/TLI, RMSEA, SRMR against Hu & Bentler cutoffs).

## Measurement invariance before group comparison

Comparing a latent construct across groups (countries, waves, conditions) is only valid if the
measure means the same thing in each group. Require a **multi-group CFA invariance** sequence —
configural → metric → scalar — before interpreting group differences in means. Without at least
metric/scalar invariance, a "difference" may be a measurement artifact, not a real effect.
Route the test to `alterlab-sem-psychometrics`.

## The Design Passport (hand-off)

Append to each construct in the Passport: `reliability` (alpha + omega + caveat),
`validity_evidence` (which of the four kinds are shown), `dimensionality` (CFA fit), and
`invariance` (level established) before any group comparison downstream.

## Self-Check Before Advancing

- Is each construct explicitly defined and operationalized to its items?
- Is reliability shown with omega (or alpha with the tau-equivalence caveat stated)?
- Is at least one form of validity evidence present — not just reliability?
- Was dimensionality confirmed (CFA) before trusting a single reliability coefficient?
- Is measurement invariance established before comparing the construct across groups?

## References

- `references/reliability_and_validity.md` — alpha vs omega math, the four validities, the
  invariance sequence, and reporting templates. Loaded on demand.

Part of the AlterLab Academic Skills suite.

More from AlterLab-IEU/AlterLab-Academic-Skills

SkillDescription
alterlab-abm-mesaBuilds agent-based models of social systems with Mesa 3 — the current AgentSet API (model.agents.shuffle_do('step'), auto-assigned unique_id, mandatory super().__init__(seed=...)), cell spaces (mesa.discrete_space OrthogonalMooreGrid / classic mesa.space grids), the DataCollector, batch_run parameter sweeps, and SolaraViz — for emergence, segregation, diffusion, opinion dynamics, and cooperation models. It uses the Mesa 3.x API (the old mesa.time schedulers like RandomActivation are removed) and treats the model as a generative theory to be validated, not just run. Use when the request mentions an agent-based model, Mesa, simulating interacting agents, or emergent macro behavior from micro rules. For discrete-event (queueing/process) simulation prefer alterlab-simpy; for reinforcement learning prefer alterlab-stable-baselines3. Part of the AlterLab Academic Skills suite.
alterlab-academic-careerPrepares academic career documents and professional-development materials for faculty and researchers. Use when drafting academic CVs, research statements, teaching philosophies, diversity statements, faculty-position cover letters, tenure dossiers, promotion narratives, academic portfolios, or mentorship statements, and when planning conference-networking strategy, building academic web presence, setting up ORCID profiles, or understanding impact metrics. Part of the AlterLab Academic Skills suite.
alterlab-accreditation-aolScaffolds program-level Assurance-of-Learning (AoL) documentation for AACSB (2020 Standard 5) and ABET (Criterion 3 Student Outcomes, Criterion 4 Continuous Improvement) accreditation — program learning outcomes / competency goals, curriculum-to-outcome mapping matrices, direct- and indirect-assessment plans, rubric design, and closing-the-loop continuous-improvement narratives — and validates the structure of an outcome-mapping matrix with scripts/aol_matrix.py. Use when the user needs AACSB or ABET assurance-of-learning material, a program-learning-outcomes set, a curriculum/outcome map or coverage matrix, a direct/indirect assessment plan, a closing-the-loop report, or accreditation self-study text. For single-course design or course rubrics prefer alterlab-teaching-design; for post-award grant reports prefer alterlab-grant-reporting. Part of the AlterLab Academic Skills suite.
alterlab-adaptyvSubmits and tracks protein-testing experiments on the Adaptyv Bio Foundry cloud lab (wet-lab validation), and optimizes protein sequences before submission with computational tools (NetSolP, SoluProt, SolubleMPNN, ESM). Use when designing proteins that need wet-lab validation - binding/affinity screening, expression testing, thermostability, or fluorescence assays - or when submitting experiments to the Foundry API, browsing the target catalog, tracking experiment status, retrieving results, or pre-screening sequences for solubility/expression. Triggers on "Adaptyv", "Foundry API", "cloud lab", "biolayer interferometry / BLI", "wet-lab validation". Part of the AlterLab Academic Skills suite.
alterlab-aeonRuns time series machine learning with the aeon library — classification, regression, clustering, forecasting, anomaly detection, segmentation, and similarity search via scikit-learn compatible APIs. Use when working with temporal data, sequential patterns, or time-indexed observations (univariate or multivariate) that need specialized algorithms beyond standard ML approaches. Part of the AlterLab Academic Skills suite.
alterlab-akademik-tesvikComputes the annual Turkish academic-incentive (akademik teşvik) score for Devlet (state) university öğretim elemanı (academic staff) from a classified activity list, applying the verified Akademik Teşvik Ödeneği Yönetmeliği (2018/11834) Faaliyet Hesaplama Tablosu headline puanları (Proje 20, Araştırma 15, Yayın 30, Patent 30, Atıf 30, Tebliğ 20, Ödül 20) with the k (author-count), p (Q1 1/Q2 0.8/Q3 0.5/Q4 0.25), and r (project-role) coefficients via türü puanı = Σ(faaliyet oranları) × headline (MADDE 8/2), then enforcing the per-type headline ceilings, the 100 cap (MADDE 8/3), and the net-30 payment gate (MADDE 10/3) in a deterministic scripts/tesvik_score.py. Use when the user wants to calculate akademik teşvik puanı, check the en az 30 (net-30) threshold, apply TABLO 4 ceilings, or estimate teşvik eligibility; keeps its coefficients separate from doçentlik scoring. For doçentlik point eligibility prefer alterlab-docentlik-eligibility. Part of the AlterLab Academic Skills suite.
alterlab-alpha-vantageAccesses real-time and historical stock market data, forex rates, cryptocurrency prices, commodities, economic indicators, and 50+ technical indicators via the Alpha Vantage API (requires a free API key from alphavantage.co). Use when fetching stock prices (OHLCV), company fundamentals (income statement, balance sheet, cash flow), earnings, options data, market news/sentiment, insider transactions, GDP, CPI, treasury yields, gold/silver/oil prices, Bitcoin/crypto prices, forex exchange rates, or calculating technical indicators (SMA, EMA, MACD, RSI, Bollinger Bands). Part of the AlterLab Academic Skills suite.
alterlab-alphafoldPredict protein 3D structures with AlphaFold2 via ColabFold — MMseqs2-accelerated MSAs, monomer and AlphaFold2-Multimer complex folding, and confidence-based validation (pLDDT, pTM/ipTM, PAE). Use when folding a protein sequence or complex from FASTA, generating a predicted structure with confidence metrics, ranking models, or checking self-consistency of a design. For co-folding a protein WITH a small-molecule ligand or predicting binding affinity prefer alterlab-boltz; for antibody–antigen or one-FASTA multi-entity complexes prefer alterlab-chai; to LOOK UP an already-computed structure prefer alterlab-alphafold-db; for ESM embeddings or inverse folding prefer alterlab-esm. Part of the AlterLab Academic Skills suite.
alterlab-alphafold-dbAccess the AlphaFold DB of 200M+ AI-PREDICTED protein structures — retrieve models by UniProt accession, download PDB/mmCIF files, and analyze prediction confidence metrics (pLDDT, PAE). Use when a UniProt ID needs a computationally predicted 3D structure or when no experimental structure exists, for homology modeling, protein engineering, or structure-based drug discovery; for EXPERIMENTALLY determined structures (X-ray, cryo-EM, NMR) prefer alterlab-pdb, and for protein sequences, annotations, or accession ID mapping prefer alterlab-uniprot instead. Part of the AlterLab Academic Skills suite.
alterlab-anndataBuild, slice, concatenate, read, and write AnnData annotated data matrices (obs, var, X, layers, obsm, uns) — the scverse data STRUCTURE, not an analysis pipeline. Use when creating or wrangling .h5ad/zarr files, managing cell and gene annotations, concatenating batches, or handling layers/obsm/backed-mode; for the QC, normalization, clustering, UMAP, and differential-expression analysis pipeline prefer alterlab-scanpy instead, and for RNA velocity from spliced/unspliced layers prefer alterlab-scvelo instead. Part of the AlterLab Academic Skills suite.