alterlab-geo

$npx mdskill add AlterLab-IEU/AlterLab-Academic-Skills/alterlab-geo

Search and download gene expression datasets from NCBI GEO.

  • Locate public microarray and RNA-seq datasets by accession number.
  • Uses NCBI E-utilities REST API with email requirement.
  • Determines dataset type and retrieves SOFT, MINiML, or matrix files.
  • Returns processed expression matrices and supplementary files.

SKILL.md

.github/skills/alterlab-geoView on GitHub ↗
---
name: alterlab-geo
description: Access NCBI GEO (Gene Expression Omnibus) for gene expression and functional genomics data — search and download microarray and RNA-seq datasets by GSE, GSM, GPL, or GDS accession and retrieve SOFT, MINiML, and series matrix files. Use when locating public expression datasets, fetching processed expression matrices, downloading a study's supplementary files, or sourcing per-study transcriptomics data for differential-expression analysis. For raw FASTQ sequencing reads by SRA/ENA run accession use alterlab-ena; for reference tissue-expression baselines (median TPM across human tissues) use alterlab-gtex; for cancer cohort somatic mutations and copy-number use alterlab-cbioportal. Part of the AlterLab Academic Skills suite.
license: MIT
allowed-tools: Read WebFetch Bash(curl:*) Bash(uv:*)
compatibility: Keyless NCBI E-utilities REST API (email required by NCBI); optional NCBI API key raises rate limits
metadata:
    skill-author: AlterLab
    version: "1.0.0"
---

# GEO Database

## Overview

The Gene Expression Omnibus (GEO) is NCBI's public repository for high-throughput gene expression and functional genomics data. It holds hundreds of thousands of studies (Series) and millions of samples from both array-based and sequence-based experiments. For current totals see the GEO browser at https://www.ncbi.nlm.nih.gov/geo/.

## When to Use This Skill

Use this skill when searching for gene expression datasets, retrieving experimental data, downloading raw and processed files, querying expression profiles, or integrating GEO data into computational analysis workflows.

## Core Workflow

1. **Search** for relevant studies (E-utilities / `Bio.Entrez`) → see `references/searching.md`.
2. **Retrieve** a series with GEOparse: `GEOparse.get_GEO(geo="GSE...", destdir="./data")` → see `references/geoparse_usage.md`.
3. **Extract** the expression matrix via `gse.pivot_samples('VALUE')` (genes × samples).
4. **Analyze** — QC, differential expression, clustering, meta-analysis → see `references/expression_analysis.md`.

For bulk downloads, skip GEOparse and pull files directly over FTP → see `references/data_access.md`.

## GEO Data Organization

GEO organizes data hierarchically using different accession types:

- **Series (GSE)** — a complete experiment with related samples (e.g. `GSE123456`).
  Largest organizational unit. Holds experimental design, samples, study info.
- **Sample (GSM)** — a single experimental sample / biological replicate
  (e.g. `GSM987654`). Linked to platforms and series.
- **Platform (GPL)** — the microarray or sequencing platform (e.g. `GPL570`,
  Affymetrix Human Genome U133 Plus 2.0 Array). Shared across experiments.
- **DataSet (GDS)** — curated, consistently-formatted collections (e.g. `GDS5678`)
  processed for differential analysis. Ideal for quick comparative analyses, but
  the curated GDS set is legacy and no longer growing — most recent studies exist
  only as GSE Series, so prefer GSE for new work.
- **Profiles** — gene-specific expression data linked to sequence features,
  queryable by gene name, cross-referenced to Entrez Gene.

## Access Methods (routing)

- **GEOparse (recommended)** — easiest series-level access; download, parse
  metadata, extract expression matrices, supplementary files, filter samples.
  → `references/geoparse_usage.md`
- **E-utilities (`Bio.Entrez`)** — metadata searching and batch queries across
  `gds`, `geoprofiles`. Always set `Entrez.email`. → `references/searching.md`
- **Direct FTP / wget / curl** — bulk downloads of series matrix, SOFT, MINiML,
  and supplementary files; no rate limits. → `references/data_access.md`
- **GEO2R web tool** — no-code differential expression analysis in the browser at
  `https://www.ncbi.nlm.nih.gov/geo/geo2r/?acc=GSExxxxx`; generates reproducible
  R scripts. Useful for exploratory analysis before downloading.

## Installation and Setup

```bash
uv pip install GEOparse        # primary GEO access (recommended)
uv pip install biopython       # E-utilities / programmatic NCBI access
uv pip install pandas numpy scipy statsmodels scikit-learn  # analysis
uv pip install matplotlib seaborn                           # visualization
```

Configure NCBI E-utilities access (email is required by NCBI; an API key raises
the rate limit from 3 to 10 requests/second):

```python
from Bio import Entrez

Entrez.email = "[email protected]"   # required
# Optional API key — https://www.ncbi.nlm.nih.gov/account/
Entrez.api_key = "your_api_key_here"
```

## Key Concepts

- **SOFT (Simple Omnibus Format in Text):** GEO's primary text format with
  metadata and data tables. Easily parsed by GEOparse.
- **MINiML (MIAME Notation in Markup Language):** XML format for programmatic
  access and data exchange.
- **Series Matrix:** tab-delimited expression matrix (samples as columns,
  genes/probes as rows). Fastest format for getting expression data.
- **MIAME Compliance:** minimum standardized annotation GEO enforces on
  submissions.
- **Expression Value Types:** raw signal, normalized, or log-transformed — always
  check platform and processing methods.
- **Platform Annotation:** maps probe/feature IDs to genes; essential for
  biological interpretation.

## Common Use Cases

- **Transcriptomics research** — download expression data, compare profiles
  across studies, identify DEGs, run meta-analyses.
- **Drug response studies** — analyze post-treatment expression changes, identify
  response biomarkers, build sensitivity models.
- **Disease biology** — disease vs. normal expression, disease signatures,
  patient subgroup/stage comparisons, expression–outcome correlation.
- **Biomarker discovery** — screen diagnostic/prognostic markers, validate across
  cohorts, integrate with clinical data.

## Rate Limiting and Best Practices

- **E-utilities limits:** 3 req/s without an API key, 10 req/s with one. Insert
  delays: `time.sleep(0.34)` (no key) or `time.sleep(0.1)` (with key).
- **FTP:** no rate limits — preferred for bulk downloads (`wget -r` for whole
  directories).
- **GEOparse caching:** files cached in `destdir`; subsequent calls reuse them.
- **Method selection:** GEOparse for series-level access, E-utilities for search /
  batch metadata, FTP for direct/bulk file pulls. Cache locally; always set
  `Entrez.email` with Biopython.

## Important Notes

- **Data quality:** GEO accepts user-submitted data of varying quality. Check
  platform annotation and processing methods, verify metadata and design, watch
  for batch effects, and consider reprocessing raw data.
- **File sizes:** series matrix files can exceed 1 GB; supplementary files (e.g.
  CEL) can be very large. Plan disk space; download incrementally.
- **Citation:** GEO data is free for research. Cite original studies plus the GEO
  database (Barrett et al. 2013, *Nucleic Acids Research*). Check per-dataset
  usage restrictions and follow NCBI guidelines.
- **Common pitfalls:** platforms use different probe IDs (need annotation
  mapping); values may be raw/normalized/log-transformed (check metadata); sample
  metadata is inconsistently formatted; older submissions may lack series matrix
  files; platform annotations may be outdated.

## Reference Index

- `references/searching.md` — E-utilities search (datasets, profiles, advanced
  queries, search→summary→fetch workflow, batch metadata).
- `references/geoparse_usage.md` — GEOparse install/usage, expression matrices,
  supplementary files, sample filtering, caching.
- `references/data_access.md` — direct FTP access (ftplib, wget, curl) and the
  accession-to-path rule.
- `references/expression_analysis.md` — QC/preprocessing, differential expression,
  correlation/clustering, batch processing, cross-study meta-analysis.
- `references/geo_reference.md` — in-depth technical reference: E-utilities API
  endpoints, SOFT/MINiML format docs, FTP directory structure, normalization
  pipelines, platform quirks, and error-handling troubleshooting.

## Additional Resources

- **GEO Website:** https://www.ncbi.nlm.nih.gov/geo/
- **GEO Submission Guidelines:** https://www.ncbi.nlm.nih.gov/geo/info/submission.html
- **GEOparse Documentation:** https://geoparse.readthedocs.io/
- **E-utilities Documentation:** https://www.ncbi.nlm.nih.gov/books/NBK25501/
- **GEO FTP Site:** ftp://ftp.ncbi.nlm.nih.gov/geo/
- **GEO2R Tool:** https://www.ncbi.nlm.nih.gov/geo/geo2r/
- **NCBI API Keys:** https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/
- **Biopython Tutorial:** https://biopython.org/DIST/docs/tutorial/Tutorial.html

## Scripts

`scripts/query_geo.py` — runnable helper for GEO DataSets via NCBI E-utilities (no key; `NCBI_API_KEY` lifts the rate limit). Has a PEP 723 inline-dependency header, so `uv run` installs `requests` automatically:

```bash
uv run scripts/query_geo.py search "breast cancer AND Homo sapiens[ORGN]" --retmax 5
uv run scripts/query_geo.py summary 200000001,200000002
```

More from AlterLab-IEU/AlterLab-Academic-Skills

SkillDescription
alterlab-abm-mesaBuilds agent-based models of social systems with Mesa 3 — the current AgentSet API (model.agents.shuffle_do('step'), auto-assigned unique_id, mandatory super().__init__(seed=...)), cell spaces (mesa.discrete_space OrthogonalMooreGrid / classic mesa.space grids), the DataCollector, batch_run parameter sweeps, and SolaraViz — for emergence, segregation, diffusion, opinion dynamics, and cooperation models. It uses the Mesa 3.x API (the old mesa.time schedulers like RandomActivation are removed) and treats the model as a generative theory to be validated, not just run. Use when the request mentions an agent-based model, Mesa, simulating interacting agents, or emergent macro behavior from micro rules. For discrete-event (queueing/process) simulation prefer alterlab-simpy; for reinforcement learning prefer alterlab-stable-baselines3. Part of the AlterLab Academic Skills suite.
alterlab-academic-careerPrepares academic career documents and professional-development materials for faculty and researchers. Use when drafting academic CVs, research statements, teaching philosophies, diversity statements, faculty-position cover letters, tenure dossiers, promotion narratives, academic portfolios, or mentorship statements, and when planning conference-networking strategy, building academic web presence, setting up ORCID profiles, or understanding impact metrics. Part of the AlterLab Academic Skills suite.
alterlab-accreditation-aolScaffolds program-level Assurance-of-Learning (AoL) documentation for AACSB (2020 Standard 5) and ABET (Criterion 3 Student Outcomes, Criterion 4 Continuous Improvement) accreditation — program learning outcomes / competency goals, curriculum-to-outcome mapping matrices, direct- and indirect-assessment plans, rubric design, and closing-the-loop continuous-improvement narratives — and validates the structure of an outcome-mapping matrix with scripts/aol_matrix.py. Use when the user needs AACSB or ABET assurance-of-learning material, a program-learning-outcomes set, a curriculum/outcome map or coverage matrix, a direct/indirect assessment plan, a closing-the-loop report, or accreditation self-study text. For single-course design or course rubrics prefer alterlab-teaching-design; for post-award grant reports prefer alterlab-grant-reporting. Part of the AlterLab Academic Skills suite.
alterlab-adaptyvSubmits and tracks protein-testing experiments on the Adaptyv Bio Foundry cloud lab (wet-lab validation), and optimizes protein sequences before submission with computational tools (NetSolP, SoluProt, SolubleMPNN, ESM). Use when designing proteins that need wet-lab validation - binding/affinity screening, expression testing, thermostability, or fluorescence assays - or when submitting experiments to the Foundry API, browsing the target catalog, tracking experiment status, retrieving results, or pre-screening sequences for solubility/expression. Triggers on "Adaptyv", "Foundry API", "cloud lab", "biolayer interferometry / BLI", "wet-lab validation". Part of the AlterLab Academic Skills suite.
alterlab-aeonRuns time series machine learning with the aeon library — classification, regression, clustering, forecasting, anomaly detection, segmentation, and similarity search via scikit-learn compatible APIs. Use when working with temporal data, sequential patterns, or time-indexed observations (univariate or multivariate) that need specialized algorithms beyond standard ML approaches. Part of the AlterLab Academic Skills suite.
alterlab-akademik-tesvikComputes the annual Turkish academic-incentive (akademik teşvik) score for Devlet (state) university öğretim elemanı (academic staff) from a classified activity list, applying the verified Akademik Teşvik Ödeneği Yönetmeliği (2018/11834) Faaliyet Hesaplama Tablosu headline puanları (Proje 20, Araştırma 15, Yayın 30, Patent 30, Atıf 30, Tebliğ 20, Ödül 20) with the k (author-count), p (Q1 1/Q2 0.8/Q3 0.5/Q4 0.25), and r (project-role) coefficients via türü puanı = Σ(faaliyet oranları) × headline (MADDE 8/2), then enforcing the per-type headline ceilings, the 100 cap (MADDE 8/3), and the net-30 payment gate (MADDE 10/3) in a deterministic scripts/tesvik_score.py. Use when the user wants to calculate akademik teşvik puanı, check the en az 30 (net-30) threshold, apply TABLO 4 ceilings, or estimate teşvik eligibility; keeps its coefficients separate from doçentlik scoring. For doçentlik point eligibility prefer alterlab-docentlik-eligibility. Part of the AlterLab Academic Skills suite.
alterlab-alpha-vantageAccesses real-time and historical stock market data, forex rates, cryptocurrency prices, commodities, economic indicators, and 50+ technical indicators via the Alpha Vantage API (requires a free API key from alphavantage.co). Use when fetching stock prices (OHLCV), company fundamentals (income statement, balance sheet, cash flow), earnings, options data, market news/sentiment, insider transactions, GDP, CPI, treasury yields, gold/silver/oil prices, Bitcoin/crypto prices, forex exchange rates, or calculating technical indicators (SMA, EMA, MACD, RSI, Bollinger Bands). Part of the AlterLab Academic Skills suite.
alterlab-alphafoldPredict protein 3D structures with AlphaFold2 via ColabFold — MMseqs2-accelerated MSAs, monomer and AlphaFold2-Multimer complex folding, and confidence-based validation (pLDDT, pTM/ipTM, PAE). Use when folding a protein sequence or complex from FASTA, generating a predicted structure with confidence metrics, ranking models, or checking self-consistency of a design. For co-folding a protein WITH a small-molecule ligand or predicting binding affinity prefer alterlab-boltz; for antibody–antigen or one-FASTA multi-entity complexes prefer alterlab-chai; to LOOK UP an already-computed structure prefer alterlab-alphafold-db; for ESM embeddings or inverse folding prefer alterlab-esm. Part of the AlterLab Academic Skills suite.
alterlab-alphafold-dbAccess the AlphaFold DB of 200M+ AI-PREDICTED protein structures — retrieve models by UniProt accession, download PDB/mmCIF files, and analyze prediction confidence metrics (pLDDT, PAE). Use when a UniProt ID needs a computationally predicted 3D structure or when no experimental structure exists, for homology modeling, protein engineering, or structure-based drug discovery; for EXPERIMENTALLY determined structures (X-ray, cryo-EM, NMR) prefer alterlab-pdb, and for protein sequences, annotations, or accession ID mapping prefer alterlab-uniprot instead. Part of the AlterLab Academic Skills suite.
alterlab-anndataBuild, slice, concatenate, read, and write AnnData annotated data matrices (obs, var, X, layers, obsm, uns) — the scverse data STRUCTURE, not an analysis pipeline. Use when creating or wrangling .h5ad/zarr files, managing cell and gene annotations, concatenating batches, or handling layers/obsm/backed-mode; for the QC, normalization, clustering, UMAP, and differential-expression analysis pipeline prefer alterlab-scanpy instead, and for RNA velocity from spliced/unspliced layers prefer alterlab-scvelo instead. Part of the AlterLab Academic Skills suite.