---
name: "caio-review"
description: "Use for /c-level-agents:caio-review, or to challenge a plan involving AI (no eval set, API vs fine-tune vs self-host, EU AI Act class)."
---
# /c-level-agents:caio-review — CAIO Forcing Questions
**Command:** `/c-level-agents:caio-review <plan>`
The eval-demanding CAIO pressure-tests any plan that involves AI. Six questions before any AI feature ships, any multi-year vendor commitment, or any AI team expansion.
## When to Run
- Before shipping any new AI-powered feature
- Before signing a multi-year AI vendor contract (API or self-hosted infra)
- Before EU launch of any AI feature
- Before a major AI team hire (especially ML engineer or research scientist)
- Before a fine-tuning project commitment
- Before adopting AI in a regulated domain (employment, credit, healthcare, education, etc.)
- When the founder uses the word "AI" near "competitive advantage" or "moat"
## The Six CAIO Questions
### 1. What does this AI need to be good at, and how would you measure it?
**No eval set = no ship.** Before any AI feature deploys, define the eval criteria.
- 50-100 representative inputs minimum
- Expected outputs OR rubric for grading
- Edge cases: ambiguous, adversarial, format-edge
- If you can't write down what "good" looks like, you don't have a feature; you have a vibe.
### 2. What's the SLO on hallucination / error rate, and what's the fallback?
**Every AI feature has a failure mode. Plan for it.**
- Quantified SLO: "<5% hallucination on factual queries"
- Detection mechanism: monitoring, sampling, customer feedback loop
- Fallback: human-in-loop review, lower-risk default response, refuse-to-answer
- Blast radius if SLO breached: how many users affected, what is the cost?
### 3. What's the risk tier under EU AI Act, and is conformity assessment required?
**Run `ai_risk_classifier.py` if any EU residents are affected OR domain is regulated.**
- PROHIBITED → cannot launch in EU; re-scope
- HIGH → conformity assessment + EU DB registration + 10 Articles of obligations (3-12 months, $50-200K)
- LIMITED → transparency obligations (chatbot disclosure, AI-generated content marking)
- MINIMAL → no specific obligations; NIST AI RMF voluntary
### 4. API, fine-tune, or build?
**Run `model_buildvsbuy_calculator.py` for the specific use case.**
- 80% of B2B SaaS use cases: API
- 15%: fine-tune (when domain-specific behavior + labeled data + ML team + high volume)
- <1%: build from scratch
- Decision must consider economic breakeven AND practical feasibility (data, team, compliance)
### 5. What's the 12-month cost trajectory at expected scale?
**Run `ai_cost_economics.py` for the workload.**
- API: variable, scales linearly
- Self-hosted: mostly fixed, breakeven typically 1-10B tokens/month for 70B-class
- Hidden costs of self-hosted: ops, monitoring, model updates, capacity, failover, security
- Hidden costs of API: vendor lock-in, capability drift, rate limits, data residency
- Prompt caching is the most underrated lever; check provider support
### 6. What role unblocks this — and have we hired prerequisites first?
**Map AI capability to specific role. Founders confuse AI engineer / ML engineer / research scientist.**
- AI engineer: applied + full-stack + prompts + evals + deployment (most startups need this)
- ML engineer: fine-tuning + retraining infra (only after platform engineer + labeled data)
- Research scientist: model invention (only if model IS the product)
- Don't hire research scientist as first AI hire — they need infrastructure to be productive
## Workflow
```bash
# 1. Model selection check
python ${CLAUDE_PLUGIN_ROOT}/vendor/c-level-advisor/chief-ai-officer-advisor/scripts/model_buildvsbuy_calculator.py use_case.json
# 2. Regulatory classification
python ${CLAUDE_PLUGIN_ROOT}/vendor/c-level-advisor/chief-ai-officer-advisor/scripts/ai_risk_classifier.py use_case.json
# 3. Cost projection
python ${CLAUDE_PLUGIN_ROOT}/vendor/c-level-advisor/chief-ai-officer-advisor/scripts/ai_cost_economics.py workload.json
```
## Output Format
```markdown
# CAIO Review: <plan>
**Date:** YYYY-MM-DD
## The Decision Being Made
[one sentence — which CAIO decision: model selection | risk classification | economics | next hire]
## Eval Discipline
- Eval set committed: yes/no
- SLO defined: <metric> < <threshold>
- Fallback behavior: <one line>
## Model Selection (if applicable)
- Recommended: API / FINE_TUNE / BUILD
- 3-year TCO: $X (chosen path) vs $Y (alternatives)
- Breakeven: <volume>
## Risk Classification (if applicable)
- EU AI Act tier: PROHIBITED / HIGH / LIMITED / MINIMAL
- Conformity assessment required: yes/no
- US state triggers: [list]
- Required controls open: N
## Cost Economics (if applicable)
- Monthly cost at current volume: $X
- Breakeven for self-hosted migration: <volume>
- Migration cost if applicable: $X (3-6 months)
## Org (if applicable)
- Next hire: <role>
- Why this, not the alternative: <one line>
- Prerequisite hires in place: yes/no
## Verdict
🟢 SHIP | 🟡 SHARPEN | 🔴 BLOCK
## Next Steps
[3 concrete actions]
```
## Routing
- `/c-level-agents:cdo-review` — for any training-data implications
- `/c-level-agents:gc-review` — for AI vendor contracts, output liability, training-data licensing
- `/c-level-agents:ciso-review` — for prompt injection / jailbreak / training-data poisoning threat model
- `/c-level-agents:cfo-review` — for multi-year vendor or GPU commitment TCO
- `cs-chro-advisor` agent — for AI team hires (comp, ladder, leveling)
- `/c-level-agents:decide` — log the verdict
- `/c-level-agents:freeze 60` — on multi-year AI commitments
## Related
- Agent: [`cs-caio-advisor`](../../agents/cs-caio-advisor.md)
- Skill: [`chief-ai-officer-advisor`](${CLAUDE_PLUGIN_ROOT}/vendor/c-level-advisor/chief-ai-officer-advisor/SKILL.md)
- Adjacent: `${CLAUDE_PLUGIN_ROOT}/vendor/c-level-advisor/chief-data-officer-advisor/` (training data rights, data strategy)
---
**Version:** 1.0.0