---
name: context-intelligence-eval-design
version: 1.0.0
description: Use when designing evaluation scenarios for a context-intelligence tool signal. Derives success criteria from domain-concepts.md and produces evaluation-scenarios.md entries plus DTU profile templates.
user-invocable: false
allowed-tools: read_file, glob, grep, delegate, load_skill, todo
model_role: reasoning
license: MIT
---
# Context Intelligence Evaluation Design
Phase 3 specialist skill for context-intelligence mode. Called by context-intelligence-tool-designer agent via self-delegation with context_depth="none", scoped to one concept at a time.
---
## Scope
**In scope:**
- Derive success criteria from `domain-concepts.md`
- Design one evaluation scenario per concept
- Populate DTU profile templates including mandatory Gitea URL rewrite block
**Out of scope:**
- Investigating signals
- Classifying detection strategies
- Running evaluations
Note: The evaluation runner is `amplifier-bundle-evaluation` invoked inside a DTU built from this scenario.
---
## Criteria Derivation Rule
Success criteria MUST come from the user-confirmed concept definition in `domain-concepts.md` — specifically the `Agreed definition`, `Boundary conditions`, `Data availability`, and `Explicitly excludes` fields. NEVER derive from implementation details.
What a detector returns is not the criterion; what correct detection looks like in plain terms from the user's domain perspective is the criterion.
If the definition is too vague to produce a testable criterion, return a structured gap entry rather than guessing.
---
## Per-Concept Evaluation Scenario Format
Each scenario is ready for insertion into `evaluation-scenarios.md`:
```markdown
## Scenario [N]: [Concept] — [description]
**Derived from:** [field reference in domain-concepts.md — Agreed definition / Boundary conditions / Data availability / Explicitly excludes]
**Success criterion:** [Plain-language statement of what correct detection looks like from the user's domain perspective. Not what the detector returns — what it means.]
**Failing scenario:** [Concrete example of a case where detection fails or produces a false positive, derived from Explicitly excludes or Boundary conditions.]
**DTU environment:** [Reference to DTU profile template produced by this skill for this concept.]
**Pass threshold:** [Quantitative or qualitative threshold — e.g., "detects in ≥8 of 10 representative sessions" or "no false positives on the Explicitly excludes examples".]
**Iteration question:** [The question this scenario answers. Used to route delta back through Phase 4. E.g., "Does the signal correctly distinguish X from Y as the user defined it?"]
```
---
## DTU Profile Template — Mandatory Gitea URL Rewrite
Every DTU profile generated by this skill MUST include the URL rewrite block below. This block is non-optional.
The explanatory comment is included inline so readers cannot remove the block without understanding why it exists.
```yaml
# MANDATORY: Gitea URL rewrite for local branch skill resolution.
# The bundle ships skills with a git+https://github.com/microsoft/amplifier-bundle-context-intelligence@main#subdirectory=skills source in behaviors/context-intelligence.yaml.
# When running in DTU, local working branch is source of truth — NOT GitHub @main. Without rewrite, skill registration inside DTU fetches from GitHub @main and silently runs stale code, producing test results that do not reflect local changes.
# The digital-twin-universe:dtu-profile-builder agent mirrors the local amplifier-bundle-context-intelligence submodule to Gitea on current branch. These rewrites point every reference at that mirror.
url_rewrites:
- from: "git+https://github.com/microsoft/amplifier-bundle-context-intelligence@main"
to: "git+http://{gitea_host}/microsoft/amplifier-bundle-context-intelligence@main"
```
This block must appear in EVERY profile this skill produces. `{gitea_host}` is resolved by `digital-twin-universe:dtu-profile-builder` from the running Gitea instance.
---
## Two-Layer Validation Structure
### Layer 1 — Structural Validation
Seven deterministic test points encoded as assertions in the DTU profile:
1. **Mode inactive blocks delegate.** When context-intelligence mode is not active, delegate calls to mode-scoped agents are blocked.
2. **Mode activates correctly.** Activating the mode enforces tool policies and injects both @mention context files into the session.
3. **Both specialist agents available under mode.** `context-intelligence-design-facilitator` AND `context-intelligence-tool-designer` are invokable while mode is active.
4. **Tool-designer NOT accessible without mode.** `context-intelligence-tool-designer` is not invokable when mode is inactive.
5. **`domain-concepts.md` exists after Phase 0 with all required fields per entry.** Required fields: `User intent`, `Agreed definition`, `Boundary conditions`, `Nesting`, `Data availability`, `Explicitly excludes`.
6. **`domain-signals.md` enriched after Phase 2.** Each signal entry contains: `detection_strategy`, `detection_notes`, `ai_dependency`, `reasoning_requirement`, `suggested_primitive`.
7. **`context-intelligence-primitives-reference.md` NOT in session context when mode inactive.** The primitives reference must not leak into sessions where mode has not been activated.
8. **`context-intelligence-strategy.md` absent when mode inactive.** The thin mode-orientation strategy file must NOT appear in session context unless the context-intelligence mode is active (parallels point 7; guards mode-only injection via `contributes.context:`).
9. **Strategy-file pointer correctness.** Every referenced path and skill name inside `context-intelligence-strategy.md` resolves to an existing file/skill in the bundle (guards against rename-induced dead pointers). These are NON-loading references (plain paths / skill names), in contrast to the LOADING `@mention` that `session-navigator` uses for the discipline file — this check covers the non-loading pointers.
10. **Tool-design enrichment marker present in-mode / absent in baseline.** The new R1 marker string **"module vs CLI"** is PRESENT in tool-design context when the mode is active, and ABSENT from the always-on baseline. (Anchors the test on a concrete, buildable marker introduced by this change, rather than the non-falsifiable "enriched content loads only when mode active".)
### Layer 2 — Behavioural Evaluation
Run by `amplifier-bundle-evaluation` inside the same DTU, after Layer 1 passes.
**Scenario A — concept elicitation:**
User enters mode and says: *"investigate how testing sessions are performing."*
Pass criterion: `domain-concepts.md` contains a user-confirmed definition for "testing session" with non-empty `Boundary conditions`, `Data availability`, and `Explicitly excludes`. The definition must not reference Amplifier agent names.
**Scenario B — signal gap loop:**
Tool-designer encounters an ambiguous concept during Phase 2 or Phase 3.
Pass criterion: A `signal-gaps.md` entry is written with `Gap type`, `Question`, and `Blocks` fields; the facilitator resolves it; the entry status changes to `resolved`.
**Scenario C — bounded navigation (discipline still fires after extraction):**
Reuses the workspace harness `evaluations/01-delegation-runaway` with `SCENARIO=disk_absent`
(the `after` arm carries the local working branch). `session-navigator`, given a non-existent
session ID against a seeded corpus, must terminate in ≤3 strategies with "not found".
Pass criterion: `runaway_detected=false`, `meta.exit_code=0`, and bounded `context_health`
(peak/total tokens not climbing; `compaction_count` 0–1). This proves the `@mention`ed
`navigation-budget-discipline.md` is actually followed.
---
## Iteration Loop
```
Layer 1 → Layer 2 → compare against success criteria in evaluation-scenarios.md
→ route delta back via mode's Phase 4 validation gap protocol
```
- **Signal or concept definition wrong** → re-enter Phase 0 or Phase 1 via Facilitator.
- **Detection strategy or tool choice wrong** → re-enter Phase 2 or Phase 3 via Tool Designer.
**Iteration is the user's call, not the agent's.** This skill produces scenarios; the user decides when to iterate.