autolab-managed-experiment

---
name: autolab-managed-experiment
description: "Run one Autolab benchmark experiment safely on Hugging Face Jobs. Use when a planner, reviewer, or experiment worker is preparing, auditing, launching, or reviewing a single train.py hypothesis against the current local promoted master."
version: 1.0.0
metadata:
  hermes:
    category: autolab
    requires_toolsets: [terminal]
---

Use this for any single Autolab experiment that should result in exactly one
managed benchmark run.

## Workflow

1. Load the local operator env and refresh from local master:
   - `. ~/.autolab/credentials`
   - `uv run scripts/refresh_master.py --fetch-dag`
2. Edit only `train.py` for the single intended hypothesis.
3. Run preflight before launch:
   - `uv run scripts/hf_job.py preflight`
4. Launch exactly one managed experiment:
   - `uv run scripts/hf_job.py launch --mode experiment`
5. Stream logs and parse the final metric:
   - `uv run scripts/hf_job.py logs <JOB_ID> --follow --output /tmp/autolab-run.log`
   - `uv run scripts/parse_metric.py /tmp/autolab-run.log`
6. Record the run locally:
   - `uv run scripts/submit_patch.py --comment "..."`
7. Promotion is local and only happens if `val_bpb` beats current master.

## Guardrails

- Treat `train_orig.py` as the refreshed local-master base. If preflight reports
  multiple known hypothesis categories, stop and inspect the diff before
  launching.
- Ignore repo git `main` and `origin/main` when judging freshness. In this rig
  repo those refs describe control-plane history, not the benchmark master. The
  comparable base is whatever `refresh_master.py` just wrote into
  `train_orig.py`, `research/live/master.json`, and `research/results.tsv`.
- Never run `uv run scripts/hf_job.py launch --mode prepare` from an
  experiment-scoped worktree. `prepare` is shared bootstrap work, not
  per-experiment work.
- Do not launch a second experiment job for the same experiment unless you have a
  specific reason and intentionally override the duplicate check.
- If the workspace looks stale against the current local master, stop and rewrite
  the experiment rather than rationalizing the mismatch.

## Fast Checks

- `uv run scripts/hf_job.py preflight --json`
  Use this when you need to inspect the diff preview, active conflicts, or
  detected change categories programmatically.
- `uv run scripts/trackio_reporter.py summary --max-jobs 25`
  Use this to confirm the experiment id or hypothesis is not already active.