pr-test

---
name: pr-test
description: "E2E manual testing of PRs/branches using docker compose, agent-browser, and API calls. TRIGGER when user asks to manually test a PR, test a feature end-to-end, or run integration tests against a running system."
user-invocable: true
argument-hint: "[worktree path or PR number] — tests the PR in the given worktree. Optional flags: --fix (auto-fix issues found)"
metadata:
  author: autogpt-team
  version: "2.3.0"
---

# Manual E2E Test

Test a PR/branch end-to-end by building the full platform, interacting via browser and API, capturing screenshots, and reporting results.

**Changelog 2.2.0** — auth flow updated for Better Auth (Supabase signup is
gone), env-setup gaps closed, a proven Playwright fallback for agent-browser,
a billing-test trap that produces false passes, safer process cleanup, and a
mock-provider pattern for deterministic $0 testing. Learned on
[#14206](https://github.com/Significant-Gravitas/AutoGPT/pull/14206) — see the
[evidence comment](https://github.com/Significant-Gravitas/AutoGPT/pull/14206#issuecomment-5511429524).
**2.2.1** — corrects two claims from 2.2.0 that didn't survive live-stack
verification (JWKS does **not** rotate on a frontend restart; the local
Postgres port is `5432`, not `54322`) and hardens the auth setup (explicit
password-length/allowlist/rate-limit failure modes, fail-fast on an empty
token, password kept out of process args).
**2.3.0** — screenshots are posted as GitHub comment attachments
(`gh pr comment --attach`, needs gh >= 2.99.0) instead of being pushed to a
`test-screenshots/*` branch, which accumulated one branch per tested PR and
made every old report's images depend on that branch surviving.

## Critical Requirements

These are NON-NEGOTIABLE. Every test run MUST satisfy ALL the following:

### 1. Screenshots at Every Step
- Take a screenshot at EVERY significant test step — not just at the end
- Every test scenario MUST have at least one BEFORE and one AFTER screenshot
- Name screenshots sequentially: `{NN}-{action}-{state}.png` (e.g., `01-credits-before.png`, `02-credits-after.png`)
- If a screenshot is missing for a scenario, the test is INCOMPLETE — go back and take it

### 2. Screenshots MUST Be Posted to PR
- Attach ALL screenshots to the PR comment with `gh pr comment --attach` (see Step 7). **Never push image files to a repo branch** — attachments live in GitHub's own asset store and leave nothing behind in the repo
- Post a PR comment with ALL screenshots embedded inline, each with its explanation
- This is NOT optional — every test run MUST end with a PR comment containing screenshots
- If the upload fails, retry. If it still fails, list failed files and require manual drag-and-drop/paste attachment in the PR comment, and mark the run INCOMPLETE

### 3. State Verification with Before/After Evidence
- For EVERY state-changing operation (API call, user action), capture the state BEFORE and AFTER
- Log the actual API response values (e.g., `credits_before=100, credits_after=95`)
- Screenshot MUST show the relevant UI state change
- Compare expected vs actual values explicitly — do not just eyeball it

### 4. Negative Test Cases Are Mandatory
- Test at least ONE negative case per feature (e.g., insufficient credits, invalid input, unauthorized access)
- Verify error messages are user-friendly and accurate
- Verify the system state did NOT change after a rejected operation

### 5. Test Report Must Include Full Evidence
Each test scenario in the report MUST have:
- **Steps**: What was done (exact commands or UI actions)
- **Expected**: What should happen
- **Actual**: What actually happened
- **API Evidence**: Before/after API response values for state-changing operations
- **Screenshot Evidence**: Before/after screenshots with explanations

## State Manipulation for Realistic Testing

**Billing-test trap — this one produces a false pass, not a visible failure.**
LLM block cost filters key on the *platform-owned* credential id. A run made
with the test user's own API key bills nothing by design, so a credits
before/after assertion silently passes on zero deltas either way. Any test
that verifies credit reconciliation MUST use the system credential, not a
user-supplied key. Related: Ollama block entries are configured with an
explicit `$0` run-based cost (`BlockCostType.RUN, cost_amount=0` in
`block_cost_config.py`), not a token-metered one — so pre/post-flight cost
deltas are always 0 for them regardless of credential. Never use an Ollama
model to test credit reconciliation; pick any hosted model billed through the
system credential instead.

When testing features that depend on specific states (rate limits, credits, quotas):

1. **Use Redis CLI to set counters directly:**
   ```bash
   # Find the Redis container
   REDIS_CONTAINER=$(docker ps --format '{{.Names}}' | grep redis | head -1)
   # Set a key with expiry
   docker exec $REDIS_CONTAINER redis-cli SET key value EX ttl
   # Example: Set rate limit counter to near-limit
   docker exec $REDIS_CONTAINER redis-cli SET "rate_limit:user:$PR_TEST_USER_EMAIL" 99 EX 3600
   # Example: Check current value
   docker exec $REDIS_CONTAINER redis-cli GET "rate_limit:user:$PR_TEST_USER_EMAIL"
   ```

2. **Use API calls to check before/after state:**
   ```bash
   # BEFORE: Record current state
   BEFORE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   echo "Credits BEFORE: $BEFORE"

   # Perform the action...

   # AFTER: Record new state and compare
   AFTER=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   echo "Credits AFTER: $AFTER"
   echo "Delta: $(( BEFORE - AFTER ))"
   ```

3. **Take screenshots BEFORE and AFTER state changes** — the UI must reflect the backend state change

4. **Never rely on mocked/injected browser state** — always use real backend state. Do NOT use `agent-browser eval` to fake UI state. The backend must be the source of truth.

5. **Use direct DB queries when needed:**
   ```bash
   # Query via Supabase's PostgREST or docker exec into the DB
   docker exec supabase-db psql -U supabase_admin -d postgres -c "SELECT credits FROM user_credits WHERE user_id = '...';"
   ```

6. **After every API test, verify the state change actually persisted:**
   ```bash
   # Example: After a credits purchase, verify DB matches API
   API_CREDITS=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   DB_CREDITS=$(docker exec supabase-db psql -U supabase_admin -d postgres -t -c "SELECT credits FROM user_credits WHERE user_id = '...';" | tr -d ' ')
   [ "$API_CREDITS" = "$DB_CREDITS" ] && echo "CONSISTENT" || echo "MISMATCH: API=$API_CREDITS DB=$DB_CREDITS"
   ```

## Arguments

- `$ARGUMENTS` — worktree path (e.g. `$REPO_ROOT`) or PR number
- If `--fix` flag is present, auto-fix bugs found and push fixes (like pr-address loop)

## Step 0: Resolve the target

```bash
# If argument is a PR number, find its worktree
gh pr view {N} --json headRefName --jq '.headRefName'
# If argument is a path, use it directly
```

Determine:
- `REPO_ROOT` — the root repo directory: `git -C "$WORKTREE_PATH" worktree list | head -1 | awk '{print $1}'` (or `git rev-parse --show-toplevel` if not a worktree)
- `WORKTREE_PATH` — the worktree directory
- `PLATFORM_DIR` — `$WORKTREE_PATH/autogpt_platform`
- `BACKEND_DIR` — `$PLATFORM_DIR/backend`
- `FRONTEND_DIR` — `$PLATFORM_DIR/frontend`
- `PR_NUMBER` — the PR number (from `gh pr list --head $(git branch --show-current)`)
- `PR_TITLE` — the PR title, slugified (e.g. "Add copilot permissions" → "add-copilot-permissions")
- `RESULTS_DIR` — `$REPO_ROOT/test-results/PR-{PR_NUMBER}-{slugified-title}`

Create the results directory:
```bash
PR_NUMBER=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json number --jq '.[0].number')
PR_TITLE=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json title --jq '.[0].title' | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | head -c 50)
RESULTS_DIR="$REPO_ROOT/test-results/PR-${PR_NUMBER}-${PR_TITLE}"
mkdir -p $RESULTS_DIR
```

**Test user credentials** — required to log into the UI or call authenticated APIs.

NEVER hardcode these in this SKILL, a PR comment, a screenshot, or any committed file. Sources, in priority order:

1. **Env vars** (CI / preconfigured local shell): `$PR_TEST_USER_EMAIL` + `$PR_TEST_USER_PASSWORD`. If both are set, use them.
2. **Interactive prompt** (everything else, including dev-preview runs): if the env vars are not set, ASK the user at the start of the run — e.g. "I need test-user credentials for this run; paste the email and password." Hold them in shell vars only for the duration of the run. Do not echo, log, or write them to disk.

Acquire the variables — env first, prompt if missing — and only then lock them in:

```bash
# 1. Prefer env vars (CI / preconfigured shell). Prompt only for the
#    specific var that is unset so an already-exported credential is
#    not overwritten by the prompt when only the other one is missing.
if [ -z "${PR_TEST_USER_EMAIL:-}" ] || [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
  echo "Test user credentials required for this run."
  if [ -z "${PR_TEST_USER_EMAIL:-}" ]; then
    read -r -p   "Email:    " PR_TEST_USER_EMAIL
  fi
  if [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
    read -r -s -p "Password: " PR_TEST_USER_PASSWORD
    echo
  fi
  export PR_TEST_USER_EMAIL PR_TEST_USER_PASSWORD
fi

# 2. Lock them in — fail loudly if either is STILL unset (e.g. the user
#    pressed Enter on an empty prompt). The error message names the var so
#    the agent / operator knows what to fix.
: "${PR_TEST_USER_EMAIL:?PR_TEST_USER_EMAIL is empty after env+prompt — supply a value before re-running}"
: "${PR_TEST_USER_PASSWORD:?PR_TEST_USER_PASSWORD is empty after env+prompt — supply a value before re-running}"
```

For **local docker-compose** runs, a fresh dev user is created on first call to the signup snippet below. For **dev-preview** runs, the test user lives in the project's hosted auth backend — ask the user for the current valid credentials each session (the previously-shared `[email protected]` test account was disabled on 2026-05-23 after its credentials leaked into this very SKILL — do NOT re-introduce a default). **`PR_TEST_USER_PASSWORD` should always be a throwaway/test-only credential, never a real account's password** — the auth requests in 3h go over plain HTTP on `localhost:3000` for local runs, which has no transport encryption. If a dev-preview run's target isn't on `localhost`, confirm it's HTTPS before sending credentials to it.

## Step 1: Understand the PR

Before testing, understand what changed:

```bash
cd $WORKTREE_PATH

# Read PR description to understand the WHY
gh pr view {N} --json body --jq '.body'

git log --oneline dev..HEAD | head -20
git diff dev --stat
```

Read the PR description (Why / What / How) and changed files to understand:
0. **Why** does this PR exist? What problem does it solve?
1. **What** feature/fix does this PR implement?
2. **How** does it work? What's the approach?
3. What components are affected? (backend, frontend, copilot, executor, etc.)
4. What are the key user-facing behaviors to test?

## Step 2: Write test scenarios

Based on the PR analysis, write a test plan to `$RESULTS_DIR/test-plan.md`:

```markdown
# Test Plan: PR #{N} — {title}

## Scenarios
1. [Scenario name] — [what to verify]
2. ...

## API Tests (if applicable)
1. [Endpoint] — [expected behavior]
   - Before state: [what to check before]
   - After state: [what to verify changed]

## UI Tests (if applicable)
1. [Page/component] — [interaction to test]
   - Screenshot before: [what to capture]
   - Screenshot after: [what to capture]

## Negative Tests (REQUIRED — at least one per feature)
1. [What should NOT happen] — [how to trigger it]
   - Expected error: [what error message/code]
   - State unchanged: [what to verify did NOT change]
```

**Be critical** — include edge cases, error paths, and security checks. Every scenario MUST specify what screenshots to take and what state to verify.

## Step 3.0: Claim the testing lock (coordinate parallel agents)

Multiple worktrees share the same host — Docker infra (postgres, redis, clamav), app ports (3000/8006/…), and the test user. Two agents running `/pr-test` concurrently will corrupt each other's state (connection-pool exhaustion, port binds failing silently, cross-test assertions). Use the root-worktree lock file to take turns.

### Lock file contract

Path (**always** the root worktree so all siblings see it): `$REPO_ROOT/.ign.testing.lock`

Body (one `key=value` per line):
```
holder=<pr-XXXXX-purpose>
pid=<pid-or-"self">
started=<iso8601>
heartbeat=<iso8601, updated every ~2 min>
worktree=<full path>
branch=<branch name>
intent=<one-line description + rough duration>
```

### Claim

```bash
LOCK=$REPO_ROOT/.ign.testing.lock
NOW=$(date -u +%Y-%m-%dT%H:%MZ)
STALE_AFTER_MIN=5

if [ -f "$LOCK" ]; then
  HB=$(grep '^heartbeat=' "$LOCK" | cut -d= -f2)
  HB_EPOCH=$(date -j -f '%Y-%m-%dT%H:%MZ' "$HB" +%s 2>/dev/null || date -d "$HB" +%s 2>/dev/null || echo 0)
  AGE_MIN=$(( ( $(date -u +%s) - HB_EPOCH ) / 60 ))
  if [ "$AGE_MIN" -gt "$STALE_AFTER_MIN" ]; then
    echo "WARN: stale lock (${AGE_MIN}m old) — reclaiming"
    cat "$LOCK" | sed 's/^/  stale: /'
  else
    echo "Another agent holds the lock:"; cat "$LOCK"
    echo "Wait until released or resume after $((STALE_AFTER_MIN - AGE_MIN))m."
    exit 1
  fi
fi

cat > "$LOCK" <<EOF
holder=pr-${PR_NUMBER}-e2e
pid=self
started=$NOW
heartbeat=$NOW
worktree=$WORKTREE_PATH
branch=$(cd $WORKTREE_PATH && git branch --show-current)
intent=E2E test PR #${PR_NUMBER}, native mode, ~60min
EOF
echo "Lock claimed"
```

### Heartbeat (MUST run in background during the whole test)

Without a heartbeat a crashed agent keeps the lock forever. Run this as a background process right after claim:

```bash
(while true; do
   sleep 120
   [ -f "$LOCK" ] || exit 0   # lock released → exit heartbeat
   perl -i -pe "s/^heartbeat=.*/heartbeat=$(date -u +%Y-%m-%dT%H:%MZ)/" "$LOCK"
 done) &
HEARTBEAT_PID=$!
echo "$HEARTBEAT_PID" > /tmp/pr-test-heartbeat.pid
```

### Release (always — even on failure)

```bash
kill "$HEARTBEAT_PID" 2>/dev/null
rm -f "$LOCK" /tmp/pr-test-heartbeat.pid
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] released lock" \
    >> $REPO_ROOT/.ign.testing.log
```

Use a `trap` so release runs even on `exit 1`:
```bash
trap 'kill "$HEARTBEAT_PID" 2>/dev/null; rm -f "$LOCK"' EXIT INT TERM
```

### **Release the lock AS SOON AS the test run is done**

The lock guards **test execution**, not **app lifecycle**. Once Step 5 (record results) and Step 6 (post PR comment) are complete, release the lock IMMEDIATELY — even if:

- The native `poetry run app` / `pnpm dev` processes are still running so the user can keep poking at the app manually.
- You're leaving docker containers up.
- You're tailing logs for a minute or two.

Keeping the lock held past the test run is the single most common way `/pr-test` stalls other agents. **The app staying up is orthogonal to the lock; don't conflate them.** Sibling worktrees running their own `/pr-test` will kill the stray processes and free the ports themselves (Step 3c/3e-native handle that) — they just need the lock file gone.

Concretely, the sequence at the end of every `/pr-test` run (success or failure) is:

```bash
# 1. Write the final report + post PR comment — done above in Step 5/6.
# 2. Release the lock right now, even if the app is still up.
kill "$HEARTBEAT_PID" 2>/dev/null
rm -f "$LOCK" /tmp/pr-test-heartbeat.pid
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] released lock (app may still be running)" \
    >> $REPO_ROOT/.ign.testing.log
# 3. Optionally leave the app running and note it so the user knows:
echo "Native stack still running on :3000 / :8006 for manual poking. Kill with:"
echo "  pkill -9 -f 'poetry run app'; pkill -9 -f 'next-server|next dev'"
```

If a sibling agent's `/pr-test` needs to take over, it'll do the kill+rebuild dance from Step 3c/3e-native on its own — your only job is to not hold the lock file past the end of your test.

### Shared status log

`$REPO_ROOT/.ign.testing.log` is an append-only channel any agent can read/write. Use it for "I'm waiting", "I'm done, resources free", or post-run notes:
```bash
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] <message>" \
    >> $REPO_ROOT/.ign.testing.log
```

## Step 3: Environment setup

### 3a. Copy .env files from the root worktree

The root worktree (`$REPO_ROOT`) has the canonical `.env` files with all API keys. Copy them to the target worktree:

```bash
# CRITICAL: .env files are NOT checked into git. They must be copied manually.
cp $REPO_ROOT/autogpt_platform/.env $PLATFORM_DIR/.env
cp $REPO_ROOT/autogpt_platform/backend/.env $BACKEND_DIR/.env
cp $REPO_ROOT/autogpt_platform/frontend/.env $FRONTEND_DIR/.env
```

**A copy from the root worktree is no longer sufficient on a recent `dev`** —
auth moved from Supabase to Better Auth (see 3h) and two vars are easy to
miss because nothing fails loudly without them, it just 401s later:

- `$BACKEND_DIR/.env` needs `JWT_JWKS_URL` — the Better Auth JWKS endpoint the
  backend verifies tokens against. The `localhost:3000` value below is for
  **native mode only**. In docker mode it's harmless to have it in `.env`
  because `docker-compose.platform.yml` overrides it with the
  Compose-reachable `http://frontend:3000/api/auth/jwks` — but if you ever run
  the backend against this `.env` value directly (bypassing Compose), a
  `localhost` value inside a container resolves to itself, not the frontend,
  and every call 401s with no other symptom.
- `$FRONTEND_DIR/.env` needs its **own** `DATABASE_URL` — Better Auth runs
  inside the Next.js app and talks to Postgres directly, it does not go
  through the backend. **Derive it from `$BACKEND_DIR/.env`'s `DB_USER` /
  `DB_PASS` / `DB_PORT` / `DB_NAME` rather than copying
  `frontend/.env.default`'s placeholder verbatim** — the placeholder happens
  to match the stock local defaults, but if `backend/.env`'s `DB_PASS` was
  ever customized (rotated secret, non-default port), copying the placeholder
  silently points Better Auth at the wrong database instead of the one the
  rest of the stack actually uses.

```bash
# [ -n ... ], not grep -q alone — a present-but-empty JWT_JWKS_URL= would
# otherwise be treated as "already set" and skip the fallback, leaving the
# backend without a JWKS endpoint to verify tokens against.
[ -n "$(grep '^JWT_JWKS_URL=' $BACKEND_DIR/.env | cut -d= -f2-)" ] || echo "JWT_JWKS_URL=http://localhost:3000/api/auth/jwks" >> $BACKEND_DIR/.env  # native mode only — see note above

if [ -n "$(grep '^DATABASE_URL=' $FRONTEND_DIR/.env | cut -d= -f2-)" ]; then
  # grep -q alone matches a present-but-empty DATABASE_URL= too, which would
  # otherwise skip derivation and leave Better Auth pointed at nothing.
  echo "Frontend DATABASE_URL: already set (not touching it)"
else
  # cut -f2 (not -f2-) truncates any value containing '=' (base64 secrets do); -f2- keeps the rest.
  DB_USER=$(grep '^DB_USER=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_PASS=$(grep '^DB_PASS=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_PORT=$(grep '^DB_PORT=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_NAME=$(grep '^DB_NAME=' $BACKEND_DIR/.env | cut -d= -f2-)
  : "${DB_USER:?}" "${DB_PASS:?}" "${DB_PORT:?}" "${DB_NAME:?}"  # fail loudly, not with a silently-empty URL
  # Percent-encode user/pass — a raw '@', '#', '?', '%', or ':' in either would
  # otherwise be misparsed as URL structure instead of credential content.
  # Via env vars, not `jq --arg`, which would put DB_PASS in the process arglist.
  DB_USER_ENC=$(DB_USER_VAL="$DB_USER" jq -rn '$ENV.DB_USER_VAL|@uri')
  DB_PASS_ENC=$(DB_PASS_VAL="$DB_PASS" jq -rn '$ENV.DB_PASS_VAL|@uri')
  echo "DATABASE_URL=postgresql://${DB_USER_ENC}:${DB_PASS_ENC}@localhost:${DB_PORT}/${DB_NAME}" >> $FRONTEND_DIR/.env
  # Reconstructed, not regex-redacted — a password containing '@' would otherwise
  # leak its tail past a naive "redact up to the first @" pattern.
  echo "Frontend DATABASE_URL: postgresql://${DB_USER_ENC}:***@localhost:${DB_PORT}/${DB_NAME}"
fi
```

### 3b. Configure copilot authentication

The copilot needs an LLM API to function. Two approaches (try subscription first):

#### Option 1: Subscription mode (preferred — uses your Claude Max/Pro subscription)

The `claude_agent_sdk` Python package **bundles its own Claude CLI binary** — no need to install `@anthropic-ai/claude-code` via npm. The backend auto-provisions credentials from environment variables on startup.

Run the helper script to extract tokens from your host and auto-update `backend/.env` (works on macOS, Linux, and Windows/WSL):

```bash
# Extracts OAuth tokens and writes CLAUDE_CODE_OAUTH_TOKEN + CLAUDE_CODE_REFRESH_TOKEN into .env
bash $BACKEND_DIR/scripts/refresh_claude_token.sh --env-file $BACKEND_DIR/.env
```

**How it works:** The script reads the OAuth token from:
- **macOS**: system keychain (`"Claude Code-credentials"`)
- **Linux/WSL**: `~/.claude/.credentials.json`
- **Windows**: `%APPDATA%/claude/.credentials.json`

It sets `CLAUDE_CODE_OAUTH_TOKEN`, `CLAUDE_CODE_REFRESH_TOKEN`, and `CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true` in the `.env` file. On container startup, the backend auto-provisions `~/.claude/.credentials.json` inside the container from these env vars. The SDK's bundled CLI then authenticates using that file. No `claude login`, no npm install needed.

**Note:** The OAuth token expires (~24h). If copilot returns auth errors, re-run the script and restart: `$BACKEND_DIR/scripts/refresh_claude_token.sh --env-file $BACKEND_DIR/.env && docker compose up -d copilot_executor`

#### Option 2: OpenRouter API key mode (fallback)

If subscription mode doesn't work, switch to API key mode using OpenRouter:

```bash
# In $BACKEND_DIR/.env, ensure these are set:
CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=false
CHAT_API_KEY=<value of OPEN_ROUTER_API_KEY from the same .env>
CHAT_BASE_URL=https://openrouter.ai/api/v1
CHAT_USE_CLAUDE_AGENT_SDK=true
```

Use `sed` to update these values:
```bash
ORKEY=$(grep "^OPEN_ROUTER_API_KEY=" $BACKEND_DIR/.env | cut -d= -f2)
[ -n "$ORKEY" ] || { echo "ERROR: OPEN_ROUTER_API_KEY is missing in $BACKEND_DIR/.env"; exit 1; }
perl -i -pe 's/CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true/CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=false/' $BACKEND_DIR/.env
# Add or update CHAT_API_KEY and CHAT_BASE_URL
grep -q "^CHAT_API_KEY=" $BACKEND_DIR/.env && perl -i -pe "s|^CHAT_API_KEY=.*|CHAT_API_KEY=$ORKEY|" $BACKEND_DIR/.env || echo "CHAT_API_KEY=$ORKEY" >> $BACKEND_DIR/.env
grep -q "^CHAT_BASE_URL=" $BACKEND_DIR/.env && perl -i -pe 's|^CHAT_BASE_URL=.*|CHAT_BASE_URL=https://openrouter.ai/api/v1|' $BACKEND_DIR/.env || echo "CHAT_BASE_URL=https://openrouter.ai/api/v1" >> $BACKEND_DIR/.env
```

### 3c. Stop conflicting containers

```bash
# Stop any running app containers (keep infra: supabase, redis, rabbitmq, clamav)
docker ps --format "{{.Names}}" | grep -E "rest_server|executor|copilot|websocket|database_manager|scheduler|notification|frontend|migrate" | while read name; do
  docker stop "$name" 2>/dev/null
done
```

**Native mode also:** when running the app natively (see 3e-native), kill any stray host processes and free the app ports before starting — otherwise `poetry run app` and `pnpm dev` will fail to bind.

**Kill by port, not by broad process pattern.** A pattern-based
`pkill -f "python.*backend"` (or anything matching by worktree cwd) is too
coarse on a host running several worktrees — it has taken out the frontend
and a mock server sitting on other ports along with the intended backend
process. Target the pid actually holding each port instead — this only kills
whoever is bound to that specific port, which is narrower than a pattern
match, but **it is not worktree isolation**: if a sibling worktree's own dev
server happens to be using one of these ports (e.g. its frontend also on
:3000), this kills that too. `lsof` tells you who holds the port, not who
owns it — a `docker-proxy` pid there means a compose stack owns it.

```bash
# Free app ports one at a time — errors per port are ignored (port may simply
# be unused). `xargs -r` is GNU-only (macOS xargs rejects -r); the `[ -n ]`
# guard below is the portable equivalent.
for port in 3000 8006 8001 8002 8005 8008; do
  pids=$(lsof -ti :$port -sTCP:LISTEN 2>/dev/null)
  [ -n "$pids" ] && kill -9 $pids 2>/dev/null || true
done
```

### 3e-native. Run the app natively (PREFERRED for iterative dev)

Native mode runs infra (postgres, supabase, redis, rabbitmq, clamav) in docker but runs the backend and frontend directly on the host. This avoids the 3-8 minute `docker compose build` cycle on every backend change — code edits are picked up on process restart (seconds) instead of a full image rebuild.

**When to prefer native mode (default for this skill):**
- Iterative dev/debug loops where you're editing backend or frontend code between test runs
- Any PR that touches Python/TS source but not Dockerfiles, compose config, or infra images
- Fast repro of a failing scenario — restart `poetry run app` in a couple of seconds

**When to prefer docker mode (3e fallback):**
- Testing changes to `Dockerfile`, `docker-compose.yml`, or base images
- Production-parity smoke tests (exact container env, networking, volumes)
- CI-equivalent runs where you need the exact image that'll ship

**Note on 3b (copilot auth):** no npm install anywhere. `poetry install` pulls in `claude_agent_sdk`, which ships its own Claude CLI binary — available on `PATH` whenever you run commands via `poetry run` (native) OR whenever the copilot_executor container is built from its Poetry lockfile (docker). The OAuth token extraction still applies (same `refresh_claude_token.sh` call).

**Preamble:** before starting native, run the kill-stray + free-ports block from 3c's "Native mode also" subsection.

**1. Start infra only (one-time per session):**

```bash
cd $PLATFORM_DIR && docker compose --profile local up deps --detach --remove-orphans --build
```

This brings up postgres/supabase/redis/rabbitmq/clamav and skips all app services.

**2. Start the backend natively:**

```bash
cd $BACKEND_DIR && (poetry run app 2>&1 | tee .ign.application.logs) &
```

`poetry run app` spawns **all** app subprocesses — `rest_server`, `executor`, `copilot_executor`, `websocket`, `scheduler`, `notification_server`, `database_manager` — inside ONE parent process. No separate containers, no separate terminals. The `.ign.application.logs` prefix is already gitignored.

**3. Wait for the backend on :8006 BEFORE starting the frontend.** This ordering matters — the frontend's `pnpm dev` startup invokes `generate-api-queries`, which fetches `/openapi.json` from the backend. If the backend isn't listening yet, `pnpm dev` fails immediately.

```bash
for i in $(seq 1 60); do
  if [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8006/docs 2>/dev/null)" = "200" ]; then
    echo "Backend ready"
    break
  fi
  sleep 2
done
```

**4. Start the frontend natively:**

```bash
cd $FRONTEND_DIR && (pnpm dev 2>&1 | tee .ign.frontend.logs) &
```

**5. Wait for the frontend on :3000:**

```bash
for i in $(seq 1 60); do
  if [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:3000 2>/dev/null)" = "200" ]; then
    echo "Frontend ready"
    break
  fi
  sleep 2
done
```

Once both are up, skip 3e/3f and go straight to **3g/3h** (feature flags / test user creation).

### 3e. Build and start (docker — fallback)

```bash
cd $PLATFORM_DIR && docker compose build --no-cache 2>&1 | tail -20
if [ ${PIPESTATUS[0]} -ne 0 ]; then echo "ERROR: Docker build failed"; exit 1; fi

cd $PLATFORM_DIR && docker compose up -d 2>&1 | tail -20
if [ ${PIPESTATUS[0]} -ne 0 ]; then echo "ERROR: Docker compose up failed"; exit 1; fi
```

**Note:** If the container appears to be running old code (e.g. missing PR changes), use `docker compose build --no-cache` to force a full rebuild. Docker BuildKit may sometimes reuse cached `COPY` layers from a previous build on a different branch.

**Expected time: 3-8 minutes** for build, 5-10 minutes with `--no-cache`.

### 3f. Wait for services to be ready

```bash
# Poll until backend and frontend respond
for i in $(seq 1 60); do
  BACKEND=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8006/docs 2>/dev/null)
  FRONTEND=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null)
  if [ "$BACKEND" = "200" ] && [ "$FRONTEND" = "200" ]; then
    echo "Services ready"
    break
  fi
  sleep 5
done
```


### 3h. Create test user and get auth token

The platform moved off Supabase auth to Better Auth, embedded in the
Next.js app at `/api/auth/*`. Signup and sign-in both go through the frontend
now, not Kong on :8000 — and `/api/auth/token` mints a backend-API JWT from a
**session cookie**, it does not accept credentials directly, so sign-in has to
happen first to get that cookie.

Better Auth's default minimum password length is **12 characters** — shorter
values fail signup with `PASSWORD_TOO_SHORT` and every step below degrades
silently into an empty token unless you check for it.

```bash
COOKIE_JAR=$(mktemp)
trap 'rm -f "$COOKIE_JAR"' EXIT  # cleans up on early exit too, not just the happy path
# Via env vars, not `jq --arg`, which would put the password in the process arglist.
AUTH_PAYLOAD=$(PR_TEST_USER_EMAIL="$PR_TEST_USER_EMAIL" PR_TEST_USER_PASSWORD="$PR_TEST_USER_PASSWORD" \
  jq -nc '{email:$ENV.PR_TEST_USER_EMAIL,password:$ENV.PR_TEST_USER_PASSWORD,name:"PR Test User"}')

# Signup (idempotent — a real error body means "already exists" only if you
# check; -d passes the payload as an argument, which leaks the password into
# process listings, so pipe it through stdin with --data-binary @- instead.
# --noproxy guards against an inherited proxy env var routing the password
# through a proxy. --max-time bounds it — without one, a server that accepts
# the connection but never responds hangs setup indefinitely instead of
# reaching the empty-$TOKEN failure check below).
SIGNUP_RESULT=$(printf '%s' "$AUTH_PAYLOAD" | curl -s --max-time 15 --noproxy localhost,127.0.0.1,::1 -X POST 'http://localhost:3000/api/auth/sign-up/email' \
  -H 'Content-Type: application/json' --data-binary @-)
echo "$SIGNUP_RESULT" | grep -qi '"code"' && echo "Signup: $SIGNUP_RESULT"  # log it — "already exists" and "password too short" look identical downstream otherwise

# Sign in — sets the better-auth.session_token cookie in $COOKIE_JAR.
# Capture the body: a failure here (e.g. account exists with a different
# password) otherwise only shows up as an empty $TOKEN with no explanation.
SIGNIN_RESULT=$(printf '%s' "$AUTH_PAYLOAD" | curl -s --max-time 15 --noproxy localhost,127.0.0.1,::1 -c "$COOKIE_JAR" -X POST 'http://localhost:3000/api/auth/sign-in/email' \
  -H 'Content-Type: application/json' --data-binary @-)
echo "$SIGNIN_RESULT" | grep -qi '"code"' && echo "Sign-in: $SIGNIN_RESULT"

# Mint a backend-API JWT from the session cookie
TOKEN=$(curl -s -b "$COOKIE_JAR" 'http://localhost:3000/api/auth/token' | jq -r '.token // ""')
[ -n "$TOKEN" ] || { echo "ERROR: auth setup failed — TOKEN is empty. Check password length (min 12 chars), AUTH_ALLOW_NEW_ACCOUNTS, AUTH_SIGNUP_ALLOWLIST, and rate limiting on /api/auth/*."; exit 1; }
```

**Use this token for ALL API calls:**
```bash
curl -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/...
```

### 3i. Disable onboarding for test user

The frontend redirects to `/onboarding` when the `ONBOARDING_COMPLETE` step is not in `completedSteps`.
Mark it complete via the backend API so every browser test lands on the real feature UI:

```bash
ONBOARDING_RESULT=$(curl -s --max-time 30 -X POST \
  "http://localhost:8006/api/onboarding/step?step=ONBOARDING_COMPLETE" \
  -H "Authorization: Bearer $TOKEN")
echo "Onboarding bypass: $ONBOARDING_RESULT"

# Verify it took effect
ONBOARDING_STATUS=$(curl -s --max-time 30 \
  "http://localhost:8006/api/onboarding/completed" \
  -H "Authorization: Bearer $TOKEN" | jq -r '.is_completed')
echo "Onboarding completed: $ONBOARDING_STATUS"
if [ "$ONBOARDING_STATUS" != "true" ]; then
  echo "ERROR: onboarding bypass failed — browser tests will hit /onboarding instead of the target feature. Investigate before proceeding."
  exit 1
fi
```

## Step 4: Run tests

### Service ports reference

| Service | Port | URL |
|---------|------|-----|
| Frontend | 3000 | http://localhost:3000 |
| Backend REST | 8006 | http://localhost:8006 |
| Supabase Auth (via Kong) | 8000 | http://localhost:8000 |
| Executor | 8002 | http://localhost:8002 |
| Copilot Executor | 8008 | http://localhost:8008 |
| WebSocket | 8001 | http://localhost:8001 |
| Database Manager | 8005 | http://localhost:8005 |
| Redis | 6379 | localhost:6379 |
| RabbitMQ | 5672 | localhost:5672 |

### API testing

Use `curl` with the auth token for backend API tests. **For EVERY API call that changes state, record before/after values:**

```bash
# Example: List agents
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/graphs | jq . | head -20

# Example: Create an agent
curl -s -X POST http://localhost:8006/api/graphs \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{...}' | jq .

# Example: Run an agent
curl -s -X POST "http://localhost:8006/api/graphs/{graph_id}/execute" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"data": {...}}'

# Example: Get execution results
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8006/api/graphs/{graph_id}/executions/{exec_id}" | jq .
```

**State verification pattern (use for EVERY state-changing API call):**
```bash
# 1. Record BEFORE state
BEFORE_STATE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/{resource} | jq '{relevant_fields}')
echo "BEFORE: $BEFORE_STATE"

# 2. Perform the action
ACTION_RESULT=$(curl -s -X POST ... | jq .)
echo "ACTION RESULT: $ACTION_RESULT"

# 3. Record AFTER state
AFTER_STATE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/{resource} | jq '{relevant_fields}')
echo "AFTER: $AFTER_STATE"

# 4. Log the comparison
echo "=== STATE CHANGE VERIFICATION ==="
echo "Before: $BEFORE_STATE"
echo "After: $AFTER_STATE"
echo "Expected change: {describe what should have changed}"
```

### Mock-provider pattern (deterministic, $0)

For timeout/latency/error-handling behavior that would otherwise need a real
LLM call, point the OpenAI SDK at a local mock instead — it honours
`OPENAI_BASE_URL`, so a small local Responses-API server can stand in for the
provider while everything downstream (executor, credentials, billing) still
runs for real. This proved out 4/4 test items at $0 in this run. **This
covers LLM blocks (`providers.py`) and the Codex block** — the copilot's own
LLM calls go through `backend/util/clients.py`, which passes `base_url`
explicitly and does not read `OPENAI_BASE_URL`, so this trick doesn't reach
copilot chat.

```bash
# In $BACKEND_DIR/.env, point the OpenAI provider at a local mock server
# that implements the subset of the Responses API you need (e.g. delayed
# responses to test timeout handling, or a 500 to test error surfacing).
# Restart the backend after changing this — it's read at startup.
OPENAI_BASE_URL=http://localhost:{mock_port}/v1
```

**Use a throwaway/dummy provider credential with the mock, never the system
credential** — only the transport is faked, so the credential lookup still
runs for real and whatever key you configure gets sent as a header to your
local mock server. A dummy key also means the mock server's logs (which may
end up pasted into a PR comment) can't leak a real one. Aside from the
credential, this is safe to use for anything that isn't itself testing model
*output* quality.

In **docker mode**, `localhost` inside the backend container isn't reachable
from your host-side mock server — point `OPENAI_BASE_URL` at a
Compose-reachable hostname instead (or `host.docker.internal` with a
`host-gateway` entry), and set it before `docker compose up` or restart the
affected services after changing `$BACKEND_DIR/.env`.

Keep the mock server's own response timeout short — the OpenAI SDK's default
client timeout is 600s, so a hung mock stalls every LLM-backed block for that
long instead of failing fast.

### Browser testing with agent-browser

Primary tool — use this wherever `agent-browser` is installed:

```bash
# Close any existing session
agent-browser close 2>/dev/null || true

# Use --session-name to persist cookies across navigations
# This means login only needs to happen once per test session
agent-browser --session-name pr-test open 'http://localhost:3000/login' --timeout 15000

# Get interactive elements
agent-browser --session-name pr-test snapshot | grep "textbox\|button"

# Login (read creds from env — set PR_TEST_USER_EMAIL / PR_TEST_USER_PASSWORD or ask the user)
agent-browser --session-name pr-test fill {email_ref} "$PR_TEST_USER_EMAIL"
agent-browser --session-name pr-test fill {password_ref} "$PR_TEST_USER_PASSWORD"
agent-browser --session-name pr-test click {login_button_ref}
sleep 5

# Dismiss cookie banner if present
agent-browser --session-name pr-test click 'text=Accept All' 2>/dev/null || true

# Navigate — cookies are preserved so login persists
agent-browser --session-name pr-test open 'http://localhost:3000/copilot' --timeout 10000

# Take screenshot
agent-browser --session-name pr-test screenshot $RESULTS_DIR/01-page.png

# Interact with elements
agent-browser --session-name pr-test fill {ref} "text"
agent-browser --session-name pr-test press "Enter"
agent-browser --session-name pr-test click {ref}
agent-browser --session-name pr-test click 'text=Button Text'

# Read page content
agent-browser --session-name pr-test snapshot | grep "text:"
```

**Key pages:**
- `/copilot` — CoPilot chat (for testing copilot features)
- `/build` — Agent builder (for testing block/node features)
- `/build?flowID={id}` — Specific agent in builder
- `/library` — Agent library (for testing listing/import features)
- `/library/agents/{id}` — Agent detail with run history
- `/marketplace` — Marketplace

**Fallback — if `agent-browser` isn't installed on the host, don't let `npx`
download it.** Use `@playwright/test` instead — the package is already in the
frontend's `node_modules`, but **the Chromium binary itself is not**;
Playwright caches browser binaries separately (`~/.cache/ms-playwright/` by
default) and nothing installs them automatically. Run
`pnpm exec playwright install chromium` once per host if `chromium.launch()`
fails looking for an executable, then use it for finer control over timing
(`waitForSelector`, explicit timeouts) than agent-browser's CLI gives you:

```bash
cd $FRONTEND_DIR  # required — @playwright/test is only declared here, not at the repo root
# Check the installer's own exit status, not grep's — piping through grep to
# drop blank lines would otherwise swallow a real install failure and let
# chromium.launch() below fail later with a much less clear error.
pnpm exec playwright install chromium
[ $? -eq 0 ] || { echo "ERROR: playwright install chromium failed"; exit 1; }

node -e "
const { chromium } = require('@playwright/test');
(async () => {
  const browser = await chromium.launch();
  try {
    const page = await browser.newPage();
    await page.goto('http://localhost:3000/login');
    await page.screenshot({ path: '$RESULTS_DIR/01-login.png' });
  } finally {
    await browser.close();  // otherwise a goto timeout leaks a headless Chromium process
  }
})();
"
```

### Checking logs

**Native mode:** when running via `poetry run app` + `pnpm dev`, all app logs stream to the `.ign.*.logs` files written by the `tee` pipes in 3e-native. `rest_server`, `executor`, `copilot_executor`, `websocket`, `scheduler`, `notification_server`, and `database_manager` are all subprocesses of the single `poetry run app` parent, so their output is interleaved in `.ign.application.logs`.

```bash
# Backend (all app subprocesses interleaved)
tail -f $BACKEND_DIR/.ign.application.logs

# Frontend (Next.js dev server)
tail -f $FRONTEND_DIR/.ign.frontend.logs

# Filter for errors across either log
grep -iE "error|exception|traceback" $BACKEND_DIR/.ign.application.logs | tail -20
grep -iE "error|exception|traceback" $FRONTEND_DIR/.ign.frontend.logs | tail -20
```

**Docker mode:**

```bash
# Backend REST server
docker logs autogpt_platform-rest_server-1 2>&1 | tail -30

# Executor (runs agent graphs)
docker logs autogpt_platform-executor-1 2>&1 | tail -30

# Copilot executor (runs copilot chat sessions)
docker logs autogpt_platform-copilot_executor-1 2>&1 | tail -30

# Frontend
docker logs autogpt_platform-frontend-1 2>&1 | tail -30

# Filter for errors
docker logs autogpt_platform-executor-1 2>&1 | grep -i "error\|exception\|traceback" | tail -20
```

### Copilot chat testing

The copilot uses SSE streaming. To test via API:

```bash
# Create a session
SESSION_ID=$(curl -s -X POST 'http://localhost:8006/api/chat/sessions' \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{}' | jq -r '.id // .session_id // ""')

# Stream a message (SSE - will stream chunks)
curl -N -X POST "http://localhost:8006/api/chat/sessions/$SESSION_ID/stream" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"message": "Hello, what can you help me with?"}' \
  --max-time 60 2>/dev/null | head -50
```

Or test via browser (preferred for UI verification):
```bash
agent-browser --session-name pr-test open 'http://localhost:3000/copilot' --timeout 10000
# ... fill chat input and press Enter, wait 20-30s for response
```

## Step 5: Record results and take screenshots

**Take a screenshot at EVERY significant test step** — before and after interactions, on success, and on failure. This is NON-NEGOTIABLE.

**Required screenshot pattern for each test scenario:**
```bash
# BEFORE the action
agent-browser --session-name pr-test screenshot $RESULTS_DIR/{NN}-{scenario}-before.png

# Perform the action...

# AFTER the action
agent-browser --session-name pr-test screenshot $RESULTS_DIR/{NN}-{scenario}-after.png
```

**Naming convention:**
```bash
# Examples:
# $RESULTS_DIR/01-login-page-before.png
# $RESULTS_DIR/02-login-page-after.png
# $RESULTS_DIR/03-credits-page-before.png
# $RESULTS_DIR/04-credits-purchase-after.png
# $RESULTS_DIR/05-negative-insufficient-credits.png
# $RESULTS_DIR/06-error-state.png
```

**Minimum requirements:**
- At least TWO screenshots per test scenario (before + after)
- At least ONE screenshot for each negative test case showing the error state
- If a test fails, screenshot the failure state AND any error logs visible in the UI

## Step 6: Show results to user with screenshots

**CRITICAL: After all tests complete, you MUST show every screenshot to the user using the Read tool, with an explanation of what each screenshot shows.** This is the most important part of the test report — the user needs to visually verify the results.

For each screenshot:
1. Use the `Read` tool to display the PNG file (Claude can read images)
2. Write a 1-2 sentence explanation below it describing:
   - What page/state is being shown
   - What the screenshot proves (which test scenario it validates)
   - Any notable details visible in the UI

Format the output like this:

```markdown
### Screenshot 1: {descriptive title}
[Read the PNG file here]

**What it shows:** {1-2 sentence explanation of what this screenshot proves}

---
```

After showing all screenshots, output a **detailed** summary table:

| # | Scenario | Result | API Evidence | Screenshot Evidence |
|---|----------|--------|-------------|-------------------|
| 1 | {name} | PASS/FAIL | Before: X, After: Y | 01-before.png, 02-after.png |
| 2 | ... | ... | ... | ... |

**IMPORTANT:** As you show each screenshot and record test results, persist them in shell variables for Step 7:

```bash
# Build these variables during Step 6 — they are required by Step 7's script
# NOTE: declare -A requires Bash 4.0+. This is standard on modern systems (macOS ships zsh
# but Homebrew bash is 5.x; Linux typically has bash 5.x). If running on Bash <4, use a
# plain variable with a lookup function instead.
declare -A SCREENSHOT_EXPLANATIONS=(
  ["01-login-page.png"]="Shows the login page loaded successfully with SSO options visible."
  ["02-builder-with-block.png"]="The builder canvas displays the newly added block connected to the trigger."
  # ... one entry per screenshot, using the same explanations you showed the user above
)

TEST_RESULTS_TABLE="| 1 | Login flow | PASS | N/A | 01-login-before.png, 02-login-after.png |
| 2 | Credits purchase | PASS | Before: 100, After: 95 | 03-credits-before.png, 04-credits-after.png |
| 3 | Insufficient credits (negative) | PASS | Credits: 0, rejected | 05-insufficient-credits-error.png |"
# ... one row per test scenario with actual results
```

## Step 7: Post test report as PR comment with screenshots

Post the report with `gh pr comment --attach`. Each attached file is uploaded to GitHub's own asset store, and any image the body already references by that path is rewritten to point at the uploaded asset. Screenshots never touch a repo branch.

**This step is MANDATORY. Every test run MUST post a PR comment — with the screenshots attached, or an image-free INCOMPLETE report when attachment is unavailable (see the fallback below). Never nothing.**

**Attachments are permanent.** A `user-attachments` asset stays reachable by anyone holding its URL regardless of repo visibility, and deleting the comment does not reliably revoke it — there is no undo. Look at every screenshot for credentials, tokens, or customer data before this step runs.

**Requires `gh` >= 2.99.0** — `--attach` landed there. Distro packages lag (Fedora's rpm tops out at 2.97.0), so check the version you are actually running before building the body:

```bash
# Too-old gh is not fatal here: the report still gets posted, just without
# images, through the same fallback path a failed upload takes (below).
GH_VERSION=$(gh version 2>/dev/null | head -1 | awk '{print $3}')
# Numeric compare, not `sort -V`: BSD sort on macOS may lack -V, and an empty
# substitution would silently read as "too old" on a perfectly good gh.
GH_MAJOR=${GH_VERSION%%.*}; GH_REST=${GH_VERSION#*.}; GH_MINOR=${GH_REST%%.*}
ATTACH_OK=0
ATTACH_SKIP_REASON=""
if [ "${GH_MAJOR:-0}" -gt 2 ] 2>/dev/null || { [ "${GH_MAJOR:-0}" -eq 2 ] && [ "${GH_MINOR:-0}" -ge 99 ]; } 2>/dev/null; then
  ATTACH_OK=1
else
  ATTACH_SKIP_REASON="gh ${GH_VERSION:-not found} is below 2.99.0 — upgrade gh"
  echo "WARN: ${ATTACH_SKIP_REASON}; posting the report without images."
fi
```

**CRITICAL — NEVER post a bare directory link like `https://github.com/.../tree/...`.** Every screenshot MUST appear as `![name](url)` inline in the PR comment so reviewers can see them without clicking any links. After posting, the verification step below counts the rewritten asset URLs against the number of screenshots and exits 1 on any shortfall — the test run is incomplete until this passes.

**CRITICAL — NEVER paste absolute local paths or credentials into the PR comment.** Strings like `/Users/…`, `/home/…`, `C:\…` are useless to every reviewer except you, and a bearer token or JWT copied from the live stack's evidence is a real leak in a public comment. The build block below runs an egress check on the exact bytes about to be posted and aborts on either. The `./01-shot.png` references the attachments use are relative, not local paths, and are rewritten on upload — they are fine. Keep local paths in `$RESULTS_DIR/test-report.md` for yourself; only copy the *content* they reference (excerpts, test names, log lines) into the PR comment, not the path.

Build the body with **relative image references**, then attach the same paths:

```bash
REPO="Significant-Gravitas/AutoGPT"
MAX_ATTACHMENTS=50   # per gh pr comment invocation

# Fail closed: an empty RESULTS_DIR would glob the PR worktree's PNGs into a public comment.
# gh matches --attach to the body by exact path, so the cd is required; undone at the end.
[ -n "$RESULTS_DIR" ] && [ -d "$RESULTS_DIR" ] || { echo "ERROR: RESULTS_DIR is unset or not a directory: '$RESULTS_DIR'"; exit 1; }
STEP7_ORIG_DIR=$PWD
cd "$RESULTS_DIR" || exit 1
NULLGLOB_WAS=$(shopt -p nullglob)   # restore the caller's setting, don't force it off
shopt -s nullglob
SCREENSHOT_FILES=(*.png)
$NULLGLOB_WAS
if [ ${#SCREENSHOT_FILES[@]} -eq 0 ]; then
  echo "ERROR: No screenshots found in $RESULTS_DIR. Test run is incomplete."
  exit 1
fi
# Over the cap, post the report without images rather than nothing at all.
if [ ${#SCREENSHOT_FILES[@]} -gt "$MAX_ATTACHMENTS" ]; then
  ATTACH_SKIP_REASON="${#SCREENSHOT_FILES[@]} screenshots exceed the ${MAX_ATTACHMENTS}-attachment limit per comment"
  echo "WARN: ${ATTACH_SKIP_REASON}; posting the report without images."
  ATTACH_OK=0
fi

IMAGE_MARKDOWN=""
ATTACH_ARGS=()
for BASENAME in "${SCREENSHOT_FILES[@]}"; do
  TITLE=$(echo "${BASENAME%.png}" | sed 's/^[0-9]*-//' | sed 's/-/ /g' | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) tolower(substr($i,2))}1')
  EXPLANATION="${SCREENSHOT_EXPLANATIONS[$BASENAME]}"
  if [ -z "$EXPLANATION" ]; then
    echo "ERROR: Missing screenshot explanation for $BASENAME. Add it to SCREENSHOT_EXPLANATIONS in Step 6."
    exit 1
  fi
  IMAGE_MARKDOWN="${IMAGE_MARKDOWN}
### ${TITLE}
![${TITLE}](./${BASENAME})
${EXPLANATION}
"
  # Alt text after '#' is the fallback; the body reference above wins when both exist.
  ATTACH_ARGS+=(--attach "./${BASENAME}#${TITLE}")
done

# Keep the report and the image section separate: the no-image fallback below
# must not carry "![](./file.png)" references it has no attachments for.
RUN_MARKER="<!-- pr-test-report:${PR_NUMBER}:$(date -u +%Y%m%dT%H%M%SZ) -->"
REPORT_BODY="${RUN_MARKER}
## E2E Test Report

| # | Scenario | Result | API Evidence | Screenshot Evidence |
|---|----------|--------|-------------|-------------------|
${TEST_RESULTS_TABLE}
"

COMMENT_FILE=$(mktemp)
printf '%s\n%s\n' "$REPORT_BODY" "$IMAGE_MARKDOWN" > "$COMMENT_FILE"

# Egress check on the exact bytes about to be posted. Fails closed: a missing
# body is an abort, not a pass.
LEAK_RE='(^|[^A-Za-z])(/Users/|/home/|/tmp/|/private/|C:\\|~/)[A-Za-z0-9]|eyJ[A-Za-z0-9_-]{20,}\.[A-Za-z0-9_-]{10,}|(sk|ghp|gho|github_pat)_[A-Za-z0-9_]{16,}|[Bb]earer [A-Za-z0-9._-]{20,}'
[ -s "$COMMENT_FILE" ] || { echo "ABORT: comment body missing or empty."; exit 1; }
if grep -nE "$LEAK_RE" "$COMMENT_FILE"; then
  echo "ABORT: local paths or credential-shaped strings in the PR comment body. Rewrite them before posting."
  exit 1
fi

# A blind retry duplicates a post GitHub accepted but gh reported failed; the marker
# makes "did it land?" checkable first. Real jq (gh's --jq has no --arg); --paginate
# because the per-issue endpoint ignores sort/direction.
URL_RE='https://[^[:space:]]+#issuecomment-[0-9]+'
POSTED=""
if [ "$ATTACH_OK" = 1 ]; then
  for attempt in 1 2 3; do
    # The marker was minted seconds ago, so attempt 1 cannot already be posted.
    # A lookup that errors is "unknown", not "not found": posting on unknown is
    # how the duplicate the marker exists to prevent would get made.
    if [ "$attempt" -gt 1 ]; then
      if ! FOUND=$(set -o pipefail; gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" --paginate \
          | jq -r --arg m "$RUN_MARKER" 'first(.[] | select(.body | contains($m)) | .html_url) // empty'); then
        echo "Attempt $attempt: marker lookup failed; not posting until it can be confirmed"
        sleep $((attempt * attempt * 2)); continue
      fi
      POSTED=${FOUND%%$'\n'*}
      [ -n "$POSTED" ] && break
    fi
    # Take the URL by shape, not position: gh prints its update notice on stderr at
    # exit, and a trailing one would otherwise ride into COMMENT_ID below.
    OUT=$(gh pr comment "$PR_NUMBER" --repo "$REPO" --body-file "$COMMENT_FILE" "${ATTACH_ARGS[@]}" 2>&1)
    POSTED=$(printf '%s' "$OUT" | grep -oE "$URL_RE" | tail -1)
    [ -n "$POSTED" ] && break
    echo "Attempt $attempt failed: $OUT"
    [ "$attempt" -lt 3 ] && sleep $((attempt * attempt * 2))   # 2s, 8s — secondary rate limits need room
  done
fi
```

If `gh` is too old, the run is over the cap, or all three attempts failed, post the report **without** images and say so, so the run is visibly incomplete rather than silently missing its evidence. The run continues into Step 8 — an image-free report is not approvable, and that is Step 8's job to say:

```bash
LOOKUP_UNKNOWN=0
if [ -z "$POSTED" ] && [ "$ATTACH_OK" = 1 ]; then
  # One last marker check — a final attempt may have landed despite reporting failure.
  if FOUND=$(set -o pipefail; gh api "repos/${REPO}/issues/${PR_NUMBER}/comments" --paginate \
      | jq -r --arg m "$RUN_MARKER" 'first(.[] | select(.body | contains($m)) | .html_url) // empty'); then
    POSTED=${FOUND%%$'\n'*}
  else
    LOOKUP_UNKNOWN=1
  fi
fi
RUN_INCOMPLETE=0
if [ -z "$POSTED" ] && [ "$LOOKUP_UNKNOWN" = 1 ]; then
  # Can't tell whether the report landed; a fallback post here risks a duplicate.
  RUN_INCOMPLETE=1
  echo "ERROR: could not confirm whether the report was posted (GitHub API errors). Check the PR for marker ${RUN_MARKER} before re-running Step 7."
elif [ -z "$POSTED" ]; then
  RUN_INCOMPLETE=1
  [ "$ATTACH_OK" = 1 ] && ATTACH_SKIP_REASON="the upload failed 3 times"
  FALLBACK_FILE=$(mktemp)
  {
    printf '%s\n' "$REPORT_BODY"
    printf '## ⚠️ Screenshots not attached\n\n'
    printf '%s. Filenames, for manual drag-and-drop into a reply:\n\n' "$ATTACH_SKIP_REASON"
    printf -- '- `%s`\n' "${SCREENSHOT_FILES[@]}"
    printf '\n**Run status:** INCOMPLETE until the files are attached and visible inline in the PR.\n'
  } > "$FALLBACK_FILE"
  # The degraded path gets the same egress check as the full body.
  if grep -nE "$LEAK_RE" "$FALLBACK_FILE"; then
    echo "ABORT: local paths or credential-shaped strings in the fallback body."
    exit 1
  fi
  OUT=$(gh pr comment "$PR_NUMBER" --repo "$REPO" --body-file "$FALLBACK_FILE" 2>&1)
  POSTED=$(printf '%s' "$OUT" | grep -oE "$URL_RE" | tail -1)
  rm -f "$FALLBACK_FILE"
  echo "Posted without images (RUN_INCOMPLETE=1): ${POSTED:-FAILED — $OUT}"
fi
rm -f "$COMMENT_FILE"
cd "$STEP7_ORIG_DIR"
```

Verify the rendered comment actually carries every screenshot as an uploaded asset (skipped for an image-free fallback, which Step 8 already treats as not approvable):

```bash
if [ "$RUN_INCOMPLETE" = 0 ]; then
  COMMENT_ID="${POSTED##*issuecomment-}"
  BODY=$(gh api "repos/${REPO}/issues/comments/${COMMENT_ID}" --jq '.body')
  EXPECTED=${#SCREENSHOT_FILES[@]}
  # Count, don't just detect: a partial rewrite leaves broken "](./x.png)"
  # links behind while the body still contains *some* asset URLs.
  UPLOADED=$(printf '%s' "$BODY" | grep -o 'github.com/user-attachments/' | wc -l | tr -d ' ')
  LEFTOVER=$(printf '%s' "$BODY" | grep -o '](\./' | wc -l | tr -d ' ')
  RAW_IMGS=$(printf '%s' "$BODY" | grep -oE '!\[[^]]*\]\(https://raw\.githubusercontent\.com' | wc -l | tr -d ' ')
  if [ "$RAW_IMGS" -gt 0 ]; then
    echo "ERROR: $RAW_IMGS image(s) still point at raw repo URLs. Screenshots must be attachments, not files in the repo." >&2
    exit 1
  fi
  if [ "$LEFTOVER" -gt 0 ] || [ "$UPLOADED" -ne "$EXPECTED" ]; then
    echo "ERROR: expected $EXPECTED uploaded attachments, found $UPLOADED, with $LEFTOVER unrewritten ./ reference(s) — partial upload." >&2
    exit 1
  fi
  echo "✓ $EXPECTED screenshots verified as GitHub attachments"
fi
```

**The PR comment MUST include:**
1. A summary table of all scenarios with PASS/FAIL and before/after API evidence
2. Every screenshot rendered inline as an uploaded attachment; any failure listed with manual attachment instructions and the run marked INCOMPLETE
3. A 1-2 sentence explanation below each screenshot describing what it proves

## Step 8: Evaluate and post a formal PR review

After the test comment is posted, evaluate whether the run was thorough enough to make a merge decision, then post a formal GitHub review (approve or request changes). **This step is mandatory — every test run MUST end with a formal review decision.**

### Evaluation criteria

Re-read the PR description:
```bash
gh pr view "$PR_NUMBER" --json body --jq '.body' --repo "$REPO"
```

Score the run against each criterion:

| Criterion | Pass condition |
|-----------|---------------|
| **Coverage** | Every feature/change described in the PR has at least one test scenario |
| **All scenarios pass** | No FAIL rows in the results table |
| **Negative tests** | At least one failure-path test per feature (invalid input, unauthorized, edge case) |
| **Before/after evidence** | Every state-changing API call has before/after values logged |
| **Screenshots are meaningful** | Screenshots show the actual state change, not just a loading spinner or blank page |
| **No regressions** | Existing core flows (login, agent create/run) still work |

### Decision logic

```
ALL criteria pass                            → APPROVE
Any scenario FAIL or missing PR feature      → REQUEST_CHANGES (list gaps)
Evidence weak (no before/after, vague shots) → REQUEST_CHANGES (list what's missing)
Screenshots not attached (RUN_INCOMPLETE=1)  → REQUEST_CHANGES (the evidence is not on the PR)
```

### Post the review

```bash
REVIEW_FILE=$(mktemp)

# Count results
PASS_COUNT=$(echo "$TEST_RESULTS_TABLE" | grep -c "PASS" || true)
FAIL_COUNT=$(echo "$TEST_RESULTS_TABLE" | grep -c "FAIL" || true)
TOTAL=$(( PASS_COUNT + FAIL_COUNT ))

# List any coverage gaps found during evaluation (populate this array as you assess)
# e.g. COVERAGE_GAPS=("PR claims to add X but no test covers it")
COVERAGE_GAPS=()
# Step 7 posts an image-free report instead of aborting; that run is not approvable.
[ "${RUN_INCOMPLETE:-0}" = 1 ] && COVERAGE_GAPS+=("Screenshots were not attached to the test report — attach them and re-run verification")
```

**If APPROVING** — all criteria met, zero failures, full coverage:

```bash
cat > "$REVIEW_FILE" <<REVIEWEOF
## E2E Test Evaluation — APPROVED

**Results:** ${PASS_COUNT}/${TOTAL} scenarios passed.

**Coverage:** All features described in the PR were exercised.

**Evidence:** Before/after API values logged for all state-changing operations; screenshots show meaningful state transitions.

**Negative tests:** Failure paths tested for each feature.

No regressions observed on core flows.
REVIEWEOF

gh pr review "$PR_NUMBER" --repo "$REPO" --approve --body "$(cat "$REVIEW_FILE")"
echo "✅ PR approved"
```

**If REQUESTING CHANGES** — any failure, coverage gap, or missing evidence:

```bash
FAIL_LIST=$(echo "$TEST_RESULTS_TABLE" | grep "FAIL" | awk -F'|' '{print "- Scenario" $2 "failed"}' || true)

cat > "$REVIEW_FILE" <<REVIEWEOF
## E2E Test Evaluation — Changes Requested

**Results:** ${PASS_COUNT}/${TOTAL} scenarios passed, ${FAIL_COUNT} failed.

### Required before merge

${FAIL_LIST}
$(for gap in "${COVERAGE_GAPS[@]}"; do echo "- $gap"; done)

Please fix the above and re-run the E2E tests.
REVIEWEOF

gh pr review "$PR_NUMBER" --repo "$REPO" --request-changes --body "$(cat "$REVIEW_FILE")"
echo "❌ Changes requested"
```

```bash
rm -f "$REVIEW_FILE"
```

**Rules:**
- In `--fix` mode, fix all failures before posting the review — the review reflects the final state after fixes
- Never approve if any scenario failed, even if it seems like a flake — rerun that scenario first
- Never approve a run with `RUN_INCOMPLETE=1` — the evidence is not on the PR
- Never request changes for issues already fixed in this run

## Fix mode (--fix flag)

When `--fix` is present, the standard is HIGHER. Do not just note issues — FIX them immediately.

### Fix protocol for EVERY issue found (including UX issues):

1. **Identify** the root cause in the code — read the relevant source files
2. **Write a failing test first** (TDD): For backend bugs, write a test marked with `pytest.mark.xfail(reason="...")`. For frontend/Playwright bugs, write a test with `.fixme` annotation. Run it to confirm it fails as expected.
3. **Screenshot** the broken state: `agent-browser screenshot $RESULTS_DIR/{NN}-broken-{description}.png`
4. **Fix** the code in the worktree
5. **Rebuild** ONLY the affected service (not the whole stack):
   ```bash
   cd $PLATFORM_DIR && docker compose up --build -d {service_name}
   # e.g., docker compose up --build -d rest_server
   # e.g., docker compose up --build -d frontend
   ```
6. **Wait** for the service to be ready (poll health endpoint)
7. **Re-test** the same scenario
8. **Screenshot** the fixed state: `agent-browser screenshot $RESULTS_DIR/{NN}-fixed-{description}.png`
9. **Remove the xfail/fixme marker** from the test written in step 2, and verify it passes
10. **Verify** the fix did not break other scenarios (run a quick smoke test)
11. **Commit and push** immediately:
   ```bash
   cd $WORKTREE_PATH
   git add -A
   git commit -m "fix: {description of fix}"
   git push
   ```
12. **Continue** to the next test scenario

### Fix loop (like pr-address)

```text
test scenario → find issue (bug OR UX problem) → screenshot broken state
→ fix code → rebuild affected service only → re-test → screenshot fixed state
→ verify no regressions → commit + push
→ repeat for next scenario
→ after ALL scenarios pass, run full re-test to verify everything together
```

**Key differences from non-fix mode:**
- UX issues count as bugs — fix them (bad alignment, confusing labels, missing loading states)
- Every fix MUST have a before/after screenshot pair proving it works
- Commit after EACH fix, not in a batch at the end
- The final re-test must produce a clean set of all-passing screenshots

## Known issues and workarounds

### Problem: Signup/sign-in fails with no useful error
**Cause:** Better Auth's default minimum password length is 12 characters, or
`AUTH_ALLOW_NEW_ACCOUNTS=false` / `AUTH_SIGNUP_ALLOWLIST` is blocking new
accounts, or you're rate-limited after repeated signup attempts (`429 Too many
requests`) — all three degrade into an empty `$TOKEN` further down if you
don't check the raw response (see 3h).
**Fix:** Log the raw signup/sign-in response body before minting the token,
not just the token itself.

### Problem: Copilot returns auth errors in subscription mode
**Cause:** `CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true` but `CLAUDE_CODE_OAUTH_TOKEN` is not set or expired.
**Fix:** Re-extract the OAuth token from macOS keychain (see step 3b, Option 1) and recreate the container (`docker compose up -d copilot_executor`). The backend auto-provisions `~/.claude/.credentials.json` from the env var on startup. No `npm install` or `claude login` needed — the SDK bundles its own CLI binary.

### Problem: agent-browser can't find chromium
**Cause:** The Dockerfile auto-provisions system chromium on all architectures (including ARM64). If your branch is behind `dev`, this may not be present yet.
**Fix:** Check if chromium exists: `which chromium || which chromium-browser`. If missing, install it: `apt-get install -y chromium` and set `AGENT_BROWSER_EXECUTABLE_PATH=/usr/bin/chromium` in the container environment.

### Problem: agent-browser selector matches multiple elements
**Cause:** `text=X` matches all elements containing that text.
**Fix:** Use `agent-browser snapshot` to get specific `ref=eNN` references, then use those: `agent-browser click eNN`.

### Problem: Frontend shows cookie banner blocking interaction
**Fix:** `agent-browser click 'text=Accept All'` before other interactions.

### Problem: Claude CLI not found in copilot_executor container
**Symptom:** Copilot logs say `claude: command not found` or similar when starting an SDK turn.
**Cause:** Image was built without `poetry install` (stale base layer, or Dockerfile bypass). The SDK CLI ships inside the `claude_agent_sdk` Poetry dep — it is NOT an npm package.
**Fix:** Rebuild the image cleanly: `docker compose build --no-cache copilot_executor && docker compose up -d copilot_executor`. Do NOT `docker exec ... npm install -g @anthropic-ai/claude-code` — that is outdated guidance and will pollute the container with a second CLI that the SDK won't use.

### Problem: agent-browser screenshot hangs / times out
**Symptom:** `agent-browser screenshot` exits with code 124 even on `about:blank`.
**Cause:** Stuck CDP connection or Chromium process tree. Seen on macOS when a prior `/pr-test` left a zombie Chrome for Testing.
**Fix:** `pkill -9 -f "agent-browser|chromium|Chrome for Testing" && sleep 2`, then reopen the browser with a fresh `--session-name`. If still failing, verify via `agent-browser eval` + `agent-browser snapshot` (DOM state) instead of relying on PNGs — the feature under test is the same.

### Problem: Services not starting after `docker compose up`
**Fix:** Wait and check health: `docker compose ps`. Common cause: migration hasn't finished. Check: `docker logs autogpt_platform-migrate-1 2>&1 | tail -5`. If the `db` container isn't healthy: `docker restart autogpt_platform-db-1 && sleep 10`.

### Problem: Docker uses cached layers with old code (PR changes not visible)
**Cause:** `docker compose up --build` reuses cached `COPY` layers from previous builds. If the PR branch changes Python files but the previous build already cached that layer from `dev`, the container runs `dev` code.
**Fix:** Always use `docker compose build --no-cache` for the first build of a PR branch. Subsequent rebuilds within the same branch can use `--build`.

### Problem: `agent-browser open` loses login session
**Cause:** Without session persistence, `agent-browser open` starts fresh.
**Fix:** Use `--session-name pr-test` on ALL agent-browser commands. This auto-saves/restores cookies and localStorage across navigations. Alternatively, use `agent-browser eval "window.location.href = '...'"` to navigate within the same context.