exam

Run the heavy local exam lane (T4) on the gold-standard macOS host: preflight → caffeinate keep-awake → a single RUN_FLAKY=1 pnpm exam pass (WebKit @interactive + @macos-only on Darwin + @flaky quaran...

/exam — Heavy Local Lane (T4)

Project-scope heavy-lane exam skill. Runs the platform-gated WebKit @interactive suite (the specs that need a real macOS WebKit engine and so cannot run on PR CI), triages failures with agent judgment, files deduped exam-failure tracking issues, and optionally fixes up to N in-session. Leave it running before sleep; read the morning summary when you return.

This is the local convenience lane (T4), not the enforcement layer. The scheduled T3 CI re-exam (.github/workflows/e2e-webkit-macos.yml, twice-weekly Mon/Thu cron + workflow_dispatch) is the paper-trail backstop that runs even when nobody remembered to run this. The two are a pair: /exam is the rich lane with agent triage and in-session fixes; the scheduled CI job is the thin net. See $HOME/.claude/skills/test-wisdom/docs/real-world-patterns/scheduled-re-exam.mdx for the design rationale and $HOME/.claude/skills/test-wisdom/docs/decision-guide/execution-tiers.mdx for the T0–T4 tier model.

Argument Parsing

Parse $ARGUMENTS for:

• --fix [N] — After the exam run, pick up to N open exam-failure issues (default N=3) and attempt in-session fixes. Each fix is verified with a scoped WebKit rerun. Without --fix, the skill stops after filing issues.

• --skip-preflight — Bypass the Step 1 preflight checks. Use only when explicitly requested (e.g. you know the tree is intentionally dirty during a repair session).

Step 1 — Preflight

Skip this step entirely if --skip-preflight was passed.

The night-exam pattern wants a clean, current main so the results are attributable. Run these checks; hard-block where noted, warn-but-continue otherwise.

1a. Clean working tree (hard block)

git status --porcelain

If the output is non-empty, stop:

Working tree is dirty. Commit or stash before running /exam. Dirty files: <list>

1b. On main (warn, do not block)

git branch --show-current

main is the intended branch for a night exam. If the current branch is something else (a topic branch, a worktree sub-branch), warn but continue — running the exam on a branch to validate a change pre-merge is legitimate:

Warning: you are on <name>, not main. Exam results reflect this branch's state, not main. Continuing anyway.

1c. Up to date with remote (hard block if behind)

git fetch --quiet
git status -sb

If the status line contains behind, stop (so the exam does not test stale code):

Local branch is behind the remote. Pull before running /exam. Run: git pull --ff-only

Step 2 — Keep-Awake Wrapper + Single Exam Pass

Run the entire suite in one pnpm exam invocation with RUN_FLAKY=1 in the env — not two passes. scripts/run-exam.sh runs the WebKit @interactive lane once, then (because RUN_FLAKY=1) the @flaky quarantine sweep in both projects. Invoking pnpm exam twice (once plain, once with RUN_FLAKY=1) re-runs the entire serial @interactive lane a second time for no reason — do not do that.

Wrap it in caffeinate -i on macOS so the machine does not sleep mid-suite:

# macOS: caffeinate -i holds the sleep assertion for the child's lifetime.
# On Linux/WSL caffeinate is absent and run-exam.sh requires FORCE_EXAM=1 anyway;
# the WebKit @interactive verdict is only trustworthy on real macOS regardless.
if [[ "$(uname)" == "Darwin" ]]; then
  caffeinate -i env RUN_FLAKY=1 pnpm exam
else
  RUN_FLAKY=1 pnpm exam
fi

Capture the full combined output (stdout + stderr) to a variable AND stream it live to the terminal. Record the exit code as EXAM_EXIT_CODE.

scripts/run-exam.sh is platform-gated: on a non-Darwin host without FORCE_EXAM=1 it refuses to run (the WebKit lane false-reds off real macOS — see #2604/#2649). A clean non-Darwin refusal is not a failure to triage; report it and stop.

Step 3 — Parse Lane Output

scripts/run-exam.sh emits structured status lines the skill consumes:

  [lane status] RAN: <lane-name>
  [lane status] SKIPPED: <lane-name> — <reason>

plus per-step ✅/❌ markers and a final ✅ All checks passed! / ❌ N check(s) failed: summary. Collect:

• LANES_RAN — lane names from RAN: lines (e.g. E2E tests (webkit @interactive), chromium @flaky quarantine (non-blocking), webkit @flaky quarantine (non-blocking))

• LANES_SKIPPED — { lane, reason } from SKIPPED: lines (e.g. the @macos-only/@webkit-only specs exclusion on non-Darwin, or the @flaky sweep when RUN_FLAKY was not set)

• EXAM_EXIT_CODE — from Step 2

Skipped lanes are not failures. Report them in the morning summary; never file an issue for a skip. The @flaky quarantine lanes are non-blocking — their failures do not drive EXAM_EXIT_CODE and are triaged as quarantine signal, not regressions.

Step 4 — Triage Failures (two passes)

1. Noise scan — always, regardless of EXAM_EXIT_CODE. Scan the captured output for the env-noise signatures in 4a. Noise can co-exist with a zero exit; if any noise matched, the summary Result is NOISE-ONLY (or FAIL if real regressions also exist) — never a plain PASS.

2. Failure clustering — only when EXAM_EXIT_CODE is non-zero. Parse the output for failure signatures and cluster them per 4b, excluding anything already matched as noise.

4a. Env-noise signatures (never file as exam-failure)

These are known non-regression conditions. Do NOT file issues. Surface them in the morning summary as "env noise" with the recommended action.

If a failure matches a noise signature, list it separately in the summary and skip issue filing for that cluster.

4b. Real regression failures

A failure is a real regression if it matches NO noise signature above AND:

• it is in a blocking @interactive or @macos-only spec (drove EXAM_EXIT_CODE), AND

• no open exam-failure-labeled issue already covers its cluster.

Group related failures into one cluster when they share an obvious root cause (same component, same code path, same error message). One cluster → one issue — not one per spec, not one per run.

Step 5 — Deduped Issue Filing

For each real-regression cluster:

5a. Check for an existing open issue (dedupe)

gh issue list --label "exam-failure" --state open --json number,title \
  --jq '.[] | "\(.number) \(.title)"'

If an open issue already covers this cluster (fuzzy: title shares ≥3 key tokens, or the same failing spec file), comment on it instead of opening a duplicate:

EXAM_AT=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
EXAM_HEAD=$(git rev-parse --short HEAD)
gh issue comment <existing-number> --body "$(cat <<EOF
Still failing: ${EXAM_AT} — HEAD ${EXAM_HEAD}

<same failing specs + first error detail>
EOF
)"

5b. File a new issue if no match found

Ensure the label exists once before the first create (idempotent):

gh label create "exam-failure" --color "e11d48" \
  --description "Heavy-lane exam run failed; tracks the open regression" 2>/dev/null || true

Compute metadata BEFORE the heredoc. The heredoc is unquoted (<<EOF) so ${...} expands — therefore every literal backtick inside the body must be escaped (\`) to avoid command substitution.

EXAM_AT=$(date -u '+%Y-%m-%dT%H:%M:%SZ')
EXAM_HEAD=$(git rev-parse --short HEAD)
EXAM_BRANCH=$(git branch --show-current)
SCHEDULED_XREF=$(gh run list --workflow=e2e-webkit-macos.yml --limit 3 \
  --json createdAt,status,conclusion \
  --jq '.[] | "\(.createdAt) \(.status)/\(.conclusion)"')

gh issue create \
  --label "exam-failure" \
  --title "exam: <short symptom>" \
  --body "$(cat <<EOF
## Exam failure — <cluster title>

**Detected:** ${EXAM_AT} — HEAD ${EXAM_HEAD} — branch ${EXAM_BRANCH}
**Lanes that ran:** <LANES_RAN list>

### Failing specs
<spec file list>

### Error detail
\`\`\`
<first error message / timeout detail>
\`\`\`

### Noise check
- ESM-loader noise (#2174): <yes/no — evidence>
- Keyboard-delivery noise (#2238/#2649): <yes/no — evidence>
- @flaky quarantine: <yes/no>

### Scheduled exam cross-reference (e2e-webkit-macos.yml)
${SCHEDULED_XREF}

---
*Filed automatically by \`/exam\`. Do not close manually — use \`gh issue close\` after a confirmed fix.*
EOF
)"

Collect all filed/updated issue URLs into FILED_ISSUES. The dedupe contract here mirrors scripts/file-exam-issue.sh (the CI-side filer the T3 workflow uses): at most one open exam-failure issue per failure cluster.

Step 6 — Fix Mode (--fix)

Skip this step unless --fix was passed.

6a. Collect open exam-failure issues

gh issue list --label "exam-failure" --state open --json number,title,url,body --limit <N>

Pick up to N (default 3 unless the user gave a different N). Prefer lowest-numbered / freshest.

6b. For each issue

1. Read the issue body to identify the failing spec(s).

2. Create a fix branch: git checkout -b fix/<short-slug>.

3. Investigate root cause (read the relevant source), then apply a minimal surgical fix — one bug, one commit.

4. Reproduce + verify with the scoped WebKit runner (NOT a full pnpm exam rerun, and NOT vitest):

   pnpm test:e2e --project=webkit --workers=1 --grep="<spec name or describe>"

5. If the scoped run passes: commit (git commit -m "fix(<scope>): <what and why>"), then close the issue with a comment linking the commit/branch.

6. Do NOT push (worktree push policy — only the manager session pushes). Leave the branch for review; report it in the morning summary.

Time-box each issue to ~20 min (branch → passing scoped run). If it overruns, abandon it: git checkout main, delete the branch, leave a diagnostic comment on the issue ("attempted but deferred"), and move to the next candidate. Do not let one hard issue consume the session. Do not attempt more than N issues — focus over breadth.

Step 7 — Morning Summary

Always print a structured summary as the final output:

=== /exam summary — <date> ===

HEAD:    <short SHA> — <git log -1 --pretty=%s>
Branch:  <branch>
Result:  PASS | FAIL | NOISE-ONLY

── Lanes ─────────────────────────────
RAN:
  - E2E tests (webkit @interactive)
  - webkit @flaky quarantine (non-blocking)
  - ... (only lanes that produced a RAN line)
SKIPPED:
  - <lane> — <reason>
  - ... (only skipped lanes; or "none")

── Env noise (not filed) ─────────────
  - [ESM-loader #2174 | keyboard-delivery #2238/#2649 | @flaky] <spec> — <detail>
    → <recommended action>
  - ... (or "none")

── Real regressions (<N>) ────────────
  - <cluster title> — <one-line error> → #<issue> (new | updated)
  - ... (or "none — all blocking lanes passed")

── Fix mode (--fix only) ─────────────
  Fixed:   #<n> <title> → branch fix/<slug> (commit <sha>)
  Deferred:#<n> <title> — <reason>
  (omit this section if --fix was not passed)

── Backstop ──────────────────────────
  T3 scheduled CI: e2e-webkit-macos.yml (Mon/Thu cron + workflow_dispatch).
  Dispatch on a branch before merging risky @interactive/@macos-only changes:
    gh workflow run e2e-webkit-macos.yml --ref <branch>

If nothing failed and no noise matched, end with:

All exam lanes passed. No issues filed.

Notes

• pnpm exam = scripts/run-exam.sh — it applies the Darwin platform gate and the per-lane grep logic. Run the suite through it, not by hand-assembling Playwright commands; the script handles the --workers=1 serial WebKit pass (#2174) and the @macos-only inclusion correctly.

• Single pass, not two. Set RUN_FLAKY=1 in the env of ONE pnpm exam call (Step 2). The old two-call pattern re-ran the entire serial @interactive lane twice.

• Lane skip ≠ failure. Non-Darwin platform-spec exclusion and the unset-RUN_FLAKY skip are expected; they appear in the summary for visibility but are never filed.

• @flaky is a quarantine pipeline, not a lane to fix here. Its failures are non-blocking and tracked by the inline issue URL on each tag — see e2e/README.md § "Test Placement Triage".

• Issue label: exam-failure (create on first use; see Step 5b).

• --fix is best-effort. In-session fixes are opportunistic; if a fix is uncertain, leave a diagnostic and move on.

Reference

• Heavy lane: scripts/run-exam.sh (pnpm exam)

• CI-side deduped issue filer (T3): scripts/file-exam-issue.sh

• Scheduled CI exam (T3 backstop): .github/workflows/e2e-webkit-macos.yml

• Tag taxonomy, tier model, b4push/exam contract, module-to-spec map: e2e/README.md

• Known noise: e2e/README.md § "Local e2e lanes", #2174, #2238, #2649

• Pattern rationale: $HOME/.claude/skills/test-wisdom/docs/real-world-patterns/scheduled-re-exam.mdx

• Tier model (T0–T4): $HOME/.claude/skills/test-wisdom/docs/decision-guide/execution-tiers.mdx

• Sibling skills: /l-broad-walk-test (open-ended bug sweep), /l-agent-walk-test (specific bug reproduction)