CI Verification Policy

This policy separates how often checks run, how development changes merge, and what evidence is required before a change affects users. Merging into main is allowed to detect and repair a batch-level regression; deploying a real service still requires verification of the exact candidate.

Historical note: the former CI workflow put e2e_shards, the required e2e aggregate, and styleguide browser smoke on every PR, called that setup the authoritative gate, and described b4push as a mirror of the T1 PR gate. Those phrases describe the pre-v1 topology and are superseded by this policy.

Three separate decisions

The merge-first tradeoff is delayed detection: intermediate development commits may merge before the full suite runs. Failures stay visible, owned, and bounded by a repair checkpoint. This does not permit ignoring failures, weakening regression coverage, or deploying unverified code.

Work per change and per batch

The lightweight CI job named checks always reports. Its scope selector treats unknown paths and shared inputs conservatively. It removes only _temp-resource/** paths before classification; a change consisting only of those files runs the integrity guards and no product checks.

The doc and manual content steps are selected by their own scope flags. The accepted code suite is not affected-package-only: workspace checks keep their existing broad coverage. The package help-catalog reader and MCP documentation smoke are accounted for by the scope rules. Browser E2E shards and the styleguide browser smoke are not part of the lightweight lane.

Full Verification runs the retained suite as one batch: full checks, styleguide browser smoke, both CI-safe Chromium E2E shards with a strict identity aggregate, sync-server tests, tauri-app tests, and a machine-readable summary. The identity check proves that the shard collections are disjoint and cover the unsharded expected set; only the checked-in expected-skip identities may be skipped, and any other skipped identity fails the batch. The scheduled opportunity is daily at 02:17 JST (17:17 UTC). Every batch pins one immutable SHA at its start, and jobs test that candidate rather than a moving branch tip. Full production runs queue per ref; later main pushes do not cancel the active batch. The pull-request run is limited to changes in verification machinery so the new lane itself is proved in CI.

If a scheduled run sees that main is already at the SHA of the latest successful full run, it records skipped_unchanged and does no work. This is the only no-work result in v1: it does not reuse older evidence, and it never suppresses platform or live-drift lanes.

When a full checkpoint is required

Run Full Verification before promoting a risky change to an external service, after repairing a red full run, after changing shared verification machinery, and before a release that relies on this full-suite contract. A risky development change can use an explicit full dispatch after it has reached main; the promotion gate requires its candidate to be an ancestor of main. Use diagnostic mode to investigate one area, never to satisfy one of these checkpoints.

To verify a specific main candidate, include its SHA:

gh workflow run full-verify.yml --ref main -f mode=full -f candidate_sha=SHA

The scheduled repair checkpoint is the next daily run. At the start of the next active session, check the latest scheduled runs:

gh run list --workflow full-verify.yml --limit 3

If no scheduled run completed in the last 30 hours, dispatch a full run on main and record it in the failure triage:

gh workflow run full-verify.yml --ref main -f mode=full

State definitions

• Merged means the change is incorporated into main. It may still be awaiting its full batch verification; merge does not imply verified or released.

• Verified means an authenticated Full Verification summary names the candidate SHA, records mode full, and marks every fixed required job successful and sets eligible_for_promotion true. Diagnostic results, lightweight-only results, skipped jobs, a different SHA, and missing or ambiguous evidence are not verified candidates for production promotion.

• Deferred means a required check has intentionally not run yet or is unavailable, with the missing evidence and next checkpoint recorded. A deferred check is never described as passed.

• Released means a candidate has crossed its stated external boundary. For the five gated services, the gate has deployed the verified candidate and recorded the post-deploy smoke result; a failed smoke leaves a repair alert and is not a clean release. Static sites and the package release follow their v1 exception paths below.

Red ownership and repair

Every scheduled or dispatched full-mode main result is reported, whether or not it can be promoted. Reporting is separate from promotion eligibility. Pull-request, diagnostic, and skipped_unchanged summaries do not file or close the full-verification alert. Failure and cancellation create or update the deduplicated full-verify-failure record. It contains the tested SHA, run URL and mode, failing jobs and E2E identities, retry-pass identities and attempt counts, artifacts to inspect, intervening non-success runs since the last clean green, commits since the last verified SHA with non-bot authors, the named repair owner, the next checkpoint in UTC and JST, and the continue-or-pause decision. The default repair owner is @Takazudo unless the triage assigns another owner.

Triage the record at the next active session. Repair or revert the cause by the next checkpoint and always before external impact. Continue independent work only when the failure is localized and understood, is unrelated to the next change, and remains tracked within that checkpoint. Pause the affected work chain for an unknown or cross-cutting failure, a broken build/runtime foundation, security or data-loss risk, or an expired repair checkpoint.

A cancelled run, partial green, diagnostic pass, pass-on-retry, or later unrelated green never closes an alert. Only a clean green closes it: the full required job set succeeds on the first workflow attempt, with no Playwright flaky or retry-pass identities and no retried job attempts. A later green with any retry is useful evidence but leaves the red record open.

Before rerunning a red Full Verification, preserve its reports, logs, and traces. Operators may rerun it at most twice before stopping to diagnose the failure. A pass on a rerun is a flake signal, not a repair and not a reason to close the alert.

Promotion and rollback

The deploy workflows for the web editor, sync-server, publish-server, agent-server, and landing site run automatically through workflow_run from Full Verification on main or by explicit dispatch. The automatic path names the triggering verification run. Manual dispatch requires both verification_run_id and candidate_sha. In either case, the deploy checks out the tested candidate SHA.

The shared gate reads the summary artifact for that exact run and authenticates it against GitHub run metadata: the repository and workflow file must match, the event must be schedule or workflow_dispatch, the head branch must be main, the run must be completed and successful, and the summary attempt must match the latest run attempt. It rejects a missing or invalid summary, any mode other than full, a summary SHA that differs from candidate_sha, a missing or unsuccessful required job, or a candidate that is not an ancestor of main. Missing, stale, ambiguous, mismatched, partial, diagnostic, or unsuccessful evidence cannot authorize a manual deployment. In the automatic path, a valid diagnostic or skipped_unchanged run is a clean no-op; it does not deploy.

The gate records the newest applied promotion and its smoke result. It skips a candidate when no relevant input changed after a valid record, refuses automatic regression to an older ancestor, and fails closed if the promotion record exists but its smoke evidence is ambiguous. The force input bypasses only change detection and the automatic ancestor guard; it never bypasses evidence validation, successful full verification, or candidate ancestry of main.

The dispatch-only sync-server test workflow mirrors the deploy lane's unwrapped Tests step for diagnostics; its result is never promotion evidence. It becomes dispatchable only after .github/workflows/sync-server-tests-only.yml exists on main. Until then, #7463 can use a full-verify.yml diagnostic dispatch with jobs=sync-server as its interim evidence path.

Rollback means explicitly dispatching an older, still-verified candidate with force enabled, using its run ID and exact candidate SHA. This is not a database rollback. Sync-server and publish-server deployments can apply D1 migrations; deploying older code does not reverse migrations already applied. Check schema compatibility and plan a forward repair before dispatching such a rollback.

For a diagnostic dispatch, for example:

gh workflow run full-verify.yml --ref main -f mode=diagnostic -f jobs=e2e -f e2e_grep=…

Diagnostic runs may select a job or grep, but they never satisfy the full or release contract. Use mode full for a checkpoint.

To force a rollback to a previously verified candidate, dispatch the selected deploy workflow with that verification run ID and SHA, plus force=true. For example:

gh workflow run deploy-web-editor.yml --ref main -f verification_run_id=RUN_ID -f candidate_sha=SHA -f force=true

v1 exceptions and untouched lanes

• Static documentation and manual sites remain build-verified and ungated.

• Preview Deploy remains a mock-backend preview and is not production evidence.

• MCP npm release remains tag-triggered with its ancestor-of-main check and package-local verification. Requiring a verified candidate is a named follow-up, not a v1 behavior.

Opening a PR without preview deploys

preview-deploy.yml, deploy-doc.yml and deploy-manual.yml skip their deployment job only when the pull_request payload already carries the no-preview-deploy label. gh pr create --label applies the label after the opened event has fired, so that first run still deploys. To make the label effective from creation for a PR into main, open the PR against a throwaway anchor branch that points at origin/main (the Pages workflows do not run for that base), verify the label, retarget the PR to main with gh pr edit --base (the edited event triggers nothing), then close and reopen it. The reopened payload carries the label, so ci.yml runs on the final head against main while every Pages deployment job is skipped. Confirm the deployment jobs show as skipped, then delete the anchor branch.

These paths are explicit v1 exceptions. The following independent lanes keep their existing purpose and cadence. skipped_unchanged never suppresses them, and live-drift checks are never skipped because source files are unchanged.

Returning to stricter verification

To restore the full suite on every pull request without changing deployment gating, remove the paths filter from the pull_request trigger in full-verify.yml. This is the one-line execution-cadence rollback. Reverting the whole merge-first change would also restore automatic push-to-main production deploy coupling; that broader revert is not the recommended way to restore verification frequency.

Revisit the policy when

Review this policy at the first public release, when real-user exposure grows, after repeatedly missed repair checkpoints, after a false-green selection, or when unresolved verification failures rise. A new expensive suite needs an explicit purpose, tier and frequency, affected inputs, and budget review before it is added to either lane.

Baseline

S6 measured the pre-change window from 2026-09-23 00:00 UTC through 2026-10-07 00:00 UTC (half-open). The GitHub Actions API returned 598 workflow run IDs across seven workflows and 2,087 job executions. Of those, 2,076 had valid timing intervals; 11 negative intervals were excluded from minute totals. Durations are job completion minus job start, excluding queue time. Runner-weighted minutes are a normalization (2-vCPU = 1, 4-vCPU = 2, 8-vCPU = 4, 6-vCPU macOS = 3), not a billing estimate.

The report was generated on 2026-10-06 from paginated GitHub Actions run and job responses, using all job attempts. See the S6 report for the per-job breakdown and API provenance.

S6 also preserved sample run 37429169621: its two shards covered all 583 expected identities in a disjoint 292/291 split, with identity-set SHA-256 b0c902c6b8705019138a7b4c55a3853625d954dc907448cc4b18423d2b27aef3. The run had 577 passes, six allowlisted skips, and no unexpected or flaky results. The preserved shard evidence lists the six skipped identities and the two artifact links; those artifacts expire on 2026-10-20.

The first S10 decision was NO-GO, and S11 initially kept ci.yml on its full lane. On hosted PR head 914eb4427215f0ad3dff5cc8e54f9e961782ea94, legacy CI run 37541235964 passed on merge candidate 2bb02c20a66c38dfed7dd8136d5980a1c6a9f2fc, while Full Verification run 37541235953 had three recorded attempts on that candidate. Attempt 1 failed with seven unexpected failures and four flaky identities; its summary artifact records reporting=false and eligible_for_promotion=false for the PR event. Attempt 2 used gh run rerun --failed: shard 2 passed with zero unexpected results and two flaky identities, but the e2e aggregate failed because the rerun had no attempt-2 shard-1 report (cp: cannot stat); the summary job also failed. The attempt-2 summary artifact records result=failure, e2e=null, reporting=false, and eligible_for_promotion=false. Attempt 3 reran the full workflow; all required jobs passed. Its summary artifact records result=success, 583 expected identities, 577 executed and passed, six allowlisted skips, and zero unexpected skips, unexpected failures, or flaky/retried identities. Because this was a PR event, it still records reporting=false and eligible_for_promotion=false; the successful retry does not erase attempts 1 and 2 or change the formal S10 decision. That NO-GO stood until a later green full run on main re-ran the S10 proof.

On 2026-10-27, #7368 recorded DECISION: GO from Full Verification run 40471356100, green on main candidate bbd8fa4f3: 679 expected identities, 673 passed, six allowlisted skips, and zero retries. The lightweight-lane cutover followed.

This report is a pre-change baseline only. Post-cutover savings and detection cadence remain unmeasured until a post-cutover window of the same length has elapsed.

Lightweight-lane cutover

The cutover commit rewrote ci.yml to the lightweight lane: a scope job runs scripts/ci/ci-scope.mjs against the pull-request merge base or the push range; the always-reporting checks job fails first unless scope succeeded and then runs the shared checks with the selected code, doc, and manual flags; sync-server and tauri-app run only when their scope flags are true. The styleguide, e2e_shards, and e2e jobs were removed from ci.yml and remain in Full Verification. pnpm check:ci-lanes asserts these ci.yml invariants unconditionally, together with the Full Verification and deploy invariants.

Rollback

To return per-change CI to the pre-cutover topology, revert the cutover commit. That restores the previous ci.yml (full checks, styleguide smoke, both E2E shards, and the strict e2e aggregate on every PR update and main push). The revert also restores ci-lanes.test.mjs's CI_LANES_CUTOVER guard, so the ci.yml assertions apply only when CI_LANES_CUTOVER=1 and the Full Verification and deploy invariants keep running unconditionally. Deployment gating is unaffected by this revert. To run the full suite on every pull request while keeping the lightweight ci.yml, use the one-line paths removal described in "Returning to stricter verification" instead.