CI fails, then passes on an unchanged rerun

A GitHub Actions job that fails and then passes on an identical commit did not run different code, it ran the same code against different timing. Most workflows and test suites still assume the environment responds within some assumed window: a fixed-millisecond sleep, a container that is usually ready by the time the next step runs, a test process that is usually killed cleanly. Runner load, network latency, and container startup time all vary run to run, so any step that assumes a fixed window instead of confirming a real state eventually lands on the wrong side of it. The order below moves from the narrowest source of timing variance (a single sleep in test source) to the widest (an entire non-required check adding noise to the PR's overall status), because narrower causes are cheaper to confirm and rule out first.

• >

  You click rerun on the exact same commit with no code change, and the job that failed now passes.

• >

  The failure is a timeout, a connection refused, or an element/selector not found, not an assertion mismatch in your own logic.

• >

  The failure rate for one job looks steady over weeks rather than tied to a specific PR or change.

• >

  Different runs of the same job fail at different steps or after different elapsed times.

• >

  A check shows red on the PR but nobody investigates it because it has never been required to merge.

1. 01

   Pull the failing workflow's run history and compute its real failure rate from the totals GitHub already returns: gh api "repos/{owner}/{repo}/actions/workflows/{workflow_id}/runs?status=failure&per_page=1" --jq .total_count alongside the same call with status=success, then failures / (failures + successes). Quote the URL - an unquoted ? and & are shell metacharacters, so zsh rejects the command outright and bash backgrounds it and drops per_page. Read total_count rather than counting workflow_runs: a page caps at 100 entries, so counting items drives the ratio toward 50% on exactly the high-volume workflows this step is for. A rate holding well above what a handful of unlucky runs would produce means this is systemic, not a one-off.

2. 02

   For runs where run_attempt > 1, compare GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs?filter=all against ?filter=latest to see which job the earlier attempts blamed. If the same job name is the dominant failure across every retried run, you have one genuine flake to chase, not scattered unrelated causes.

3. 03

   Open that job's log for a recent failing run and read its conclusion and elapsed time. A hard error partway through a step points at causes 1 or 2 below; a run that stops only once it hits a very round elapsed time, or the platform's 360-minute ceiling, points at cause 4.

4. 04

   grep -rn 'waitForTimeout(\|cy\.wait([0-9]\|sleep [0-9]' .github/workflows/ tests/ e2e/ docker-compose*.yml - a hit inside the failing job's test files or service setup is cause 1 or 2.

5. 05

   grep -rn 'vitest' .github/workflows/ | grep -vE "vitest([[:space:]]+[^[:space:]]+)*[[:space:]]+(run|--run)([[:space:]]|$)" on the failing job's workflow - a surviving match is cause 3.

6. 06

   If the sleeps and run mode both check out clean, confirm whether the red check is even required. Resolve the branch the PR actually targets first - base=$(gh pr view --json baseRefName --jq .baseRefName) - then gh api "repos/{owner}/{repo}/branches/$base/protection/required_status_checks". Do not leave a literal {branch}, which gh expands to the branch you are standing on, and do not hardcode main, which 404s on a repo whose base is master, trunk or develop and would be misread as the no-protection case below. Then read the outcome before concluding: a 200 lists the required checks, and your check being absent from it, with no job needs:ing its output, is cause 5. A 404 means the branch has no classic protection or its rules live in a ruleset, so check gh api repos/{owner}/{repo}/rulesets before deciding; a 403 means the token cannot read protection at all. Both are "could not tell", not "not required" - only a successful read that omits the check confirms cause 5.

Ordered by how often each one turns out to be the answer. Confirm a cause with its check before you change anything.

1. 01

   A fixed-length sleep in the test source loses the timing race

   Playwright's page.waitForTimeout, Cypress's cy.wait(ms), and a raw sleep() all wait for a fixed number of milliseconds chosen against a typical run, not for the actual condition the test needs. Under normal runner load the wait is long enough and the test passes; under a slower run, contended CPU, or a slightly delayed network response, the same fixed window is not, and the test fails on an assertion that would have passed a second later. Nothing in the code changed between the failing and passing run, only how much of the fixed window the environment actually needed that time.

   Confirm it

   grep -rn 'waitForTimeout(\|cy\.wait([0-9]\|await sleep(' tests/ e2e/ packages/ --include='*.spec.ts' --include='*.test.ts' - a hit in the failing test's file is the mechanism, not a coincidence.

   Fix: Polling waits, not fixed sleeps
2. 02

   `sleep N` stands in for a container healthcheck

   A workflow step that runs sleep 10 before hitting a database or service container is guessing how long that container takes to accept connections, not confirming it. Container startup time depends on the runner's current load and image cache state, so the same 10 seconds that is plenty on one run is short on the next, and the step after it fails to connect. A rerun on the same commit can land in either window, so it looks identical to the source-level sleep above but the fix and the failing service are different.

   Confirm it

   grep -rn 'sleep [0-9]' .github/workflows/ docker-compose*.yml - if the sleep sits directly before a step that connects to a database, cache, or other service container, that step's connection failure is this cause.

   Fix: Container healthchecks
3. 03

   vitest is running in watch mode and never exits on its own

   vitest decides between watch mode and run-once mode at startup by reading the environment, and a bare vitest invocation without run or --run only stays in run-once mode because GitHub-hosted runners happen to set CI=true. Any step that reaches vitest without that variable set, a container step with a scrubbed environment, a self-hosted runner image, a wrapper script, keeps the process alive watching for file changes that never come. The job then runs until something else kills it, and whether that kill lands mid-suite, after a partial report, or exactly at a timeout boundary varies by run.

   Confirm it

   grep -rn 'vitest' .github/workflows/ | grep -vE "vitest([[:space:]]+[^[:space:]]+)*[[:space:]]+(run|--run)([[:space:]]|$)" - a surviving match is a vitest invocation with no explicit run mode.

   Fix: Explicit vitest run mode in CI
4. 04

   The job has no `timeout-minutes`, so a hang runs to the 360-minute default

   A GitHub Actions job with no timeout-minutes set inherits the platform default of 360 minutes before anything reclaims the runner. A genuinely hung step, an open socket, a container that never becomes healthy, a process waiting on input, does not fail cleanly at a consistent point under that default; it runs until GitHub's own limit or an unrelated external timeout cuts it off, which lands at a different elapsed time and sometimes a different step on every run. That inconsistency is what makes the same underlying hang look like an unrelated, unpredictable failure each time.

   Confirm it

   grep -n 'timeout-minutes' .github/workflows/*.yml against the job that failed - if it is absent, the job has no bound tighter than 360 minutes.

   Fix: Job timeouts
5. 05

   A non-required check on the critical path is the one going red

   A workflow that runs on every push, sits high in the PR's check list, and produces genuinely advisory output (a size-diff comment, a preview deploy) is not required to merge, but its red status still shows next to your real test jobs. If that check's own steps are timing-sensitive or hit an external service, it fails and passes independently of your test suite, and from the PR view that reads as "CI is flaky" even though the checks that actually gate the merge never moved. Confirm this only after enumerating what depends on the check's output; a check that looks advisory but feeds a required aggregator is required in effect.

   Confirm it

   Identify which specific check went red in the PR's checks list, then run base=$(gh pr view --json baseRefName --jq .baseRefName) and gh api "repos/{owner}/{repo}/branches/$base/protection/required_status_checks". Resolve the base rather than leaving a literal {branch} (gh expands it to your current branch) or hardcoding main (a 404 on a master/trunk base reads as the no-protection case). If the call succeeds and that check's name is absent from the required set, it is not gating the merge regardless of its color. A 404 (no classic protection, or rules kept as a ruleset - check gh api repos/{owner}/{repo}/rulesets) or a 403 (token cannot read protection) tells you nothing about whether the check is required; resolve that before treating this as the cause.

   Fix: Advisory checks non-blocking

Hand this prompt to your coding agent (Claude Code, Cursor, and the like) to run this diagnosis against your repository and report which cause it found.

Diagnose why a GitHub Actions job in this repository fails and then passes on an identical rerun, without changing anything yet. First pull the job's run history with the GitHub Actions REST API (`GET /repos/{owner}/{repo}/actions/runs`, filtered by `status`) to confirm a real failure rate rather than one unlucky run. Then check, in this order: (1) fixed-length waits in the failing test's source (`page.waitForTimeout`, `cy.wait(ms)`, raw `sleep()`); (2) a `sleep N` step standing in for a container healthcheck in the same workflow; (3) a `vitest` invocation with no `run`/`--run` flag on the failing job; (4) whether that job has `timeout-minutes` set at all; (5) whether the check that actually went red is in the repo's required-status-checks list. Stop at the first cause you confirm, report which one it is and the exact evidence (grep match, API result, or log line), and do not edit any workflow, test, or config file until you report back.