--- title: "Why does CI fail then pass on rerun? | StarSling" description: "A rerun that passes with no code change usually means a timing-dependent wait, a hung watch-mode test process, or a missing job timeout, not a real test bug." url: https://starsling.dev/github-actions/problems/flaky-tests canonicalUrl: https://starsling.dev/github-actions/problems/flaky-tests --- # CI fails, then passes on an unchanged rerun [GitHub Actions](https://starsling.dev/github-actions) / [Problems](https://starsling.dev/github-actions/problems) / CI fails, then passes on an unchanged rerun - [How StarSling works](https://starsling.dev/) Diagnosis mode: hybrid. Last updated: 2026-08-31 When the same commit fails once and then passes with nothing changed, the cause is almost always timing, not logic: a fixed sleep that sometimes loses the race, a container polled before it is ready, a test runner that never terminates on its own, or a job with no bound on how long it can hang. Confirm which one from the run history before touching test code. ## Table of contents - [Symptoms](#symptoms) - [How to diagnose it](#how-to-diagnose-it) - [Likely causes](#likely-causes) - [Hand it to an agent](#verify) - [Related pages](#related-pages) - [Related symptoms](#related-symptoms) - [Sources](#sources) ## Symptoms A GitHub Actions job that fails and then passes on an identical commit did not run different code, it ran the same code against different timing. Most workflows and test suites still assume the environment responds within some assumed window: a fixed-millisecond sleep, a container that is usually ready by the time the next step runs, a test process that is usually killed cleanly. Runner load, network latency, and container startup time all vary run to run, so any step that assumes a fixed window instead of confirming a real state eventually lands on the wrong side of it. The order below moves from the narrowest source of timing variance (a single sleep in test source) to the widest (an entire non-required check adding noise to the PR's overall status), because narrower causes are cheaper to confirm and rule out first. - You click rerun on the exact same commit with no code change, and the job that failed now passes. - The failure is a timeout, a connection refused, or an element/selector not found, not an assertion mismatch in your own logic. - The failure rate for one job looks steady over weeks rather than tied to a specific PR or change. - Different runs of the same job fail at different steps or after different elapsed times. - A check shows red on the PR but nobody investigates it because it has never been required to merge. ## How to diagnose it [ci-speedup](https://starsling.dev/ci-speedup) carries the detection logic behind the causes below and opens the fix as a reviewable pull request. Install it with `npx skills add starslingdev/skills`, then run `/ci-speedup` in your repository. To check by hand: 1. Pull the failing workflow's run history and compute its real failure rate from the totals GitHub already returns: `gh api "repos/{owner}/{repo}/actions/workflows/{workflow_id}/runs?status=failure&per_page=1" --jq .total_count` alongside the same call with `status=success`, then `failures / (failures + successes)`. Quote the URL - an unquoted `?` and `&` are shell metacharacters, so zsh rejects the command outright and bash backgrounds it and drops `per_page`. Read `total_count` rather than counting `workflow_runs`: a page caps at 100 entries, so counting items drives the ratio toward 50% on exactly the high-volume workflows this step is for. A rate holding well above what a handful of unlucky runs would produce means this is systemic, not a one-off. 2. For runs where `run_attempt > 1`, compare `GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs?filter=all` against `?filter=latest` to see which job the earlier attempts blamed. If the same job name is the dominant failure across every retried run, you have one genuine flake to chase, not scattered unrelated causes. 3. Open that job's log for a recent failing run and read its `conclusion` and elapsed time. A hard error partway through a step points at causes 1 or 2 below; a run that stops only once it hits a very round elapsed time, or the platform's 360-minute ceiling, points at cause 4. 4. `grep -rn 'waitForTimeout(\|cy\.wait([0-9]\|sleep [0-9]' .github/workflows/ tests/ e2e/ docker-compose*.yml` - a hit inside the failing job's test files or service setup is cause 1 or 2. 5. `grep -rn 'vitest' .github/workflows/ | grep -vE "vitest([[:space:]]+[^[:space:]]+)*[[:space:]]+(run|--run)([[:space:]]|$)"` on the failing job's workflow - a surviving match is cause 3. 6. If the sleeps and run mode both check out clean, confirm whether the red check is even required. Resolve the branch the PR actually targets first - `base=$(gh pr view --json baseRefName --jq .baseRefName)` - then `gh api "repos/{owner}/{repo}/branches/$base/protection/required_status_checks"`. Do not leave a literal `{branch}`, which gh expands to the branch you are standing on, and do not hardcode `main`, which 404s on a repo whose base is `master`, `trunk` or `develop` and would be misread as the no-protection case below. Then read the outcome before concluding: a `200` lists the required checks, and your check being absent from it, with no job `needs:`ing its output, is cause 5. A `404` means the branch has no classic protection or its rules live in a ruleset, so check `gh api repos/{owner}/{repo}/rulesets` before deciding; a `403` means the token cannot read protection at all. Both are "could not tell", not "not required" - only a successful read that omits the check confirms cause 5. ## Likely causes Ordered by how often each one turns out to be the answer. Confirm a cause with its check before you change anything. ### 1. A fixed-length sleep in the test source loses the timing race Playwright's `page.waitForTimeout`, Cypress's `cy.wait(ms)`, and a raw `sleep()` all wait for a fixed number of milliseconds chosen against a typical run, not for the actual condition the test needs. Under normal runner load the wait is long enough and the test passes; under a slower run, contended CPU, or a slightly delayed network response, the same fixed window is not, and the test fails on an assertion that would have passed a second later. Nothing in the code changed between the failing and passing run, only how much of the fixed window the environment actually needed that time. Confirm it: `grep -rn 'waitForTimeout(\|cy\.wait([0-9]\|await sleep(' tests/ e2e/ packages/ --include='*.spec.ts' --include='*.test.ts'` - a hit in the failing test's file is the mechanism, not a coincidence. Fix: [Polling waits, not fixed sleeps](https://starsling.dev/best-practices/github-actions/replace-fixed-sleeps-with-polling) (`ci.hygiene.polling-waits`, detection mode static) ### 2. `sleep N` stands in for a container healthcheck A workflow step that runs `sleep 10` before hitting a database or service container is guessing how long that container takes to accept connections, not confirming it. Container startup time depends on the runner's current load and image cache state, so the same 10 seconds that is plenty on one run is short on the next, and the step after it fails to connect. A rerun on the same commit can land in either window, so it looks identical to the source-level sleep above but the fix and the failing service are different. Confirm it: `grep -rn 'sleep [0-9]' .github/workflows/ docker-compose*.yml` - if the sleep sits directly before a step that connects to a database, cache, or other service container, that step's connection failure is this cause. Fix: [Container healthchecks](https://starsling.dev/best-practices/github-actions/wait-for-container-healthchecks) (`ci.hygiene.container-healthchecks`, detection mode static) ### 3. vitest is running in watch mode and never exits on its own vitest decides between watch mode and run-once mode at startup by reading the environment, and a bare `vitest` invocation without `run` or `--run` only stays in run-once mode because GitHub-hosted runners happen to set `CI=true`. Any step that reaches vitest without that variable set, a container step with a scrubbed environment, a self-hosted runner image, a wrapper script, keeps the process alive watching for file changes that never come. The job then runs until something else kills it, and whether that kill lands mid-suite, after a partial report, or exactly at a timeout boundary varies by run. Confirm it: `grep -rn 'vitest' .github/workflows/ | grep -vE "vitest([[:space:]]+[^[:space:]]+)*[[:space:]]+(run|--run)([[:space:]]|$)"` - a surviving match is a vitest invocation with no explicit run mode. Fix: [Explicit vitest run mode in CI](https://starsling.dev/github-actions/optimizations/optimize-vitest) (`ci.hygiene.vitest-run-mode`, detection mode static) ### 4. The job has no `timeout-minutes`, so a hang runs to the 360-minute default A GitHub Actions job with no `timeout-minutes` set inherits the platform default of 360 minutes before anything reclaims the runner. A genuinely hung step, an open socket, a container that never becomes healthy, a process waiting on input, does not fail cleanly at a consistent point under that default; it runs until GitHub's own limit or an unrelated external timeout cuts it off, which lands at a different elapsed time and sometimes a different step on every run. That inconsistency is what makes the same underlying hang look like an unrelated, unpredictable failure each time. Confirm it: `grep -n 'timeout-minutes' .github/workflows/*.yml` against the job that failed - if it is absent, the job has no bound tighter than 360 minutes. Fix: [Job timeouts](https://starsling.dev/best-practices/github-actions/bound-job-timeouts) (`ci.hygiene.job-timeouts`, detection mode static) ### 5. A non-required check on the critical path is the one going red A workflow that runs on every push, sits high in the PR's check list, and produces genuinely advisory output (a size-diff comment, a preview deploy) is not required to merge, but its red status still shows next to your real test jobs. If that check's own steps are timing-sensitive or hit an external service, it fails and passes independently of your test suite, and from the PR view that reads as "CI is flaky" even though the checks that actually gate the merge never moved. Confirm this only after enumerating what depends on the check's output; a check that looks advisory but feeds a required aggregator is required in effect. Confirm it: Identify which specific check went red in the PR's checks list, then run `base=$(gh pr view --json baseRefName --jq .baseRefName)` and `gh api "repos/{owner}/{repo}/branches/$base/protection/required_status_checks"`. Resolve the base rather than leaving a literal `{branch}` (gh expands it to your current branch) or hardcoding `main` (a 404 on a `master`/`trunk` base reads as the no-protection case). If the call succeeds and that check's name is absent from the required set, it is not gating the merge regardless of its color. A `404` (no classic protection, or rules kept as a ruleset - check `gh api repos/{owner}/{repo}/rulesets`) or a `403` (token cannot read protection) tells you nothing about whether the check is required; resolve that before treating this as the cause. Fix: [Advisory checks non-blocking](https://starsling.dev/best-practices/github-actions/keep-advisory-checks-non-blocking) (`ci.hygiene.advisory-non-blocking`, detection mode runtime) ## Hand it to an agent Hand this prompt to your coding agent (Claude Code, Cursor, and the like) to run this diagnosis against your repository and report which cause it found: Diagnose why a GitHub Actions job in this repository fails and then passes on an identical rerun, without changing anything yet. First pull the job's run history with the GitHub Actions REST API (`GET /repos/{owner}/{repo}/actions/runs`, filtered by `status`) to confirm a real failure rate rather than one unlucky run. Then check, in this order: (1) fixed-length waits in the failing test's source (`page.waitForTimeout`, `cy.wait(ms)`, raw `sleep()`); (2) a `sleep N` step standing in for a container healthcheck in the same workflow; (3) a `vitest` invocation with no `run`/`--run` flag on the failing job; (4) whether that job has `timeout-minutes` set at all; (5) whether the check that actually went red is in the repo's required-status-checks list. Stop at the first cause you confirm, report which one it is and the exact evidence (grep match, API result, or log line), and do not edit any workflow, test, or config file until you report back. ## Related pages - [Replace fixed CI sleeps with bounded readiness polling](https://starsling.dev/best-practices/github-actions/replace-fixed-sleeps-with-polling) - [Wait for container healthchecks instead of sleeping](https://starsling.dev/best-practices/github-actions/wait-for-container-healthchecks) - [Make vitest run mode explicit in GitHub Actions](https://starsling.dev/github-actions/optimizations/optimize-vitest) - [Bound GitHub Actions runner jobs with timeout-minutes](https://starsling.dev/best-practices/github-actions/bound-job-timeouts) - [Keep non-required checks off the GitHub Actions critical path](https://starsling.dev/best-practices/github-actions/keep-advisory-checks-non-blocking) - [Full GitHub Actions best-practices catalog](https://starsling.dev/best-practices/github-actions) ## Related symptoms Other symptoms on this site that share a likely cause with this one. If none of the causes above is yours, one of these is usually the page you wanted. - [A job hits a timeout, or hangs until GitHub kills it](https://starsling.dev/github-actions/problems/github-actions-timeouts) - [Why is the Playwright job so slow in CI](https://starsling.dev/github-actions/problems/slow-playwright) - [A GitHub Actions job stays queued or hangs with no output](https://starsling.dev/github-actions/problems/github-actions-jobs-stuck) ## Sources - [GitHub REST API: List workflow runs for a repository](https://docs.github.com/en/rest/actions/workflow-runs?apiVersion=2022-11-28) - [GitHub REST API: List jobs for a workflow run](https://docs.github.com/en/rest/actions/workflow-jobs?apiVersion=2022-11-28) - [GitHub Actions workflow syntax: jobs..timeout-minutes](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idtimeout-minutes) - [Vitest CLI reference](https://vitest.dev/guide/cli)