---
title: "Fix flaky tests from CI history"
description: "Find tests that fail and then pass on the same commit, and fix clear causes in the test itself."
url: https://starsling.dev/ci-playbooks/techniques/flaky-tests
canonicalUrl: https://starsling.dev/ci-playbooks/techniques/flaky-tests
---

# Fix flaky tests from CI history

Find tests that fail and then pass on the same commit, and fix clear causes in the test itself.

## How to choose an approach

This technique currently has one documented company example. Validate its prerequisites, tradeoffs and original CI environment against your repository before adapting it.

- **[OpenAI](https://starsling.dev/ci-playbooks/openai#flaky-tests)**: Use it when A required check fails, then passes on an identical rerun. Check before adopting: Preserve the behavior and assertions being tested. Retries, larger timeouts and skipped tests can hide the underlying failure.

## Apply this technique

````text
Evaluate and apply this CI technique to my repository: Fix flaky tests from CI history.

Find tests that fail and then pass on the same commit, and fix clear causes in the test itself.

Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/flaky-tests.md

Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence.
Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured.
Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo.
Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES.
Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch.
Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge.
````

## How OpenAI used this technique

Read what OpenAI changed and follow the original sources.

### OpenAI: Fix flaky tests from CI history

The post suggests that prompt for finding flaky tests. In an earlier Codex change (openai/codex#16825, merged April 2026), an approval-policy test launched a second shell command and could fail on Windows startup timing. The fix keeps the escalation-rejection check, asserts that no turn permissions were granted, and checks the approval decision directly.

Evidence: Documented technique

Use it when: A required check fails, then passes on an identical rerun.

Before applying it: Preserve the behavior and assertions being tested. Retries, larger timeouts and skipped tests can hide the underlying failure.

[Read more in the OpenAI playbook](https://starsling.dev/ci-playbooks/openai#flaky-tests)

[Primary source](https://developers.openai.com/blog/maintaining-software-with-the-gpt-6-family#other-ideas-to-try)

## CI optimization is continuous work.

Apply this technique yourself, or use StarSling Runners for faster GitHub Actions and optional agent-driven optimization PRs, which teams on eligible paid plans can turn on.

[Get self-improving CI](https://starsling.dev/products/runners)

## Company Playbooks

### [Anthropic](https://starsling.dev/ci-playbooks/anthropic)

Run only affected tests

### [DoorDash](https://starsling.dev/ci-playbooks/doordash)

Run build and test in parallel, sharded across CI agents

### [Linear](https://starsling.dev/ci-playbooks/linear)

Benchmark the same job on faster hardware

### [Mastra](https://starsling.dev/ci-playbooks/mastra)

StarSling runners for the slowest test workflow

### [OpenAI](https://starsling.dev/ci-playbooks/openai)

Rank CI fixes by the minutes they save

### [Ramp](https://starsling.dev/ci-playbooks/ramp)

Move a CPU-bound test suite to a faster CPU

### [Shopify](https://starsling.dev/ci-playbooks/shopify)

Let an agent iterate against one CI metric

### [Spotify](https://starsling.dev/ci-playbooks/spotify)

Build arm64 on native arm64 runners

### [Stripe](https://starsling.dev/ci-playbooks/stripe)

Run only the tests a change can reach

### [Uber](https://starsling.dev/ci-playbooks/uber)

Build the merge-queue speculations most likely to be needed

## Related guides

Implementation guides for this technique.

### [Flaky tests: CI fails, then passes on an unchanged rerun](https://starsling.dev/github-actions/problems/flaky-tests)

Flaky tests fail, then pass on an unchanged rerun. Common causes: a fixed sleep that loses a timing race, a container used before it is ready, or a hung job.