Fix flaky tests from CI history

Find tests that fail and then pass on the same commit, and fix clear causes in the test itself.

Prompt

Evaluate and apply this CI technique to my repository: Fix flaky tests from CI history.

Find tests that fail and then pass on the same commit, and fix clear causes in the test itself.

Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/flaky-tests.md

Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence.
Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured.
Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo.
Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES.
Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch.
Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge.

View all playbooks

How OpenAI used this technique

Read what OpenAI changed and follow the original sources.

Fix flaky tests from CI history

The post suggests that prompt for finding flaky tests. In an earlier Codex change (openai/codex#16825, merged April 2026), an approval-policy test launched a second shell command and could fail on Windows startup timing. The fix keeps the escalation-rejection check, asserts that no turn permissions were granted, and checks the approval decision directly.

Use it when: A required check fails, then passes on an identical rerun.
Before applying it: Preserve the behavior and assertions being tested. Retries, larger timeouts and skipped tests can hide the underlying failure.

How OpenAI used this technique

Fix flaky tests from CI history

Company Playbooks

Related guides

Flaky tests: CI fails, then passes on an unchanged rerun