--- title: "Prove a new test is stable before it joins the blocking suite" description: "Run each new end-to-end test many times in a dedicated pipeline, and admit it to the blocking checks only if it stays under a failure threshold." url: https://starsling.dev/ci-playbooks/techniques/flakiness-gate-before-blocking canonicalUrl: https://starsling.dev/ci-playbooks/techniques/flakiness-gate-before-blocking --- # Prove a new test is stable before it joins the blocking suite Run each new end-to-end test many times in a dedicated pipeline, and admit it to the blocking checks only if it stays under a failure threshold. ## How to choose an approach This technique currently has one documented company example. Validate its prerequisites, tradeoffs and original CI environment against your repository before adapting it. - **[Shopify](https://starsling.dev/ci-playbooks/shopify#flakiness-gate-before-blocking)**: Use it when A suite blocks merges, new tests are added continuously, and a flaky one would stop good pull requests. Check before adopting: The promotion runs cost runner minutes before a test is admitted, so choose the number of runs and the threshold from the suite's own history. ## Apply this technique ````text Evaluate and apply this CI technique to my repository: Prove a new test is stable before it joins the blocking suite. Run each new end-to-end test many times in a dedicated pipeline, and admit it to the blocking checks only if it stays under a failure threshold. Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/flakiness-gate-before-blocking.md Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence. Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured. Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo. Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES. Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch. Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge. ```` ## How Shopify used this technique Read what Shopify changed and follow the original sources. ### Shopify: Prove a new test is stable before it joins the blocking suite Shopify's mobile app end-to-end suite had become so flaky that it was pulled from pull request checks entirely. Shopify rebuilt the framework around a strict API in which every step carries an assertion, then promoted it back into blocking CI at 98% test stability, up from 50% using the old API, measured as individual test successes divided by total runs. Shopify also built a pre-promotion flakiness gate: a dedicated pipeline runs a new test multiple times and rejects it if it fails above a set threshold. The OPT48 recipe reads a job's failure rate; here the rate is measured per test before the test blocks anyone. Evidence: Company-reported result or observation 2026: Test stability went from 50% on the old API to 98% on the rebuilt framework, measured as individual test successes divided by total runs. ([Source](https://shopify.engineering/mobile-e2e-testing)) Use it when: A suite blocks merges, new tests are added continuously, and a flaky one would stop good pull requests. Before applying it: The promotion runs cost runner minutes before a test is admitted, so choose the number of runs and the threshold from the suite's own history. [Read more in the Shopify playbook](https://starsling.dev/ci-playbooks/shopify#flakiness-gate-before-blocking) [Primary source](https://shopify.engineering/mobile-e2e-testing) ## CI optimization is continuous work. Apply this technique yourself, or use StarSling Runners for faster GitHub Actions and optional agent-driven optimization PRs, which teams on eligible paid plans can turn on. [Get self-improving CI](https://starsling.dev/products/runners) ## Related techniques ### [Check for leaks after this test cleans up](https://starsling.dev/ci-playbooks/techniques/check-cleanup-at-test-end) Run the leak check at the end of the test, where it inspects this test's own resources. 1 implementation · 1 company (Uber) ### [Make unordered results deterministic before comparing](https://starsling.dev/ci-playbooks/techniques/normalize-unordered-results) Sort values whose order was never guaranteed, so an equivalent result passes on the first run. 1 implementation · 1 company (Uber) ### [Wait for the state transition before asserting](https://starsling.dev/ci-playbooks/techniques/synchronize-assertions) Synchronize with the event that makes the assertion true, so the test stops needing reruns. 1 implementation · 1 company (Uber) ## Company Playbooks ### [Anthropic](https://starsling.dev/ci-playbooks/anthropic) Run only affected tests ### [DoorDash](https://starsling.dev/ci-playbooks/doordash) Run build and test in parallel, sharded across CI agents ### [Linear](https://starsling.dev/ci-playbooks/linear) Benchmark the same job on faster hardware ### [Mastra](https://starsling.dev/ci-playbooks/mastra) StarSling runners for the slowest test workflow ### [OpenAI](https://starsling.dev/ci-playbooks/openai) Rank CI fixes by the minutes they save ### [Ramp](https://starsling.dev/ci-playbooks/ramp) Move a CPU-bound test suite to a faster CPU ### [Shopify](https://starsling.dev/ci-playbooks/shopify) Let an agent iterate against one CI metric ### [Spotify](https://starsling.dev/ci-playbooks/spotify) Build arm64 on native arm64 runners ### [Stripe](https://starsling.dev/ci-playbooks/stripe) Run only the tests a change can reach ### [Uber](https://starsling.dev/ci-playbooks/uber) Build the merge-queue speculations most likely to be needed ## Related guides Implementation guides for this technique. ### [Flaky tests: CI fails, then passes on an unchanged rerun](https://starsling.dev/github-actions/problems/flaky-tests) Flaky tests fail, then pass on an unchanged rerun. Common causes: a fixed sleep that loses a timing race, a container used before it is ready, or a hung job.