--- title: "Split long test suites across runners" description: "Distribute a suite across CI jobs while keeping complete test discovery and collecting the results from every shard." url: https://starsling.dev/ci-playbooks/techniques/shard-test-suites canonicalUrl: https://starsling.dev/ci-playbooks/techniques/shard-test-suites --- # Split long test suites across runners Distribute a suite across CI jobs while keeping complete test discovery and collecting the results from every shard. ## How to choose an approach Compare the examples by repository setup, prerequisites, and risks before adopting one. Each company implemented this in its own environment; measure each change against your own baseline. - **[Anthropic](https://starsling.dev/ci-playbooks/anthropic#shard-test-files)**: Use it when One test job runs a large suite on a single runner, and the suite can be split by file or run in more than one process. Check before adopting: Splitting by path balances file counts. Compare shard durations after the change, and move to duration-based splitting if one shard stays the long pole. Tests that share state need isolation before they run in parallel. - **[DoorDash](https://starsling.dev/ci-playbooks/doordash#shard-test-phase)**: Use it when One test job runs long, or runs out of memory, after raising its worker count on one machine. Check before adopting: Every shard repeats setup, so pick the shard count by measurement, and collate coverage so the report still covers the whole suite. Adapted for GitHub Actions; compare the original system with yours. - **[Mastra](https://starsling.dev/ci-playbooks/mastra#test-shards)**: Use it when A long test job remains the critical path after removing repeated waits and setup. Check before adopting: Each shard repeats setup and consumes runner minutes. Verify the union of tests, unique artifacts and required aggregate status. - **[OpenAI](https://starsling.dev/ci-playbooks/openai#split-slow-job)**: Use it when One job sets the workflow's finish time and its tests fall into independent groups. Check before adopting: Each split job repeats setup and adds runner minutes. - **[Shopify](https://starsling.dev/ci-playbooks/shopify#measure-shards)**: Use it when One test job is the longest in the workflow after its tests are isolated and its worker count is tuned. Check before adopting: Each shard repeats checkout, install and authentication, so total runner minutes can rise while elapsed time falls. - **[Spotify](https://starsling.dev/ci-playbooks/spotify#shard-e2e)**: Use it when A test suite runs serially because its tests share a fixed resource on one machine, and it takes most of the pull request run. Check before adopting: Each shard repeats checkout and setup, and the slowest test file sets the floor for every shard. - **[Stripe](https://starsling.dev/ci-playbooks/stripe#shard-with-complete-discovery)**: Use it when One long test job remains the bottleneck after setup and worker tuning. Check before adopting: Each shard repeats setup. Verify the shards together run every original test exactly once, and keep device and diagnostic coverage. ## Apply this technique ````text Evaluate and apply this CI technique to my repository: Split long test suites across runners. Distribute a suite across CI jobs while keeping complete test discovery and collecting the results from every shard. Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/shard-test-suites.md Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence. Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured. Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo. Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES. Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch. Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge. ```` ## How companies used this technique Read what each company changed and follow the original sources. ### Anthropic: Split test files across CI shards that stay balanced to within one file Bun v1.3.13 added bun test --shard=M/N to split test files across CI runners, in the syntax Jest, Vitest and Playwright use. Files are sorted by path for determinism and distributed round-robin, so each shard stays balanced to within one file of the others. Inside one runner, bun test --parallel distributes files across worker processes, the CPU count by default, and idle workers steal work from the busiest remaining queue. Output stays identical to serial execution. Evidence: Company-reported result or observation Split test files across multiple CI runners with the new --shard flag, matching the syntax used by Jest, Vitest, and Playwright. ([Source](https://bun.sh/blog/bun-v1.3.13)) Use it when: One test job runs a large suite on a single runner, and the suite can be split by file or run in more than one process. Before applying it: Splitting by path balances file counts. Compare shard durations after the change, and move to duration-based splitting if one shard stays the long pole. Tests that share state need isolation before they run in parallel. [Read more in the Anthropic playbook](https://starsling.dev/ci-playbooks/anthropic#shard-test-files) [Primary source](https://bun.sh/blog/bun-v1.3.13) ### DoorDash: Shard the test phase across agents **Translated to GitHub Actions** - What DoorDash built: DoorDash added sharding to Rush: a sharded operation becomes a pre-shard node, n shard nodes and a collate node that merges coverage. - Your GitHub Actions version: A test matrix of shards with each shard's index and its own output folder, followed by a job that merges coverage into one report. DoorDash's tests already ran eight Jest workers per machine and had started to hit memory errors, so DoorDash added sharding to Rush upstream. A sharded operation becomes a pre-shard node that keeps the dependencies, n shard nodes, and a collate node that merges coverage into one report; each shard gets its index and its own output folder so the build cache stays valid. A follow-up clusters shards by operation name, so shards of one package can run on different agents. Four shards with seven workers each ran fastest, and every shard finished within six minutes, the same time as the build. DoorDash's Kotlin build service shards unit tests the same way. Evidence: Company-reported result or observation All of the test shards completed within six minutes, which tied build time and test time. ([Source](https://careersatdoordash.com/blog/how-doordash-slashed-web-developer-build-times/)) Use it when: One test job runs long, or runs out of memory, after raising its worker count on one machine. Before applying it: Every shard repeats setup, so pick the shard count by measurement, and collate coverage so the report still covers the whole suite. GitHub Actions adaptation: Compare the source system with your own workflow before applying this translation. [Read more in the DoorDash playbook](https://starsling.dev/ci-playbooks/doordash#shard-test-phase) [Primary source](https://careersatdoordash.com/blog/how-doordash-slashed-web-developer-build-times/) ### Mastra: Split a long end-to-end suite across runners StarSling’s agent added three Playwright shards to the kitchen-sink job, passed each shard index to the test runner and kept fail-fast false. Evidence: Source PR/commit evidence Mastra’s kitchen-sink E2E suite still runs sharded today, now across four shards in e2e-tests.yml. ([Source](https://github.com/mastra-ai/mastra/blob/f92e947a448a47515acf14df1eddc5fd5c36f424/.github/workflows/e2e-tests.yml)) Use it when: A long test job remains the critical path after removing repeated waits and setup. Before applying it: Each shard repeats setup and consumes runner minutes. Verify the union of tests, unique artifacts and required aggregate status. [Read more in the Mastra playbook](https://starsling.dev/ci-playbooks/mastra#test-shards) [Primary source](https://github.com/mastra-ai/mastra/pull/15888) ### OpenAI: Split the slowest job across runners The CLI's help-compatibility workflow split its dispatch and contract tests into separate jobs on Intel macOS and Windows ARM, and moved an exhaustive flag-inventory test, which starts 482 subprocesses per platform, out of that matrix; the full Linux suite still runs it. Evidence: Source PR/commit evidence Workflow: 7m10s in the first successful run after merging current main, against the earlier 23-run median of 10m21s ([Source](https://github.com/openai/openai-cli/pull/511)) Use it when: One job sets the workflow's finish time and its tests fall into independent groups. Before applying it: Each split job repeats setup and adds runner minutes. [Read more in the OpenAI playbook](https://starsling.dev/ci-playbooks/openai#split-slow-job) [Primary source](https://github.com/openai/openai-cli/pull/511) ### Shopify: Give a long suite more than one runner After measuring worker and shard combinations, the CLI raised Playwright's workers setting from 5 to 10 and added a two-shard matrix to its end-to-end job, giving each shard's artifacts a unique name. Each shard is a separate job that runs Playwright with the configured worker limit, so ten workers means up to ten on each machine. The default branch still runs two shards. The CLI's Windows unit tests are sharded the same way. Evidence: Source PR/commit evidence The PR reports ~4 min faster wall clock: 7m 44s against 11m 43s, a 34% reduction, comparing 10 workers on two shards with 5 workers on one runner. ([Source](https://github.com/Shopify/cli/pull/7350)) Use it when: One test job is the longest in the workflow after its tests are isolated and its worker count is tuned. Before applying it: Each shard repeats checkout, install and authentication, so total runner minutes can rise while elapsed time falls. [Read more in the Shopify playbook](https://starsling.dev/ci-playbooks/shopify#measure-shards) [Primary source](https://github.com/Shopify/cli/pull/7350) ### Spotify: Shard a serial end-to-end suite across runners The confidence-cli end-to-end suite runs serially because it uses a fixed auth callback port, and it took about 4.4 minutes of a 5.5-minute pull request run. The change splits it into four matrix shards with vitest --shard, each on its own runner, and adds an aggregate E2E tests job so the check name stays the same. Evidence: Estimate Before the change, E2E took about 4.4 minutes of a 5.5-minute pull request run; the author estimated about 1.5 minutes after it, with the slowest test file at 74 seconds as the floor (estimate). ([Source](https://github.com/spotify/confidence-cli/pull/73)) Use it when: A test suite runs serially because its tests share a fixed resource on one machine, and it takes most of the pull request run. Before applying it: Each shard repeats checkout and setup, and the slowest test file sets the floor for every shard. [Read more in the Spotify playbook](https://starsling.dev/ci-playbooks/spotify#shard-e2e) [Primary source](https://github.com/spotify/confidence-cli/pull/73) ### Stripe: Shard a long suite across runners Stripe Android partitions discovered PaymentSheet end-to-end test classes across five parallel workflows. Stripe React Native split its end-to-end tests into matrix shards, and Stripe iOS shards full SDK size reports into three jobs on master and releases. Evidence: Source PR/commit evidence Use it when: One long test job remains the bottleneck after setup and worker tuning. Before applying it: Each shard repeats setup. Verify the shards together run every original test exactly once, and keep device and diagnostic coverage. [Read more in the Stripe playbook](https://starsling.dev/ci-playbooks/stripe#shard-with-complete-discovery) [Primary source](https://github.com/stripe/stripe-android/pull/13225) ## CI optimization is continuous work. Apply this technique yourself, or use StarSling Runners for faster GitHub Actions and optional agent-driven optimization PRs, which teams on eligible paid plans can turn on. [Get self-improving CI](https://starsling.dev/products/runners) ## Related techniques ### [Balance test work across shards](https://starsling.dev/ci-playbooks/techniques/balance-test-shards) Use measured runtimes or smaller test files to reduce the slowest shard, which determines when the suite finishes. 4 implementations · 4 companies (Linear, OpenAI, Ramp, Stripe) ### [Reduce the setup floor before increasing shard count](https://starsling.dev/ci-playbooks/techniques/setup-cost-limits-sharding) Each extra shard repeats checkout, dependencies and container setup. 1 implementation · 1 company (Linear) ### [Split independent work off the longest job](https://starsling.dev/ci-playbooks/techniques/split-long-lanes) Run independent suites in parallel jobs when the overlap saves more than the repeated setup costs. 1 implementation · 1 company (Stripe) ## Company Playbooks ### [Anthropic](https://starsling.dev/ci-playbooks/anthropic) Run only affected tests ### [DoorDash](https://starsling.dev/ci-playbooks/doordash) Run build and test in parallel, sharded across CI agents ### [Linear](https://starsling.dev/ci-playbooks/linear) Benchmark the same job on faster hardware ### [Mastra](https://starsling.dev/ci-playbooks/mastra) StarSling runners for the slowest test workflow ### [OpenAI](https://starsling.dev/ci-playbooks/openai) Rank CI fixes by the minutes they save ### [Ramp](https://starsling.dev/ci-playbooks/ramp) Move a CPU-bound test suite to a faster CPU ### [Shopify](https://starsling.dev/ci-playbooks/shopify) Let an agent iterate against one CI metric ### [Spotify](https://starsling.dev/ci-playbooks/spotify) Build arm64 on native arm64 runners ### [Stripe](https://starsling.dev/ci-playbooks/stripe) Run only the tests a change can reach ### [Uber](https://starsling.dev/ci-playbooks/uber) Build the merge-queue speculations most likely to be needed ## Related guides Implementation guides for this technique. ### [Shard tests across parallel jobs](https://starsling.dev/best-practices/github-actions/shard-tests) Split a long suite with a matrix and your test framework’s sharding support. ### [Cut repeated test setup](https://starsling.dev/best-practices/github-actions/cut-repeated-test-setup) Move shared fixture work out of individual test cases and keep necessary cleanup. ### [Diagnose slow tests in CI](https://starsling.dev/github-actions/problems/slow-tests) Check common causes in order and confirm each with workflow evidence. ### [Optimize Playwright tests](https://starsling.dev/github-actions/optimizations/optimize-playwright) Choose trace and video retention and install the browsers your suite uses. ### [Make Vitest run mode explicit](https://starsling.dev/github-actions/optimizations/optimize-vitest) Pin run mode to the test command and bound the job with a timeout. ### [Parallelize independent jobs](https://starsling.dev/github-actions/optimizations/parallelize-independent-jobs) Run independent validation lanes concurrently and keep genuine job dependencies.