Balance test work across shards

Use measured runtimes or smaller test files to reduce the slowest shard, which determines when the suite finishes.

Prompt

Evaluate and apply this CI technique to my repository: Balance test work across shards.

Use measured runtimes or smaller test files to reduce the slowest shard, which determines when the suite finishes.

Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/balance-test-shards.md

Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence.
Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured.
Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo.
Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES.
Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch.
Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge.

View all playbooks

How companies used this technique

Read what each company changed and follow the original sources.

Balance test files before buying more shards

Linear split oversized test files into smaller files, then benchmarked shard and runner configurations. Moving from four to eight shards made the critical job roughly 19% faster and 19% cheaper in the initial benchmark.

Use it when: A few files dominate a file-based shard assignment.
Before applying it: Preserve test discovery, assertions and artifacts. Vitest assigns work by file, so one large file still sets its shard’s time.

Split long test jobs into balanced shards

Codex split its single Windows Bazel test job into four shard jobs behind one aggregate check. A later change assigns targets by estimated duration, longest first, to the least-loaded shard.

Windows tests: one sharded run in about 7m29s, against a recent single-job median around 22m26s Source

Use it when: A test job dominates the critical path, or existing shards finish at very different times.
Before applying it: Balance shard duration and total runner minutes together. A checked-in timing table goes stale as new tests arrive, so refresh it or use the runner's own timing data.

Pack tests onto workers so they finish together

With the suite running in parallel pytest-xdist workers, Ramp saw the workers finish at different times, so the slowest worker set the job's length. The team wrote a more effective packing algorithm, with ChatGPT's help, as a proof of concept on a fork of pytest.

The more effective packing algorithm saved 2 to 3 minutes. Source

Use it when: A test job runs in parallel workers or matrix shards, and the last one regularly finishes well after the first.
Before applying it: A packing algorithm that uses test timings needs them recorded; refresh them as tests are added so new slow tests spread across the workers.

Balance shards and keep a catch-all for new tests

Stripe iOS balanced PaymentSheet UI tests across four jobs: three with explicit assignments and a fourth that runs everything else, including new tests. A validator rejects duplicate assignments and stale exclusions.

In Stripe’s combined CI runs, the slowest PaymentSheet UI job went from about 26 to about 12 minutes, with worker and machine changes in the same PR. Source

Use it when: One shard sets the finish time while others wait idle.
Before applying it: Keep the union of test IDs identical and track new, renamed and deleted tests.

How companies used this technique

Balance test files before buying more shards

Split long test jobs into balanced shards

Pack tests onto workers so they finish together

Balance shards and keep a catch-all for new tests

Related techniques

Split long test suites across runners

Reduce the setup floor before increasing shard count

Split independent work off the longest job

Company Playbooks

Related guides

Shard tests across parallel jobs

Cut repeated test setup

Diagnose slow tests in CI

Optimize Playwright tests

Make Vitest run mode explicit

Parallelize independent jobs