Run independent test packages concurrently
Let independent packages or test environments run at the same time instead of invoking each one serially.
How companies used this technique
Read what each company changed and follow the original sources.
Run independent test packages concurrently
Bun's Miri script builds every crate once, then runs one cargo miri test per crate, as many at a time as the runner has cores, longest crates first. It waits for all of them and exits nonzero if any crate reports an error.
The PR expects the Miri job to go from about 14.5 minutes to about 4 to 5 minutes (estimate), with the concurrent crates and a smaller Miri-sized tracker test together. Source
- Use it when
- A serial test driver runs independent packages, crates or modules one after another on a multi-core runner.
- Before applying it
- Each process keeps its own log, and the job reports every failing package.
Run the slowest test environment in parallel workers
This is a community contribution to Spotify's luigi project. Luigi's core test environment took about five minutes while the others took about a minute or less. The change adds pytest-xdist and runs the core tests with -n auto --dist=loadfile, which spreads test files across one worker per CPU and keeps each file's tests on one worker.
The author reports the core workflows running in 2 to 4 minutes, 1 to 2 minutes faster. Source
- Use it when
- One test environment takes most of the CI time and runs in a single process on a multi-core runner.
- Before applying it
- Tests that share files, ports or global state need per-worker isolation; grouping by file keeps a file's tests together.
Let the test runner schedule independent packages
After isolating its unit suite, Cadence replaced a shell loop that ran go test once per package with one invocation over the package list, so Go schedules independent packages concurrently and writes one coverage profile. OPT88's serial-suite diagnosis is adapted.
The PR reports unit and integration test runs and builds on the dependency isolation in cadence-workflow/cadence#6800. Source
- Use it when
- A shell loop runs independent packages one at a time even though the language test runner schedules packages in parallel.
- Before applying it
- Remove shared-resource dependencies first, keep special-case tests, race settings and combined coverage, and measure CPU and memory at several concurrency levels.
Company Playbooks
- Read Anthropic’s playbookRun only affected tests
- Read DoorDash’s playbookRun build and test in parallel, sharded across CI agents
- Read Linear’s playbookBenchmark the same job on faster hardware
- Read Mastra’s playbookStarSling runners for the slowest test workflow
- Read OpenAI’s playbookRank CI fixes by the minutes they save
- Read Ramp’s playbookMove a CPU-bound test suite to a faster CPU
- Read Shopify’s playbookLet an agent iterate against one CI metric
- Read Spotify’s playbookBuild arm64 on native arm64 runners
- Read Stripe’s playbookRun only the tests a change can reach
- Read Uber’s playbookBuild the merge-queue speculations most likely to be needed