- All playbooks
- Slow tests
Slow tests
Find where tests spend their time, from repeated setup to the longest shard. Compare changes companies made to their test workflows.
Where to start
Diagnose first: is the long pole test execution, repeated setup, slow fixtures, fixed sleeps, or an imbalanced shard?
Measure those phases separately. A slow test job can spend its time in fixture setup, execution, waiting, or cleanup. Measure those phases first, then choose a technique that addresses the expensive part while preserving what the suite verifies.
17 techniques with source-backed company examples.
Isolate state before running tests in parallel
Give concurrent tests independent databases, services, or fixtures so they can share runner capacity without interfering with each other.
Run isolated end-to-end journeys in parallel
The packaged end-to-end suite ran 66 journeys on one worker. Each journey already had its own home directory, workspace, runtime and server port, so the PR set Playwright to three workers with fullyParallel. Global setup sweeps orphaned processes once and installs the Electron runtime once, before any worker starts.
- Use it when
- A browser or desktop suite runs on one worker while its tests share no files, ports or processes.
- Before applying it
- Parallel journeys are safe when each owns its state; a shared file, port or install step becomes a race. Measure the worker count on your own runner.
Give parallel test workers more databases
After the CPU change, raising the pytest-xdist worker count left the suite's time flat at first. Profiling showed the test database buckling under the load: it hit lock contention once more than 20 workers shared it. Ramp ran more test databases, measured the ratio of workers to databases, settled on about 12 workers per database, and tuned Postgres connection settings.
Together with the faster CPUs, the database changes took Ramp to a ~70% speedup of the test suite two days in, a saving of about 30 minutes. Source
- Use it when
- Tests run in parallel workers against one shared database or service, and adding workers leaves the job's time flat while the database is saturated.
- Before applying it
- Every extra database uses CPU and memory the test workers would otherwise get; Ramp saw slight regressions past its measured ratio, so keep the count that measured fastest.
Isolate tests before adding workers
The CLI end-to-end suite shared one development store. Shopify/cli#7279 gave each test its own store and each worker its own ports; #7309 then turned on fullyParallel and raised Playwright workers from one to five. The default branch runs the suite fully parallel with four workers.
The PR reports about 10 minutes with 5 workers, down from about 27 minutes with 1 worker. Source
- Use it when
- Tests run one at a time because they share ports, temporary files, accounts or database state.
- Before applying it
- More workers add load on shared external APIs; keep teardown observable and keep genuinely serial tests on a serial lane.
Give concurrent tests their own state
Stripe sync-engine gives each end-to-end test file its own Postgres container on a random port and enables file parallelism; webhook tests run in a separate sequential configuration because they share a remote endpoint. TaxJar, a Stripe company, made its remote transaction IDs unique per version and build so its WooCommerce 7.x and 9.x test lanes can run in parallel.
- Use it when
- A shared database, port or remote account forces the whole suite to run serially.
- Before applying it
- Keep intentionally shared-state tests in a serial lane, keep teardown reliable, and use isolated sandbox accounts for remote services.
Give concurrent suites their own services
SubmitQueue replaced shared manual service setup with a Compose stack per test context. Docker allocates an ephemeral host port and the test discovers it at runtime, so the suites run as separate parallel CI jobs. OPT88's resource-isolation step is adapted; the readiness and image-tag fixes on this page complete it.
The PR creates separate CI jobs for the suites and an isolated Compose project per test context. Source
- Use it when
- Independent integration suites collide on one port, database or container name, forcing serial execution.
- Before applying it
- Use collision-proof namespaces and guaranteed cleanup after every run. Separate data and image tags as well as ports, and use a protocol-level readiness probe in place of a fixed startup sleep.
Balance test work across shards
Use measured runtimes or smaller test files to reduce the slowest shard, which determines when the suite finishes.
Balance test files before buying more shards
Linear split oversized test files into smaller files, then benchmarked shard and runner configurations. Moving from four to eight shards made the critical job roughly 19% faster and 19% cheaper in the initial benchmark.
- Use it when
- A few files dominate a file-based shard assignment.
- Before applying it
- Preserve test discovery, assertions and artifacts. Vitest assigns work by file, so one large file still sets its shard’s time.
Split long test jobs into balanced shards
Codex split its single Windows Bazel test job into four shard jobs behind one aggregate check. A later change assigns targets by estimated duration, longest first, to the least-loaded shard.
Windows tests: one sharded run in about 7m29s, against a recent single-job median around 22m26s Source
- Use it when
- A test job dominates the critical path, or existing shards finish at very different times.
- Before applying it
- Balance shard duration and total runner minutes together. A checked-in timing table goes stale as new tests arrive, so refresh it or use the runner's own timing data.
Pack tests onto workers so they finish together
With the suite running in parallel pytest-xdist workers, Ramp saw the workers finish at different times, so the slowest worker set the job's length. The team wrote a more effective packing algorithm, with ChatGPT's help, as a proof of concept on a fork of pytest.
The more effective packing algorithm saved 2 to 3 minutes. Source
- Use it when
- A test job runs in parallel workers or matrix shards, and the last one regularly finishes well after the first.
- Before applying it
- A packing algorithm that uses test timings needs them recorded; refresh them as tests are added so new slow tests spread across the workers.
Balance shards and keep a catch-all for new tests
Stripe iOS balanced PaymentSheet UI tests across four jobs: three with explicit assignments and a fourth that runs everything else, including new tests. A validator rejects duplicate assignments and stale exclusions.
In Stripe’s combined CI runs, the slowest PaymentSheet UI job went from about 26 to about 12 minutes, with worker and machine changes in the same PR. Source
- Use it when
- One shard sets the finish time while others wait idle.
- Before applying it
- Keep the union of test IDs identical and track new, renamed and deleted tests.
Match test concurrency to the runner
Measure worker counts against available CPU and memory. More workers help only while the runner has capacity.
Set test worker width from the actual host
Bun removed the CPU and thread caps that held its Linux test agents to two usable cores, and sized the parallel test width from the available cores. A follow-up capped the width at four on macOS, where each M2 Ultra host runs two tart virtual machines and every coordinator forks its own workers.
- Use it when
- A fixed worker count has never been measured, or workers contend for memory, physical cores or a shared host.
- Before applying it
- Nested worker processes and other virtual machines on the same host count toward the width.
Fit test concurrency to the runner’s memory
The memory integration configuration capped Vitest threads at two after the source reported runner disconnects and memory pressure.
The source reports 61 failures in 771 runs. Source
The current Vitest configuration expresses the limit as maxWorkers: 2. Source
- Use it when
- Parallel tests and build output compete for limited runner memory.
- Before applying it
- Preserve the separate build heap requirement and measure peak memory, elapsed time and failures across worker counts.
Measure how many tests to run at once
Codex compared Windows Bazel shards at 4, 8 and 12 local test jobs in a forced-test-execution experiment, then set the CI default to 8 with one .bazelrc line. Sharding spreads work across runners; this tunes how much runs at once on each runner.
Slowest shard: 835.5s to 609.3s at 8 local test jobs, in the experiment the PR cites Source
- Use it when
- Test parallelism is a default or a guess for this runner.
- Before applying it
- More parallel workers can increase memory pressure and contention. Compare the same workload and runner, including reliability and total minutes.
Measure workers against the runner
Stripe iOS tried six simulator workers and kept four, which was faster and more reliable, and split Crypto UI tests into separate classes so Xcode could run them in parallel. Stripe Android sized individual device-heavy lanes and raised its managed-device ceiling to match its six shards. Metronome, a Stripe company, moved a publishing job whose doc build was timing out to an 8-core runner, on a suggestion from the Stainless team.
In the same combined runs, the Crypto UI test step went from about 10 to about 6 minutes. Source
- Use it when
- Tests run serially, or worker counts and runner sizes were chosen without measurement.
- Before applying it
- Isolate shared accounts and ports before adding workers, and keep tests that verify serial behavior in a serial lane. Measure each change separately; the source PRs changed workers and machines together.
Reduce repeated process and build-graph startup
Batch compatible test work or share its build graph to avoid paying process startup and initialization for every file or target.
Run reliable test files in one parallel invocation per shard
Each Bun test shard now runs its fast, historically reliable test files as one bun test --parallel batch, an invocation that distributes the files across a pool of worker processes, where each file used to start its own process. Files that have shown batch sensitivity go on a denylist and run on their own, and so do memory-measuring tests.
Per-lane test shard time for the whole change (batching, the full core count on test agents and one up-front addon build per shard), one main build against one branch build: debian x64-asan 8.5 to 4.9 minutes on average (42% lower), windows 2019 x64 8.6 to 6.0 minutes (30%), darwin 26 aarch64 15.5 to 12.6 minutes (19%). Source
- Use it when
- Process startup for each test file takes a large share of a big suite, and a reviewed set of files shares no mutable state.
- Before applying it
- Files in unlisted directories stay on the per-file path, and a new test file in a listed directory joins the batch; the batch reports every file it runs, and any error in the batch turns the job red.
Run tests in fewer processes
Codex moved every integration test under tests/ into one suite, so cargo builds and starts a single test binary. The Python SDK compiles its dependency-policy checker once per pytest worker and runs each of its policy cases in-process with fresh globals, keeping real subprocess tests for CLI parity.
Python 3.14 test job: 7m00s to 2m41s (2m24s before the policy matrix landed) Source
- Use it when
- A suite repeatedly starts interpreters, CLIs or separate test binaries.
- Before applying it
- Sharing a process can expose leaked state and environment changes. Keep tests isolated and compare results as well as runtime.
Run independent test packages concurrently
Let independent packages or test environments run at the same time instead of invoking each one serially.
Run independent test packages concurrently
Bun's Miri script builds every crate once, then runs one cargo miri test per crate, as many at a time as the runner has cores, longest crates first. It waits for all of them and exits nonzero if any crate reports an error.
The PR expects the Miri job to go from about 14.5 minutes to about 4 to 5 minutes (estimate), with the concurrent crates and a smaller Miri-sized tracker test together. Source
- Use it when
- A serial test driver runs independent packages, crates or modules one after another on a multi-core runner.
- Before applying it
- Each process keeps its own log, and the job reports every failing package.
Run the slowest test environment in parallel workers
This is a community contribution to Spotify's luigi project. Luigi's core test environment took about five minutes while the others took about a minute or less. The change adds pytest-xdist and runs the core tests with -n auto --dist=loadfile, which spreads test files across one worker per CPU and keeps each file's tests on one worker.
The author reports the core workflows running in 2 to 4 minutes, 1 to 2 minutes faster. Source
- Use it when
- One test environment takes most of the CI time and runs in a single process on a multi-core runner.
- Before applying it
- Tests that share files, ports or global state need per-worker isolation; grouping by file keeps a file's tests together.
Let the test runner schedule independent packages
After isolating its unit suite, Cadence replaced a shell loop that ran go test once per package with one invocation over the package list, so Go schedules independent packages concurrently and writes one coverage profile. OPT88's serial-suite diagnosis is adapted.
The PR reports unit and integration test runs and builds on the dependency isolation in cadence-workflow/cadence#6800. Source
- Use it when
- A shell loop runs independent packages one at a time even though the language test runner schedules packages in parallel.
- Before applying it
- Remove shared-resource dependencies first, keep special-case tests, race settings and combined coverage, and measure CPU and memory at several concurrency levels.
Keep unit tests independent of slow services
Use controlled substitutes for external dependencies when the test is intended to verify local behavior, keeping integration coverage separately.
Mock the slow dependency
OpenAI added a missing mock to a 63-test resolver suite and kept all 63 tests. In the public evals repository, the workflow validates new evaluations against dummy chat and completion models, so the check runs without a live model call.
63-test resolver suite: 592.6s to 29.8s after adding the missing mock Source
- Use it when
- A slow test calls a real dependency outside the code it exists to verify.
- Before applying it
- Keep one job that exercises the real dependency, nightly or after merge. A deterministic mock proves the code under test, and the live job proves the service.
Keep unit tests independent of database servers
Cadence removed Cassandra, MySQL, PostgreSQL and MongoDB services from its unit-test container. Component tests that only needed persistence switched to SQLite, and database-specific persistence and CLI tests moved into the integration tree, whose Makefile gained an explicit discovery pass. OPT87's boundary-isolation recipe is adapted with a local SQLite store.
The PR reports unit and integration test runs and a comparison of coverage reports. Source
- Use it when
- Unit or component checks wait for database servers whose engine-specific behavior they never exercise.
- Before applying it
- Keep real-engine integration tests and compare their discovered test sets and combined coverage, since unit-only coverage shifts when tests move. The server-startup test also drops a Cassandra schema-version assertion; keep an equivalent engine-specific check in the integration lane.
Shorten deliberate test waits
Reduce waits whose timing requirements are known, preserving the timeout or consistency behavior the test is intended to cover.
Shorten deliberate sleeps in tests and examples
The Claude Agent SDK's error-handling example, which CI runs on every pull request, asked for a 60-second sleep to trigger a 10-second timeout. The change asks for 5 seconds and times out at 3, so the timeout path still runs.
The PR's estimate is that the shorter sleep removes 50+ seconds from the examples job; the same PR also raises that step's timeout. Source
- Use it when
- A test or a CI-run example sleeps for a fixed time only to exercise a timeout or a slow path.
- Before applying it
- The sleep and the timeout it triggers shrink together, so the timeout branch still runs and is asserted.
Shorten only waits whose consistency is already proved
Chroma reduced 80 fixed 2-second waits, from three call sites, to 200 milliseconds; Couchbase reduced per-operation waits while adding a service readiness probe.
Chroma’s PR estimates the Run Chroma tests step drops from 160 seconds of sleeping to about 16 seconds. Source
- Use it when
- A service exposes no usable readiness predicate and repeated conservative waits dominate tests.
- Before applying it
- Prefer a real completion signal; retain coverage and stress slow runners before choosing a lower bound.
Wait for observable readiness instead of sleeping
Poll the service or state the test actually needs, with a bounded timeout and useful failure output.
Poll the condition the test actually needs
MongoDB vector tests replaced fixed sleeps with polling. Couchbase used its Search service ping for startup.
- Use it when
- Repeated waits dominate an eventual-consistency integration suite.
- Before applying it
- Validate content or version readiness and fail clearly on a deadline; a matching count alone can miss same-size updates.
Wait for readiness with a bounded poll
Stripe sync-engine replaced fixed sleeps with a waitFor helper that polls a condition, returns as soon as it holds and throws when the deadline passes. Stripe Android’s UI test waits for a snackbar to hide only when it overlaps the element it needs to tap.
- Use it when
- Tests sleep for a fixed period before checking an asynchronous result.
- Before applying it
- Probe the capability the next step needs; an open port can come before migrations or workers are ready. Keep tests that intentionally verify timeouts unchanged.
Build the test database once for all workers
Have one parallel worker apply the schema while the others wait for it, so more workers stop repeating the same setup.
Build the test database once for all workers
Ramp's test setup recreated the database from scratch by applying every migration, and each pytest-xdist worker ran that setup in parallel. At higher worker counts the concurrent schema changes ran Postgres out of working memory and repeated the same work. Ramp restructured the setup so one worker applies the schema, which let it run more workers on a machine twice the size.
Coordinating the schema setup let Ramp double the machine size and halve CI test times, which had been creeping up towards 20 minutes. Source
- Use it when
- Tests run in parallel workers that each repeat the same database or fixture setup, and raising the worker count stops paying off.
- Before applying it
- The other workers wait for the one that builds the schema, so the setup itself still sets a floor on the job's time.
Complete the fixture before its timeout fires
Exercise the event that advances the test, so it stops waiting on the timer that rescues it.
Complete the fixture before its timeout fires
Cadence integration tests polled the foreign workflow's activity, so the next decision waited for the original activity to time out. The fix completes the original workflow's activity and updates expected event IDs for the extra ActivityStarted event. OPT19 is adapted to an accidental activity timeout.
The author identifies a 10-second wait in each affected test. Source
- Use it when
- Tests regularly spend a fixed interval waiting for an operation the fixture could complete immediately.
- Before applying it
- Confirm the test is about the completion path, keep dedicated timeout tests, and update event histories when completion adds events.
Let an agent break import cycles and make the app load lazily
Point a coding agent at the import cycles and eager imports behind a slow app boot, and merge what it lands.
Let an agent break import cycles and make the app load lazily
Ramp's engineer in the August 2026 talk described telling the agent to fix all the app's import cycles and to make the app lazy, because the app boots up an enormous amount of Python modules. The agent made a lot of progress on both, and a lot of that code was merged. The talk presents this among the hard problems Ramp gave the agent.
2026: Ramp reported that a lot of the agent's import-cycle and lazy-loading code was merged. Source
- Use it when
- Tests and tools start the application, and its startup imports far more modules than a single test or command uses.
- Before applying it
- Lazy imports move a failure from startup to the first use, so keep a test that imports every module.
Move unit tests to a parallel-by-default framework, gated on ten clean runs
Migrate serial tests to a framework that runs them in parallel, and ship each package after ten green runs.
Move unit tests to a parallel-by-default framework, gated on ten clean runs
Mostly serialized XCTest runs had stretched DoorDash's iOS CI test step into double-digit minutes. DoorDash migrated its shared iOS packages to Swift Testing, which runs synchronous and asynchronous tests in parallel by default. A Cursor rule made the mechanical edits, and the migration prompt told the agent to run each package's tests with Bazel after every file and fix failures before moving on. A package counted as migrated once its tests passed ten times in a row, which surfaced tests that depended on order, shared state or timing. The unit test step in CI got about 60% faster, and overall builds for shared iOS packages about 40% faster.
The unit test step in CI became roughly 60% faster. Source
This contributed to an average of about 40% faster overall builds for shared iOS packages. Source
- Use it when
- A test step runs mostly serially, and the test framework or a newer one can run the same tests in parallel.
- Before applying it
- Parallel runs expose hidden order, shared-state and timing assumptions; fix each root cause, and leave tests on the old framework where the new one lacks a needed feature.
Reduce the setup floor before increasing shard count
Each extra shard repeats checkout, dependencies and container setup.
Reduce the setup floor before increasing shard count
Each extra shard repeats checkout, dependencies and container setup. Before its setup work, Linear’s test job used 4 shards and spent 8.3 minutes on setup; afterward it ran 8 shards with 7.5 minutes of setup.
- Use it when
- More shards would repeat substantial fixed setup work.
- Before applying it
- Remeasure per-shard setup before changing shard count.
Split independent work off the longest job
Run independent suites in parallel jobs when the overlap saves more than the repeated setup costs.
Split independent work off the longest job
Stripe iOS split installation and integration tests into separate jobs, keeping all five installation variants together so they share setup. Earlier, it moved one measured slow test class into its own job.
The combined job took about 27 minutes; after the split, installation took about 11 to 12 minutes and integration about 13 minutes, running in parallel, in Stripe’s combined CI runs. Source
- Use it when
- Two independent suites run one after another in the job that holds up the pull request.
- Before applying it
- Each new job repeats setup and may queue. Keep every test variant and simulator version, and keep one stable required check that reports every job’s result.
Test supported runtimes concurrently
Give each target framework its own job that installs only what it needs.
Test supported runtimes concurrently
Stripe .NET replaced one sequential build-and-test job with per-framework matrix jobs and a native Windows .NET Framework lane. Each lane installs the latest SDK plus only its target runtime, and example builds run in their own job.
- Use it when
- A multi-runtime suite runs sequentially or installs every toolchain in every lane.
- Before applying it
- Keep every supported runtime gated, let every diagnostic matrix leg run to completion, and update branch protection for the new job names.
Company Playbooks
- Read Anthropic’s playbookRun only affected tests
- Read DoorDash’s playbookRun build and test in parallel, sharded across CI agents
- Read Linear’s playbookBenchmark the same job on faster hardware
- Read Mastra’s playbookStarSling runners for the slowest test workflow
- Read OpenAI’s playbookRank CI fixes by the minutes they save
- Read Ramp’s playbookMove a CPU-bound test suite to a faster CPU
- Read Shopify’s playbookLet an agent iterate against one CI metric
- Read Spotify’s playbookBuild arm64 on native arm64 runners
- Read Stripe’s playbookRun only the tests a change can reach
- Read Uber’s playbookBuild the merge-queue speculations most likely to be needed