- All playbooks
- Slow tests
- Wait for observable readiness instead of sleeping
Wait for observable readiness instead of sleeping
Poll the service or state the test actually needs, with a bounded timeout and useful failure output.
How companies used this technique
Read what each company changed and follow the original sources.
Poll the condition the test actually needs
MongoDB vector tests replaced fixed sleeps with polling. Couchbase used its Search service ping for startup.
- Use it when
- Repeated waits dominate an eventual-consistency integration suite.
- Before applying it
- Validate content or version readiness and fail clearly on a deadline; a matching count alone can miss same-size updates.
Wait until the database answers a query
StarSling’s agent replaced MSSQL’s TCP probe with sqlcmd SELECT 1 and let Compose --wait coordinate readiness. ClickHouse received a native query healthcheck too. This adapts the fixed-sleep recipe to semantic health checks and redundant polling.
- Use it when
- Integration tests race database startup or wait through conservative startup delays.
- Before applying it
- Probe the actual operation required by tests and fail on timeout. Follow-up #15895 awaited asynchronous deletions exposed by faster storage.
Wait for readiness with a bounded poll
Stripe sync-engine replaced fixed sleeps with a waitFor helper that polls a condition, returns as soon as it holds and throws when the deadline passes. Stripe Android’s UI test waits for a snackbar to hide only when it overlaps the element it needs to tap.
- Use it when
- Tests sleep for a fixed period before checking an asynchronous result.
- Before applying it
- Probe the capability the next step needs; an open port can come before migrations or workers are ready. Keep tests that intentionally verify timeouts unchanged.
Wait for the connection the test actually uses
SubmitQueue's MySQL healthcheck used localhost, which selected a Unix socket, while its clients connected over TCP and could start too early. The fix probes 127.0.0.1 and removes the second retry loop from test setup. OPT17 is adapted to a readiness probe that checked the wrong interface.
The author reproduced the local startup errors, then reported four consecutive orchestrator passes and three gateway passes after the fix. Source
- Use it when
- Containers report healthy before the network interface used by the tests accepts connections.
- Before applying it
- Probe the same protocol, authentication and readiness condition the consumer uses, keep a bounded startup timeout with logs, and keep any retries until the new probe is proven.
Company Playbooks
- Read Anthropic’s playbookRun only affected tests
- Read DoorDash’s playbookRun build and test in parallel, sharded across CI agents
- Read Linear’s playbookBenchmark the same job on faster hardware
- Read Mastra’s playbookStarSling runners for the slowest test workflow
- Read OpenAI’s playbookRank CI fixes by the minutes they save
- Read Ramp’s playbookMove a CPU-bound test suite to a faster CPU
- Read Shopify’s playbookLet an agent iterate against one CI metric
- Read Spotify’s playbookBuild arm64 on native arm64 runners
- Read Stripe’s playbookRun only the tests a change can reach
- Read Uber’s playbookBuild the merge-queue speculations most likely to be needed