--- title: "Queue and runner speed" description: "Separate time waiting for a runner from time executing on it. Compare runner changes and capacity choices against the same workload." url: https://starsling.dev/ci-playbooks/categories/runners canonicalUrl: https://starsling.dev/ci-playbooks/categories/runners --- # Queue and runner speed Separate time waiting for a runner from time executing on it. Compare runner changes and capacity choices against the same workload. ## Where to start Diagnose first: are jobs waiting for capacity, or executing slowly once assigned a runner? Queue time and execution time have different causes. Measure when a run was triggered, when its jobs started, and which resources those jobs use. Benchmark equivalent commits and cache states before choosing a runner or changing capacity. ## Apply these techniques ````text Evaluate this CI improvement area in my repository: Queue and runner speed. Separate time waiting for a runner from time executing on it. Compare runner changes and capacity choices against the same workload. Read the techniques and source-backed company examples: https://starsling.dev/ci-playbooks/categories/runners.md Inspect this repository’s CI workflows, recent run timings, build and test commands, and dependency graph. Identify the relevant bottleneck and rank applicable techniques using evidence from this repository. Measure a baseline and compare the relevant techniques, prerequisites, company context, and primary sources. Rank candidate changes by fit and observable opportunity. Recommend the smallest useful change and the verification plan. ASK ME TO APPROVE THE EXACT CHANGE BEFORE EDITING ANY FILES. Only after approval, implement on a new feature branch and open a reviewable pull request. Never merge or push to a default or protected branch. Preserve required checks, test coverage, release and deployment behavior, credential boundaries, and necessary fallback paths. Run relevant checks and compare equivalent runs for elapsed time, queue time, runner minutes, and reliability. Report any unmeasured outcomes. ```` ## Techniques 3 techniques with source-backed company examples. ### [Move bottlenecked jobs to faster runners](https://starsling.dev/ci-playbooks/techniques/faster-runners) Benchmark the same workload on faster hardware and compare elapsed time and cost before changing runner capacity. #### How to choose an approach Compare the examples by repository setup, prerequisites, and risks before adopting one. Each company implemented this in its own environment; measure each change against your own baseline. - **[Linear](https://starsling.dev/ci-playbooks/linear#faster-runners)**: Use it when Build or test execution still dominates after cheaper setup changes. Check before adopting: Match operating system, architecture, workload and event mix. Compare elapsed time, queue time and runner cost; StarSling sells a runner option worth benchmarking. - **[Mastra](https://starsling.dev/ci-playbooks/mastra#starsling-runners)**: Use it when One workflow sets the critical path and its jobs run on standard hosted runners. Check before adopting: Swap one workflow first and compare wall-clock p95 over matched windows before moving the rest. - **[OpenAI](https://starsling.dev/ci-playbooks/openai#faster-runner)**: Use it when The critical-path job is compute-bound and cheaper setup, cache and test fixes are exhausted. Check before adopting: Benchmark comparable workloads and include runner cost. Fork or event restrictions may require the existing label for some runs. - **[Ramp](https://starsling.dev/ci-playbooks/ramp#faster-cpu)**: Use it when Profiling the machine that runs the slowest test job shows the CPU saturated for most of the run. Check before adopting: A faster runner class bills at a different rate, so compare runner minutes and cost per run alongside elapsed time. #### Linear: Benchmark the same job on faster hardware Linear moved the same pipeline to runners with faster CPUs, higher-performance storage and better cache infrastructure. Across the two days either side of the switch, jobs ran 34% faster on average. Evidence: Documented technique Use it when: Build or test execution still dominates after cheaper setup changes. Before applying it: Match operating system, architecture, workload and event mix. Compare elapsed time, queue time and runner cost; StarSling sells a runner option worth benchmarking. [Read more in the Linear playbook](https://starsling.dev/ci-playbooks/linear#faster-runners) [Primary source](https://linear.app/now/ci-bottleneck-reworked#upgraded-infrastructure-and-tooling) #### Mastra: Move the slowest test workflow onto StarSling runners StarSling’s agent moved every job in Mastra’s Combined store Tests workflow onto StarSling runners. Follow-up PRs moved the memory, E2E, workspace, prebuild and changed-test workflows the same way. Evidence: StarSling-measured workflow cohort result StarSling measured Combined store Tests at 5.87x faster wall-clock p95 after the runner swap and the agents’ test fixes: 29m 56s to 5m 06s. Successful push and same-repository PR runs across all branches: Feb 21 to 24 before migration versus Apr 1 to 4, 2026 after. The after window was the highest-ranked of seven candidate windows. ([Source](https://starsling.dev/customers/mastra)) Use it when: One workflow sets the critical path and its jobs run on standard hosted runners. Before applying it: Swap one workflow first and compare wall-clock p95 over matched windows before moving the rest. [Read more in the Mastra playbook](https://starsling.dev/ci-playbooks/mastra#starsling-runners) [Primary source](https://github.com/mastra-ai/mastra/pull/13466) #### OpenAI: Put the slowest job on a faster runner Codex moved its Windows Bazel jobs, often the long pole for PR checks, from the default hosted runners to a dedicated runner group. Evidence: Source PR/commit evidence Windows Bazel jobs moved to a dedicated runner group ([Source](https://github.com/openai/codex/pull/24952)) Use it when: The critical-path job is compute-bound and cheaper setup, cache and test fixes are exhausted. Before applying it: Benchmark comparable workloads and include runner cost. Fork or event restrictions may require the existing label for some runs. [Read more in the OpenAI playbook](https://starsling.dev/ci-playbooks/openai#faster-runner) [Primary source](https://github.com/openai/codex/pull/24952) #### Ramp: Move a CPU-bound test suite to a faster CPU In July 2023 Ramp's Python monolith test suite took 45 minutes, about 38 of them in the pytest invocation. The team watched the resource usage of a machine running the suite and found it CPU bound. Moving the suite to a faster CPU was the first change they made and the single largest speedup of the project. Evidence: Company-reported result or observation 2023: The faster CPU made the tests run about 50% faster, the first step of the week that took the suite from 45 minutes to under 5. ([Source](https://builders.ramp.com/post/faster-python-tests-part-two)) Use it when: Profiling the machine that runs the slowest test job shows the CPU saturated for most of the run. Before applying it: A faster runner class bills at a different rate, so compare runner minutes and cost per run alongside elapsed time. [Read more in the Ramp playbook](https://starsling.dev/ci-playbooks/ramp#faster-cpu) [Primary source](https://builders.ramp.com/post/faster-python-tests-part-two) ### [Build each architecture on native runners](https://starsling.dev/ci-playbooks/techniques/native-architecture-builds) Run architecture-specific builds on matching hardware to avoid the overhead of emulation. #### How to choose an approach Compare the examples by repository setup, prerequisites, and risks before adopting one. Each company implemented this in its own environment; measure each change against your own baseline. - **[Spotify](https://starsling.dev/ci-playbooks/spotify#native-arm-runners)**: Use it when A workflow builds arm64 images or binaries on x86 runners through emulation, and native arm64 runners are available to it. Check before adopting: Each architecture gets its own job and pays its own setup, and a final job merges the per-platform digests into one manifest. - **[Stripe](https://starsling.dev/ci-playbooks/stripe#native-container-builds)**: Use it when Cross-architecture emulation dominates a multi-platform image build. Check before adopting: Publish the multi-platform manifest only after every platform build and check succeeds, and keep every supported platform. #### Spotify: Build arm64 on native arm64 runners Confidence's release job built its deployer image for amd64 and arm64 on one x86 runner, compiling the arm64 Rust code under QEMU emulation. The change runs one matrix job per platform, amd64 on ubuntu-latest and arm64 on ubuntu-24.04-arm, pushes each image by digest, and adds a job that combines the digests into a single multi-arch manifest. The QEMU setup step is gone. Evidence: Single-run demonstration The last release run under emulation took 2 hours 28 minutes; a test workflow mirroring the change finished in about 17 minutes of wall-clock time, which the author reports as a ~8.7x speedup. ([Source](https://github.com/spotify/confidence-resolver/pull/479)) Use it when: A workflow builds arm64 images or binaries on x86 runners through emulation, and native arm64 runners are available to it. Before applying it: Each architecture gets its own job and pays its own setup, and a final job merges the per-platform digests into one manifest. [Read more in the Spotify playbook](https://starsling.dev/ci-playbooks/spotify#native-arm-runners) [Primary source](https://github.com/spotify/confidence-resolver/pull/479) #### Stripe: Build each container architecture natively Stripe sync-engine split its Docker build into parallel arm64 and amd64 jobs on matching native GitHub runners, replacing an emulated multi-platform build, and a manifest job combines both with docker buildx imagetools. Evidence: Source PR/commit evidence Use it when: Cross-architecture emulation dominates a multi-platform image build. Before applying it: Publish the multi-platform manifest only after every platform build and check succeeds, and keep every supported platform. [Read more in the Stripe playbook](https://starsling.dev/ci-playbooks/stripe#native-container-builds) [Primary source](https://github.com/stripe/sync-engine-fork/pull/212) ### [Match test concurrency to the runner](https://starsling.dev/ci-playbooks/techniques/tune-test-workers) Measure worker counts against available CPU and memory. More workers help only while the runner has capacity. #### How to choose an approach This technique currently has one documented company example. Validate its prerequisites, tradeoffs and original CI environment against your repository before adapting it. - **[Stripe](https://starsling.dev/ci-playbooks/stripe#measure-worker-count)**: Use it when Tests run serially, or worker counts and runner sizes were chosen without measurement. Check before adopting: Isolate shared accounts and ports before adding workers, and keep tests that verify serial behavior in a serial lane. Measure each change separately; the source PRs changed workers and machines together. #### Stripe: Measure workers against the runner Stripe iOS tried six simulator workers and kept four, which was faster and more reliable, and split Crypto UI tests into separate classes so Xcode could run them in parallel. Stripe Android sized individual device-heavy lanes and raised its managed-device ceiling to match its six shards. Metronome, a Stripe company, moved a publishing job whose doc build was timing out to an 8-core runner, on a suggestion from the Stainless team. Evidence: Source PR/commit evidence In the same combined runs, the Crypto UI test step went from about 10 to about 6 minutes. ([Source](https://github.com/stripe/stripe-ios/pull/7333)) Use it when: Tests run serially, or worker counts and runner sizes were chosen without measurement. Before applying it: Isolate shared accounts and ports before adding workers, and keep tests that verify serial behavior in a serial lane. Measure each change separately; the source PRs changed workers and machines together. [Read more in the Stripe playbook](https://starsling.dev/ci-playbooks/stripe#measure-worker-count) [Primary source](https://github.com/stripe/stripe-ios/pull/7333) ## CI optimization is continuous work. Apply these techniques yourself, or use StarSling Runners for faster GitHub Actions and optional agent-driven optimization PRs, which teams on eligible paid plans can turn on. [Get self-improving CI](https://starsling.dev/products/runners) ## Company Playbooks ### [Anthropic](https://starsling.dev/ci-playbooks/anthropic) Run only affected tests ### [DoorDash](https://starsling.dev/ci-playbooks/doordash) Run build and test in parallel, sharded across CI agents ### [Linear](https://starsling.dev/ci-playbooks/linear) Benchmark the same job on faster hardware ### [Mastra](https://starsling.dev/ci-playbooks/mastra) StarSling runners for the slowest test workflow ### [OpenAI](https://starsling.dev/ci-playbooks/openai) Rank CI fixes by the minutes they save ### [Ramp](https://starsling.dev/ci-playbooks/ramp) Move a CPU-bound test suite to a faster CPU ### [Shopify](https://starsling.dev/ci-playbooks/shopify) Let an agent iterate against one CI metric ### [Spotify](https://starsling.dev/ci-playbooks/spotify) Build arm64 on native arm64 runners ### [Stripe](https://starsling.dev/ci-playbooks/stripe) Run only the tests a change can reach ### [Uber](https://starsling.dev/ci-playbooks/uber) Build the merge-queue speculations most likely to be needed ## Related guides Implementation guides for the approaches in this category. ### [Cut CI queue time](https://starsling.dev/best-practices/github-actions/cut-queue-time) Measure wait-to-start time and examine concurrency groups and available capacity. ### [Right-size GitHub Actions runners](https://starsling.dev/best-practices/github-actions/right-size-runners) Compare runner sizes on the same workload and account for elapsed time and cost. ### [Diagnose high CI queue time](https://starsling.dev/github-actions/problems/high-queue-time) Check concurrency, cancellation, runner capacity, and triggers against run history.