Queue and runner speed

Separate time waiting for a runner from time executing on it. Compare runner changes and capacity choices against the same workload.

View in Markdown

Prompt

Evaluate this CI improvement area in my repository: Queue and runner speed.

Separate time waiting for a runner from time executing on it. Compare runner changes and capacity choices against the same workload.

Read the techniques and source-backed company examples: https://starsling.dev/ci-playbooks/categories/runners.md

Inspect this repository’s CI workflows, recent run timings, build and test commands, and dependency graph. Identify the relevant bottleneck and rank applicable techniques using evidence from this repository.
Measure a baseline and compare the relevant techniques, prerequisites, company context, and primary sources. Rank candidate changes by fit and observable opportunity.
Recommend the smallest useful change and the verification plan. ASK ME TO APPROVE THE EXACT CHANGE BEFORE EDITING ANY FILES.
Only after approval, implement on a new feature branch and open a reviewable pull request. Never merge or push to a default or protected branch.
Preserve required checks, test coverage, release and deployment behavior, credential boundaries, and necessary fallback paths. Run relevant checks and compare equivalent runs for elapsed time, queue time, runner minutes, and reliability. Report any unmeasured outcomes.

View all playbooks

Where to start

Diagnose first: are jobs waiting for capacity, or executing slowly once assigned a runner?

Queue time and execution time have different causes. Measure when a run was triggered, when its jobs started, and which resources those jobs use. Benchmark equivalent commits and cache states before choosing a runner or changing capacity.

3 techniques with source-backed company examples.

CI technique

Move bottlenecked jobs to faster runners

Benchmark the same workload on faster hardware and compare elapsed time and cost before changing runner capacity.

Benchmark the same job on faster hardware

Linear moved the same pipeline to runners with faster CPUs, higher-performance storage and better cache infrastructure. Across the two days either side of the switch, jobs ran 34% faster on average.

Use it when: Build or test execution still dominates after cheaper setup changes.
Before applying it: Match operating system, architecture, workload and event mix. Compare elapsed time, queue time and runner cost; StarSling sells a runner option worth benchmarking.

Move the slowest test workflow onto StarSling runners

StarSling’s agent moved every job in Mastra’s Combined store Tests workflow onto StarSling runners. Follow-up PRs moved the memory, E2E, workspace, prebuild and changed-test workflows the same way.

StarSling measured Combined store Tests at 5.87x faster wall-clock p95 after the runner swap and the agents’ test fixes: 29m 56s to 5m 06s. Successful push and same-repository PR runs across all branches: Feb 21 to 24 before migration versus Apr 1 to 4, 2026 after. The after window was the highest-ranked of seven candidate windows. Source

Use it when: One workflow sets the critical path and its jobs run on standard hosted runners.
Before applying it: Swap one workflow first and compare wall-clock p95 over matched windows before moving the rest.

Put the slowest job on a faster runner

Codex moved its Windows Bazel jobs, often the long pole for PR checks, from the default hosted runners to a dedicated runner group.

Windows Bazel jobs moved to a dedicated runner group Source

Use it when: The critical-path job is compute-bound and cheaper setup, cache and test fixes are exhausted.
Before applying it: Benchmark comparable workloads and include runner cost. Fork or event restrictions may require the existing label for some runs.

Move a CPU-bound test suite to a faster CPU

In July 2023 Ramp's Python monolith test suite took 45 minutes, about 38 of them in the pytest invocation. The team watched the resource usage of a machine running the suite and found it CPU bound. Moving the suite to a faster CPU was the first change they made and the single largest speedup of the project.

2023: The faster CPU made the tests run about 50% faster, the first step of the week that took the suite from 45 minutes to under 5. Source

Use it when: Profiling the machine that runs the slowest test job shows the CPU saturated for most of the run.
Before applying it: A faster runner class bills at a different rate, so compare runner minutes and cost per run alongside elapsed time.

CI technique

Build each architecture on native runners

Run architecture-specific builds on matching hardware to avoid the overhead of emulation.

Build arm64 on native arm64 runners

Confidence's release job built its deployer image for amd64 and arm64 on one x86 runner, compiling the arm64 Rust code under QEMU emulation. The change runs one matrix job per platform, amd64 on ubuntu-latest and arm64 on ubuntu-24.04-arm, pushes each image by digest, and adds a job that combines the digests into a single multi-arch manifest. The QEMU setup step is gone.

The last release run under emulation took 2 hours 28 minutes; a test workflow mirroring the change finished in about 17 minutes of wall-clock time, which the author reports as a ~8.7x speedup. Source

Use it when: A workflow builds arm64 images or binaries on x86 runners through emulation, and native arm64 runners are available to it.
Before applying it: Each architecture gets its own job and pays its own setup, and a final job merges the per-platform digests into one manifest.

Build each container architecture natively

Stripe sync-engine split its Docker build into parallel arm64 and amd64 jobs on matching native GitHub runners, replacing an emulated multi-platform build, and a manifest job combines both with docker buildx imagetools.

Use it when: Cross-architecture emulation dominates a multi-platform image build.
Before applying it: Publish the multi-platform manifest only after every platform build and check succeeds, and keep every supported platform.

CI technique

Match test concurrency to the runner

Measure worker counts against available CPU and memory. More workers help only while the runner has capacity.

Measure workers against the runner

Stripe iOS tried six simulator workers and kept four, which was faster and more reliable, and split Crypto UI tests into separate classes so Xcode could run them in parallel. Stripe Android sized individual device-heavy lanes and raised its managed-device ceiling to match its six shards. Metronome, a Stripe company, moved a publishing job whose doc build was timing out to an 8-core runner, on a suggestion from the Stainless team.

In the same combined runs, the Crypto UI test step went from about 10 to about 6 minutes. Source

Use it when: Tests run serially, or worker counts and runner sizes were chosen without measurement.
Before applying it: Isolate shared accounts and ports before adding workers, and keep tests that verify serial behavior in a serial lane. Measure each change separately; the source PRs changed workers and machines together.

Where to start

Benchmark the same job on faster hardware

Move the slowest test workflow onto StarSling runners

Put the slowest job on a faster runner

Move a CPU-bound test suite to a faster CPU

Build arm64 on native arm64 runners

Build each container architecture natively

Measure workers against the runner

Company Playbooks

Related guides

Cut CI queue time

Right-size GitHub Actions runners

Diagnose high CI queue time