--- title: "Let an agent iterate against one CI metric" description: "Pick one measurable CI number, such as a build time, and let an agent keep each change that beats the baseline and discard each one that crashes or runs slower." url: https://starsling.dev/ci-playbooks/techniques/agent-metric-loop canonicalUrl: https://starsling.dev/ci-playbooks/techniques/agent-metric-loop --- # Let an agent iterate against one CI metric Pick one measurable CI number, such as a build time, and let an agent keep each change that beats the baseline and discard each one that crashes or runs slower. ## How to choose an approach This technique currently has one documented company example. Validate its prerequisites, tradeoffs and original CI environment against your repository before adapting it. - **[Shopify](https://starsling.dev/ci-playbooks/shopify#agent-metric-loop)**: Use it when One CI step has a number that shows whether it got faster, and each change to it can be tested on its own. Check before adopting: A faster number only counts when the build still produces the same output, so read the diff of every kept change and discard the ones that remove files. Adapted for GitHub Actions; compare the original system with yours. ## Apply this technique ````text Evaluate and apply this CI technique to my repository: Let an agent iterate against one CI metric. Pick one measurable CI number, such as a build time, and let an agent keep each change that beats the baseline and discard each one that crashes or runs slower. Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/agent-metric-loop.md Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence. Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured. Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo. Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES. Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch. Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge. ```` ## How Shopify used this technique Read what Shopify changed and follow the original sources. ### Shopify: Let an agent iterate against one CI metric **Translated to GitHub Actions** - What Shopify built: On Shopify's Polaris team, an extension runs an agent in a loop on a single metric, build time. - Your GitHub Actions version: A profile-first loop where an agent forms a hypothesis, tests it and measures every change against a baseline. On Shopify's Polaris team, David Cortés wrote an extension that runs an agent in a loop on a single metric. The metric was build time, which every CI pipeline depended on, with a baseline of 19.1 seconds. Each iteration forms a hypothesis, writes it down and tests it. The loop found that the build for visual regression tests ran the full component pipeline before Storybook recompiled from source, and that the TypeScript transform processed all 580 component files when only 105 needed it. Shopify's post also says autoresearch, which people across the company use, made pnpm run faster, and links pnpm/pnpm#11073, an upstream pnpm pull request that the post credits to the loop and that skips redundant internal linking on a warm reinstall. The OPT91 recipe profiles a long compile before changing it; here a loop measures every change against the baseline. Evidence: Company-reported result or observation 2026: On the Polaris build, from a 19.1 second baseline, the loop made the build 65% faster. ([Source](https://shopify.engineering/autoresearch)) Use it when: One CI step has a number that shows whether it got faster, and each change to it can be tested on its own. Before applying it: A faster number only counts when the build still produces the same output, so read the diff of every kept change and discard the ones that remove files. GitHub Actions adaptation: Compare the source system with your own workflow before applying this translation. [Read more in the Shopify playbook](https://starsling.dev/ci-playbooks/shopify#agent-metric-loop) [Primary source](https://shopify.engineering/autoresearch) ## CI optimization is continuous work. Apply this technique yourself, or use StarSling Runners for faster GitHub Actions and optional agent-driven optimization PRs, which teams on eligible paid plans can turn on. [Get self-improving CI](https://starsling.dev/products/runners) ## Related techniques ### [Link release builds with a faster linker](https://starsling.dev/ci-playbooks/techniques/faster-release-linker) When linking dominates the build, switch to a faster linker. 1 implementation · 1 company (OpenAI) ### [Measure the cost of the release build profile](https://starsling.dev/ci-playbooks/techniques/profile-release-linking) Measure what the release build profile costs before keeping it. 1 implementation · 1 company (OpenAI) ### [Use build profiles to remove expensive work](https://starsling.dev/ci-playbooks/techniques/profile-build-hotspots) Measure where build time goes, then remove or replace the compiler work or dependencies responsible for the bottleneck. 2 implementations · 2 companies (OpenAI, Stripe) ## Company Playbooks ### [Anthropic](https://starsling.dev/ci-playbooks/anthropic) Run only affected tests ### [DoorDash](https://starsling.dev/ci-playbooks/doordash) Run build and test in parallel, sharded across CI agents ### [Linear](https://starsling.dev/ci-playbooks/linear) Benchmark the same job on faster hardware ### [Mastra](https://starsling.dev/ci-playbooks/mastra) StarSling runners for the slowest test workflow ### [OpenAI](https://starsling.dev/ci-playbooks/openai) Rank CI fixes by the minutes they save ### [Ramp](https://starsling.dev/ci-playbooks/ramp) Move a CPU-bound test suite to a faster CPU ### [Shopify](https://starsling.dev/ci-playbooks/shopify) Let an agent iterate against one CI metric ### [Spotify](https://starsling.dev/ci-playbooks/spotify) Build arm64 on native arm64 runners ### [Stripe](https://starsling.dev/ci-playbooks/stripe) Run only the tests a change can reach ### [Uber](https://starsling.dev/ci-playbooks/uber) Build the merge-queue speculations most likely to be needed ## Related guides Implementation guides for this technique. ### [The agent loop that shipped a 5.4x speedup](https://starsling.dev/blog/agent-loop) The job was green. The logs were clean. CI Harness v2 found the 11-month-old bottleneck anyway, and made that job 5.4x faster.