Let an agent iterate against one CI metric
Pick one measurable CI number, such as a build time, and let an agent keep each change that beats the baseline and discard each one that crashes or runs slower.
How Shopify used this technique
Read what Shopify changed and follow the original sources.
Let an agent iterate against one CI metric
Translated to GitHub Actions
- What Shopify built
- On Shopify's Polaris team, an extension runs an agent in a loop on a single metric, build time.
- Your GitHub Actions version
- A profile-first loop where an agent forms a hypothesis, tests it and measures every change against a baseline.
On Shopify's Polaris team, David Cortés wrote an extension that runs an agent in a loop on a single metric. The metric was build time, which every CI pipeline depended on, with a baseline of 19.1 seconds. Each iteration forms a hypothesis, writes it down and tests it. The loop found that the build for visual regression tests ran the full component pipeline before Storybook recompiled from source, and that the TypeScript transform processed all 580 component files when only 105 needed it. Shopify's post also says autoresearch, which people across the company use, made pnpm run faster, and links pnpm/pnpm#11073, an upstream pnpm pull request that the post credits to the loop and that skips redundant internal linking on a warm reinstall. The OPT91 recipe profiles a long compile before changing it; here a loop measures every change against the baseline.
2026: On the Polaris build, from a 19.1 second baseline, the loop made the build 65% faster. Source
- Use it when
- One CI step has a number that shows whether it got faster, and each change to it can be tested on its own.
- Before applying it
- A faster number only counts when the build still produces the same output, so read the diff of every kept change and discard the ones that remove files.
- GitHub Actions adaptation
- Compare the source system with your own workflow before applying this translation.
Company Playbooks
- Read Anthropic’s playbookRun only affected tests
- Read DoorDash’s playbookRun build and test in parallel, sharded across CI agents
- Read Linear’s playbookBenchmark the same job on faster hardware
- Read Mastra’s playbookStarSling runners for the slowest test workflow
- Read OpenAI’s playbookRank CI fixes by the minutes they save
- Read Ramp’s playbookMove a CPU-bound test suite to a faster CPU
- Read Shopify’s playbookLet an agent iterate against one CI metric
- Read Spotify’s playbookBuild arm64 on native arm64 runners
- Read Stripe’s playbookRun only the tests a change can reach
- Read Uber’s playbookBuild the merge-queue speculations most likely to be needed