--- title: "Replay fixed evaluations before merging agent changes" description: "Run repeatable scenarios or a golden dataset on changes to agent behavior so regressions are visible before merge." url: https://starsling.dev/ci-playbooks/techniques/gate-agent-evaluations canonicalUrl: https://starsling.dev/ci-playbooks/techniques/gate-agent-evaluations --- # Replay fixed evaluations before merging agent changes Run repeatable scenarios or a golden dataset on changes to agent behavior so regressions are visible before merge. ## How to choose an approach Compare the examples by repository setup, prerequisites, and risks before adopting one. Each company implemented this in its own environment; measure each change against your own baseline. - **[DoorDash](https://starsling.dev/ci-playbooks/doordash#simulated-eval-gate)**: Use it when Changes to prompts, models or agent code ship after a manual check, or after an eval run too slow to run on every change. Check before adopting: Fixtures freeze the world the agent sees; refresh them from production traces so the gate keeps testing current behavior. Adapted for GitHub Actions; compare the original system with yours. - **[Stripe](https://starsling.dev/ci-playbooks/stripe#golden-dataset-gate)**: Use it when A service's output depends on many interacting rules, and unit tests cover single rules but not realistic combinations. Check before adopting: Keep unit tests for small, hand-built cases. Reduce each input to the fields the behavior needs, and replace sensitive identifiers with stable test identifiers. ## Apply this technique ````text Evaluate and apply this CI technique to my repository: Replay fixed evaluations before merging agent changes. Run repeatable scenarios or a golden dataset on changes to agent behavior so regressions are visible before merge. Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/gate-agent-evaluations.md Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence. Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured. Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo. Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES. Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch. Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge. ```` ## How companies used this technique Read what each company changed and follow the original sources. ### DoorDash: Gate agent changes on a simulated, fixture-backed eval run **Translated to GitHub Actions** - What DoorDash built: For Ask DoorDash, a simulated user drives fixed scenarios against recorded tool fixtures and an LLM judge scores each session. - Your GitHub Actions version: A required check that replays fixed scenarios against recorded fixtures and gates the change on the rubric pass rate. For Ask DoorDash, a simulated user drives fixed scenarios against recorded tool fixtures, and an LLM judge calibrated against human reviewers scores each session on the same rubric used in production. Generating the 400 conversations of a comprehensive sweep by hand would take more than six hours, and the simulator reduces that generation step to about 20 minutes, which makes it practical to evaluate changes as large as a base-model migration. DoorDash Assistant changes are gated on the rubric pass rate, and DoorDash's carousel generator holds every prompt revision to launch thresholds set before evaluation. Evidence: Documented technique Use it when: Changes to prompts, models or agent code ship after a manual check, or after an eval run too slow to run on every change. Before applying it: Fixtures freeze the world the agent sees; refresh them from production traces so the gate keeps testing current behavior. GitHub Actions adaptation: Compare the source system with your own workflow before applying this translation. [Read more in the DoorDash playbook](https://starsling.dev/ci-playbooks/doordash#simulated-eval-gate) [Primary source](https://careersatdoordash.com/blog/building-ask-doordash-part-three-evaluation/) ### Stripe: Replay a golden dataset on every pull request Stripe engineers recommend a golden dataset for pull requests: a curated set of inputs and expected outputs that covers common flows, qualification boundaries and past incidents. In the workflow they describe, CI runs the candidate implementation against the inputs on every pull request and blocks the build on an unexpected difference, which tells reviewers whether observable behavior changed and where. When an incident exposes a missed edge case, the case can be preserved in the dataset. For larger changes, a Spark job replays historical inputs through the same decision logic and compares the old and new implementations. Evidence: Documented technique Use it when: A service's output depends on many interacting rules, and unit tests cover single rules but not realistic combinations. Before applying it: Keep unit tests for small, hand-built cases. Reduce each input to the fields the behavior needs, and replace sensitive identifiers with stable test identifiers. [Read more in the Stripe playbook](https://starsling.dev/ci-playbooks/stripe#golden-dataset-gate) [Primary source](https://stripe.dev/blog/microservice-testing-with-apache-spark) ## CI optimization is continuous work. Apply this technique yourself, or use StarSling Runners for faster GitHub Actions and optional agent-driven optimization PRs, which teams on eligible paid plans can turn on. [Get self-improving CI](https://starsling.dev/products/runners) ## Company Playbooks ### [Anthropic](https://starsling.dev/ci-playbooks/anthropic) Run only affected tests ### [DoorDash](https://starsling.dev/ci-playbooks/doordash) Run build and test in parallel, sharded across CI agents ### [Linear](https://starsling.dev/ci-playbooks/linear) Benchmark the same job on faster hardware ### [Mastra](https://starsling.dev/ci-playbooks/mastra) StarSling runners for the slowest test workflow ### [OpenAI](https://starsling.dev/ci-playbooks/openai) Rank CI fixes by the minutes they save ### [Ramp](https://starsling.dev/ci-playbooks/ramp) Move a CPU-bound test suite to a faster CPU ### [Shopify](https://starsling.dev/ci-playbooks/shopify) Let an agent iterate against one CI metric ### [Spotify](https://starsling.dev/ci-playbooks/spotify) Build arm64 on native arm64 runners ### [Stripe](https://starsling.dev/ci-playbooks/stripe) Run only the tests a change can reach ### [Uber](https://starsling.dev/ci-playbooks/uber) Build the merge-queue speculations most likely to be needed