Replay fixed evaluations before merging agent changes

Run repeatable scenarios or a golden dataset on changes to agent behavior so regressions are visible before merge.

Prompt

Evaluate and apply this CI technique to my repository: Replay fixed evaluations before merging agent changes.

Run repeatable scenarios or a golden dataset on changes to agent behavior so regressions are visible before merge.

Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/gate-agent-evaluations.md

Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence.
Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured.
Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo.
Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES.
Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch.
Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge.

View all playbooks

How companies used this technique

Read what each company changed and follow the original sources.

Gate agent changes on a simulated, fixture-backed eval run

Translated to GitHub Actions

What DoorDash built: For Ask DoorDash, a simulated user drives fixed scenarios against recorded tool fixtures and an LLM judge scores each session.
Your GitHub Actions version: A required check that replays fixed scenarios against recorded fixtures and gates the change on the rubric pass rate.

For Ask DoorDash, a simulated user drives fixed scenarios against recorded tool fixtures, and an LLM judge calibrated against human reviewers scores each session on the same rubric used in production. Generating the 400 conversations of a comprehensive sweep by hand would take more than six hours, and the simulator reduces that generation step to about 20 minutes, which makes it practical to evaluate changes as large as a base-model migration. DoorDash Assistant changes are gated on the rubric pass rate, and DoorDash's carousel generator holds every prompt revision to launch thresholds set before evaluation.

Use it when: Changes to prompts, models or agent code ship after a manual check, or after an eval run too slow to run on every change.
Before applying it: Fixtures freeze the world the agent sees; refresh them from production traces so the gate keeps testing current behavior.
GitHub Actions adaptation: Compare the source system with your own workflow before applying this translation.

Replay a golden dataset on every pull request

Stripe engineers recommend a golden dataset for pull requests: a curated set of inputs and expected outputs that covers common flows, qualification boundaries and past incidents. In the workflow they describe, CI runs the candidate implementation against the inputs on every pull request and blocks the build on an unexpected difference, which tells reviewers whether observable behavior changed and where. When an incident exposes a missed edge case, the case can be preserved in the dataset. For larger changes, a Spark job replays historical inputs through the same decision logic and compares the old and new implementations.

Use it when: A service's output depends on many interacting rules, and unit tests cover single rules but not realistic combinations.
Before applying it: Keep unit tests for small, hand-built cases. Reduce each input to the fields the behavior needs, and replace sensitive identifiers with stable test identifiers.

How companies used this technique

Gate agent changes on a simulated, fixture-backed eval run

Replay a golden dataset on every pull request

Company Playbooks