---
title: "Give agents a conformance suite to test against"
description: "Write a suite that checks changes against the reference behavior the system must match, and have agents run it on every change."
url: https://starsling.dev/ci-playbooks/techniques/agent-conformance-suite
canonicalUrl: https://starsling.dev/ci-playbooks/techniques/agent-conformance-suite
---

# Give agents a conformance suite to test against

Write a suite that checks changes against the reference behavior the system must match, and have agents run it on every change.

## How to choose an approach

This technique currently has one documented company example. Validate its prerequisites, tradeoffs and original CI environment against your repository before adapting it.

- **[OpenAI](https://starsling.dev/ci-playbooks/openai#agent-conformance-suite)**: Use it when Agents change code that has to match a reference: a protocol, a file format, a public API, or the system it reimplements. Check before adopting: Take the expected results from the reference implementation itself, and add a case each time an agent finds a new difference. Adapted for GitHub Actions; compare the original system with yours.

## Apply this technique

````text
Evaluate and apply this CI technique to my repository: Give agents a conformance suite to test against.

Write a suite that checks changes against the reference behavior the system must match, and have agents run it on every change.

Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/agent-conformance-suite.md

Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence.
Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured.
Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo.
Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES.
Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch.
Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge.
````

## Get the ci-speedup skill

An agent skill to find and fix slow CI.

```text
Install the ci-speedup skill in this repository with `npx skills add starslingdev/skills --skill ci-speedup`. Then run ci-speedup here, following its SKILL.md instructions. Use this repository’s workflows and recent GitHub Actions runs to find the bottlenecks and recommend fixes. Show me the evidence and ask which changes to apply before editing files.
```

[Explore the skill](https://starsling.dev/skills/ci-speedup)

## How OpenAI used this technique

Read what OpenAI changed and follow the original sources.

### OpenAI: Give agents a conformance suite to test against

**Translated to GitHub Actions**

- What OpenAI built: OpenAI wrote a conformance test suite so the agents building Pando, its Git backend, could test their changes against standard Git behavior.
- Your GitHub Actions version: A conformance suite in your repository that agents run on every change and a GitHub Actions job runs on every pull request.

OpenAI built Pando, a Git backend for its monorepo, with Codex in ten weeks: one engineer designed its core, teams of agents implemented it, and a handful of engineers deployed it. The team wrote a conformance test suite so agents could test their changes against standard Git behavior. It also built deterministic simulation testing harnesses that check the system's invariants and properties, which found subtle race conditions and regressions.

Evidence: Documented technique

Use it when: Agents change code that has to match a reference: a protocol, a file format, a public API, or the system it reimplements.

Before applying it: Take the expected results from the reference implementation itself, and add a case each time an agent finds a new difference.

GitHub Actions adaptation: Compare the source system with your own workflow before applying this translation.

[Read more in the OpenAI playbook](https://starsling.dev/ci-playbooks/openai#agent-conformance-suite)

[Primary source](https://openai.com/index/scaling-git-at-openai/)

## CI optimization is continuous work.

Apply this technique yourself, or use StarSling Runners for faster GitHub Actions and optional agent-driven optimization PRs, which teams on eligible paid plans can turn on.

[Get self-improving CI](https://starsling.dev/products/runners)

## Company Playbooks

### [Anthropic](https://starsling.dev/ci-playbooks/anthropic)

Run only affected tests

### [DoorDash](https://starsling.dev/ci-playbooks/doordash)

Run build and test in parallel, sharded across CI agents

### [Linear](https://starsling.dev/ci-playbooks/linear)

Benchmark the same job on faster hardware

### [Mastra](https://starsling.dev/ci-playbooks/mastra)

StarSling runners for the slowest test workflow

### [OpenAI](https://starsling.dev/ci-playbooks/openai)

Rank CI fixes by the minutes they save

### [Ramp](https://starsling.dev/ci-playbooks/ramp)

Move a CPU-bound test suite to a faster CPU

### [Shopify](https://starsling.dev/ci-playbooks/shopify)

Let an agent iterate against one CI metric

### [Spotify](https://starsling.dev/ci-playbooks/spotify)

Build arm64 on native arm64 runners

### [Stripe](https://starsling.dev/ci-playbooks/stripe)

Run only the tests a change can reach

### [Uber](https://starsling.dev/ci-playbooks/uber)

Build the merge-queue speculations most likely to be needed