Run every example in auto mode and check each log against its source

Give examples a non-interactive runner with per-example logs and rerun files, then carry it into Actions.

Prompt

Evaluate and apply this CI technique to my repository: Run every example in auto mode and check each log against its source.

Give examples a non-interactive runner with per-example logs and rerun files, then carry it into Actions.

Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/example-auto-runner.md

Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence.
Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured.
Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo.
Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES.
Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch.
Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge.

View all playbooks

How OpenAI used this technique

Read what OpenAI changed and follow the original sources.

Run every example in auto mode and check each log against its source

The Agents SDK repositories built a runner that executes examples in auto mode: it answers common prompts, approves tool actions where supported, keeps examples that need extra runtime setup on a skip list, writes a log for each example and writes rerun files so a rerun covers only the examples that need one. A skill then has the agent read each log against the example's source. The JavaScript repository also publishes its packages to a local registry and tests installing and running them on Node.js, Bun, Deno, Cloudflare Workers and a Vite React app.

Use it when: Examples or sample apps are validated by eye or by exit code alone.
Before applying it: An exit code of zero says little about an example that calls a real API, so compare the recorded output with the example's intended flow.

How OpenAI used this technique

Run every example in auto mode and check each log against its source

Company Playbooks