--- title: "Have security agents write a failing integration test before they fix" description: "For each security finding, an agent writes an endpoint-level test that reproduces it and confirms it fails, and a second agent patches the code until that test passes." url: https://starsling.dev/ci-playbooks/techniques/security-agent-failing-test canonicalUrl: https://starsling.dev/ci-playbooks/techniques/security-agent-failing-test --- # Have security agents write a failing integration test before they fix For each security finding, an agent writes an endpoint-level test that reproduces it and confirms it fails, and a second agent patches the code until that test passes. ## How to choose an approach This technique currently has one documented company example. Validate its prerequisites, tradeoffs and original CI environment against your repository before adapting it. - **[Ramp](https://starsling.dev/ci-playbooks/ramp#security-agent-failing-test)**: Use it when Security findings arrive faster than people can validate them, and the affected endpoints already have integration tests to copy. Check before adopting: A person still reviews and lands every patch the fixer opens, so each validated finding reaches review with the test that proves it. Adapted for GitHub Actions; compare the original system with yours. ## Apply this technique ````text Evaluate and apply this CI technique to my repository: Have security agents write a failing integration test before they fix. For each security finding, an agent writes an endpoint-level test that reproduces it and confirms it fails, and a second agent patches the code until that test passes. Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/security-agent-failing-test.md Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence. Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured. Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo. Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES. Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch. Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge. ```` ## How Ramp used this technique Read what Ramp changed and follow the original sources. ### Ramp: Have security agents write a failing integration test before they fix **Translated to GitHub Actions** - What Ramp built: Ramp's validator agent wrote a failing integration test for each security finding, and its fixer, an early version of Inspect, patched the code until that test passed. - Your GitHub Actions version: An agent workflow that writes a failing endpoint-level integration test for each security finding, patches the code until the test passes, and opens the fix as a pull request for review. In a one-week experiment, Ramp ran a team of security agents: detectors for each vulnerability class, managers that critiqued their findings, a validator and a fixer. The validator took each reported finding, read the existing tests for the endpoint and wrote an integration test that reproduced the vulnerability as an attacker would while asserting the secure result. It ran the test and confirmed it failed; a test that already passed marked the finding a false positive, and no code changed. The fixer, an early version of Inspect, ran the same test to confirm it failed, patched the code and re-ran the test until it passed, leaving the test as written, then opened a pull request for a person to review and land. Evidence: Company-reported result or observation 2026: Ramp's test-based validation captured nearly all of the false positive findings produced earlier in its triage pipeline. ([Source](https://builders.ramp.com/post/100-vulnerabilities-patched-with-0-humans)) Use it when: Security findings arrive faster than people can validate them, and the affected endpoints already have integration tests to copy. Before applying it: A person still reviews and lands every patch the fixer opens, so each validated finding reaches review with the test that proves it. GitHub Actions adaptation: Compare the source system with your own workflow before applying this translation. [Read more in the Ramp playbook](https://starsling.dev/ci-playbooks/ramp#security-agent-failing-test) [Primary source](https://builders.ramp.com/post/100-vulnerabilities-patched-with-0-humans) ## CI optimization is continuous work. Apply this technique yourself, or use StarSling Runners for faster GitHub Actions and optional agent-driven optimization PRs, which teams on eligible paid plans can turn on. [Get self-improving CI](https://starsling.dev/products/runners) ## Company Playbooks ### [Anthropic](https://starsling.dev/ci-playbooks/anthropic) Run only affected tests ### [DoorDash](https://starsling.dev/ci-playbooks/doordash) Run build and test in parallel, sharded across CI agents ### [Linear](https://starsling.dev/ci-playbooks/linear) Benchmark the same job on faster hardware ### [Mastra](https://starsling.dev/ci-playbooks/mastra) StarSling runners for the slowest test workflow ### [OpenAI](https://starsling.dev/ci-playbooks/openai) Rank CI fixes by the minutes they save ### [Ramp](https://starsling.dev/ci-playbooks/ramp) Move a CPU-bound test suite to a faster CPU ### [Shopify](https://starsling.dev/ci-playbooks/shopify) Let an agent iterate against one CI metric ### [Spotify](https://starsling.dev/ci-playbooks/spotify) Build arm64 on native arm64 runners ### [Stripe](https://starsling.dev/ci-playbooks/stripe) Run only the tests a change can reach ### [Uber](https://starsling.dev/ci-playbooks/uber) Build the merge-queue speculations most likely to be needed