Make unordered results deterministic before comparing

Sort values whose order was never guaranteed, so an equivalent result passes on the first run.

Prompt

Evaluate and apply this CI technique to my repository: Make unordered results deterministic before comparing.

Sort values whose order was never guaranteed, so an equivalent result passes on the first run.

Read the technique and its source-backed company examples: https://starsling.dev/ci-playbooks/techniques/normalize-unordered-results.md

Inspect this repository’s CI workflows, build and test commands, dependencies, and recent run timings. Identify where this technique applies and explain the expected benefit using this repository’s evidence.
Measure a baseline from recent comparable runs: elapsed time, queue time, runner minutes and failures where available. If data is unavailable, say what cannot be measured.
Compare relevant company approaches, checking prerequisites, platform differences and tradeoffs. Rank applicable options by evidence, fit and measurable opportunity; do not present company results as predictions for this repo.
Recommend the smallest safe change, show what checks and fallback paths remain, and ASK ME WHICH CHANGE TO APPROVE BEFORE EDITING ANY FILES.
Only after approval, work on a new feature branch, make one reversible change, and open a reviewable pull request. Never merge or push to a default or protected branch.
Keep required checks, test coverage, deployment and release behavior intact. Validate the change and compare equivalent runs where possible; report limitations and what should be monitored after merge.

View all playbooks

How Uber used this technique

Read what Uber changed and follow the original sources.

Make unordered results deterministic before comparing

Cadence sorted the active-cluster list before formatting a domain error. A test expected [A B] while an equivalent result arrived as [B A]. OPT48 is adapted to one order-dependent test, and the change sorts the production error output.

The PR shows the expected and actual output from the flaky run and a two-line sort. Source

Use it when: A test compares ordering the producer never guaranteed, and deterministic ordering suits the public output.
Before applying it: Decide first whether ordering is part of the output contract. Sorting in place mutates the caller's slice, so sort a copy where callers share it, and keep membership and error-field assertions.

How Uber used this technique

Make unordered results deterministic before comparing

Related techniques

Check for leaks after this test cleans up

Prove a new test is stable before it joins the blocking suite

Wait for the state transition before asserting

Company Playbooks