Catching flaky tests before they merge: a deterministic pre-pass, not another prompt.

Not every problem in a diff needs a language model to find it. The previous post was about making the LLM’s output trustworthy; this one is about the checks that skip the LLM entirely. A time.Sleep(2 * time.Second) in a test, a rand.Intn with no seed, an assertion on time.Now() — these are flaky-test anti-patterns with near-zero false-positive rates, and paying an LLM to notice them is both slower and less reliable than a few regexes that run before a single token is spent.

That’s what CommitBrief’s flaky-test detector is: a deterministic static pre-pass over the added lines of changed test files. It emits the same Finding shape the LLM review produces, so everything downstream — the cards renderer, --json, --fail-on — treats a flaky finding exactly like a review finding. It’s on by default; --no-flaky skips it for a run, and review.flaky: false disables it in config.

What it looks for

The detector scans added lines in files that look like tests, matching a small set of high-precision anti-patterns:

• Hard-coded sleeps — time.Sleep, Thread.sleep, await asyncio.sleep(…) with a literal duration, used to “wait for” async work. The classic source of tests that pass on a fast machine and fail in CI.
• Unseeded randomness — rand.Intn, Math.random(), random.choice feeding test data or control flow with no fixed seed, so the test exercises a different path every run.
• Time dependencies — assertions built on time.Now(), Date.now(), or the system clock, which fail at midnight, across a DST boundary, or in a different timezone.
• Brittle selectors — UI/E2E locators pinned to auto-generated class names or absolute XPath indices that shift the moment the DOM changes.
• Over-mocking — a test so thoroughly mocked that it asserts against its own mocks and validates nothing about the real code path.

Each match becomes a finding with a severity, a line anchor, and a suggestion — “seed the RNG,” “inject a clock,” “wait on a condition, not a duration.” The point isn’t to be clever; it’s to be precise. A flaky-test detector that cries wolf gets turned off in a week, so the rule set is deliberately narrow: high confidence, low recall, no heuristics that would fire on legitimate code.

Why deterministic, not an LLM prompt

The tempting alternative was to fold flaky-test awareness into the review prompt — “also flag flaky-test anti-patterns.” I chose the static pre-pass instead, for three reasons.

It’s free. The scan runs locally over the diff before the provider call. No tokens, no latency, no dependence on which model you’re using or how it’s feeling today.

It’s deterministic. The same diff produces the same findings every run. A prompt-augmented approach would flag a time.Sleep sometimes and miss it other times, depending on sampling — the worst property for something you want to wire into a CI gate.

It composes. Because a flaky finding is a normal Finding, it flows into --fail-on and --json with no special casing. commitbrief --staged --fail-on high blocks a merge on a critical flaky pattern the same way it blocks on a critical review finding.

The trade-off is honest: a static pre-pass only catches patterns someone encoded a rule for. It won’t reason about a subtle race that has no textual tell. That’s fine — it’s a floor, not a ceiling. The LLM review still runs on the same diff and can catch the things a regex can’t.

Confirming a flake by actually running it

A static pattern infers flakiness; it can’t prove it. A time.Sleep might be load-bearing in a way the pattern can’t see. So there’s an opt-in second stage that raises confidence by doing the obvious thing — running the flagged test in isolation, several times, and watching what happens:

# Re-run each statically-flagged test up to 5 times in isolation.
commitbrief --staged --sandbox-rerun

# Or set the rerun count explicitly.
commitbrief --staged --sandbox-rerun=10

The verdict comes from the observed pass/fail mix:

• Mixed pass and fail → confirmed flaky. The test is genuinely non-deterministic.
• All fails → a real failure. It’s red for a reason — not a flake to quarantine, an actual bug.
• All passes → transient. The flake didn’t reproduce this time, so the finding is demoted to info rather than dropped, because “didn’t reproduce in five runs” isn’t “definitely fine.”

There’s an early exit built in — a genuinely flaky test usually shows a mixed result within the first two runs, so the loop stops as soon as it has proof rather than always running the full N.

Sandbox-rerun is off by default and sits behind an executor seam: the core ships the orchestration — the rerun loop, the classification, the early exit — but not a language-specific test runner. You (or an integration) bind a func(ctx, testID) (passed bool, err error) that knows how to run one test in your project. That keeps the detector’s core pure and testable with a fake, and keeps CommitBrief out of the business of embedding a runner for every test framework in existence.

Where it sits in the pipeline

The flaky pre-pass runs before the provider call, and its findings are merged with the LLM’s — with one rule: if the detector and the model both flag the same line, the model’s richer finding wins and the duplicate is dropped. So you don’t get two findings for one time.Sleep; you get the better of the two. The net effect is a review that opens with the cheap, certain, deterministic checks and layers the model’s judgement on top.

The next post stays on the theme of feeding the review things a plain diff can’t tell it — this time, your project’s own declared architecture, read straight from a sibling tool’s config so the model can flag a diff that crosses a boundary it was never told about.

← all posts