// docs · v1.x

Flaky-test detector

A deterministic, provider-free pre-pass that flags timing-dependent and unseeded-random anti-patterns in changed test files before any model call.

Some bugs are not in the diff’s logic — they are in tests that pass or fail depending on wall-clock timing, an unseeded RNG, a brittle selector, or over-mocking. The flaky-test detector is a static, provider-free pre-pass that catches the highest-precision of these anti-patterns in your changed test files before any model call. New in v1.7.0; three more rules added in v1.8.0, and opt-in sandbox-rerun confirmation in v1.12.0 (ADR-0022).

What it scans

Only the added lines (+-prefixed) of files that look like tests in your current scope — the same diff the reviewer would see, so the detector never re-flags history already on disk. It runs entirely locally: no provider call, no extra token cost, no JSON-schema change.

What it flags

Five narrow, high-confidence rules:

hard-sleep — hard-coded sleeps and fixed waits. A test that pauses for a fixed duration to “let things settle” is the classic source of timing flakiness. Detected forms include time.Sleep, Thread.sleep, Task.Delay, asyncio.sleep, *.waitForTimeout, a numeric cy.wait, usleep, and sleep(<n>).
unseeded-random — unseeded randomness. Randomness without a fixed seed makes a test non-reproducible. Detected forms include Math.random, Python random.*, and Go math/rand.
brittle-selector (JS/TS) — position-based selectors. A UI/test selector that depends on an element’s position breaks the moment the DOM shifts. Flagged forms: :nth-child, an absolute XPath, .eq(<n>) / .nth(<n>), and a trailing [<n>] index predicate. Stable selectors — data-testid, role, or text — are not flagged; prefer those.
over-mock — excessive mocking. A single test piling on an unusual number of mock setups is a signal the test is brittle and over-coupled to implementation detail. Counted file-scoped, per test function.
time-dependency — wall-clock-coupled assertions. An assertion tied directly to the wall clock (time.Now(), Date.now(), new Date(), …) without an injected clock is non-deterministic across runs and time zones. Inject a clock / freeze time instead.

The patterns are deliberately conservative: the detector aims for precision over recall, so a flag is almost always worth a look.

False-positive caveats

Each rule trades recall for precision, but a few legitimate patterns can still trip them:

brittle-selector can fire on a [<n>] index or .nth(n) that is genuinely the cleanest way to assert against a fixed-order list. Switch to a data-testid/role/text selector where one exists, or add an inline suppression with a reason if the position really is stable.
over-mock is a heuristic count, so an integration-style test that legitimately wires up many collaborators may cross the line.
time-dependency flags wall-clock calls in assertions even when you do control time another way (a test-only env override, a fixed fixture). If the coupling is intentional and safe, suppress it inline.

Sandbox-rerun confirmation (opt-in)

The rules above infer flakiness from anti-patterns. Sandbox-rerun confirms it: it actually re-runs a flagged test in isolation N times and classifies it by the observed pass/fail mix.

Observed result	Verdict	What CommitBrief does
mixed pass + fail	confirmed flaky	Keep the finding; note the empirical confirmation.
all fail	real failure	Keep it, but relabel — the test is genuinely red, not a flake, so don’t quarantine it.
all pass	transient / resolved	Demote the finding to `info`, so a non-reproducing one-off won’t trip a commit-stage `--fail-on`.

Enable it per-run with --sandbox-rerun[=N] (the bare flag uses N=5) or persistently with review.sandbox_rerun: <N> (0 = off, the default):

commitbrief --staged --sandbox-rerun      # N=5
commitbrief --staged --sandbox-rerun=3

review:
  sandbox_rerun: 0   # 0 = off (default); >0 = rerun count; --sandbox-rerun[=N] overrides per-run

The verdict rides the existing finding (its suggestion text and severity), so there is no JSON-schema change — the findings contract stays v1.

Status — off by default, no runner shipped yet. CommitBrief ships the rerun orchestration — the N-rerun loop, the classification, and an early exit once a mixed result is proven — but binds it to a runner through a caller-provided seam (func(ctx, testID) (passed, err)). No language-specific test runner is wired in yet, so the flag and config are reserved and a transparent no-op until one is bound. Existing behaviour is byte-identical.

How findings surface

Flaky findings are first-class — they merge into the same structured output as the model’s findings, which means they:

render in the cards, JSON, and markdown just like any other finding;
count toward --fail-on for CI gating;
are included by --copy.

There is no separate flaky-only output mode — they live alongside correctness, security, and the rest. Messages are localized (en/tr).

When it runs

On by default for the API providers (Anthropic, OpenAI, Gemini, DeepSeek, Mistral, Cohere, Ollama). It does not apply to the CLI-tool-backed providers (claude-cli / gemini-cli / codex-cli), which emit pre-formatted plain text with no structured findings to merge into.

Turning it off

# Per-invocation
commitbrief --staged --no-flaky

# Per-config (whole binary)
review:
  flaky: false   # default true; --no-flaky is the per-run equivalent

commitbrief config set review.flaky false

Disable it if you have a dedicated flake-detection layer in CI, or if a code base legitimately uses these patterns in a way the detector cannot distinguish.