`--fail-on`: turning an LLM review into a CI gate without locking yourself in.

A code-review CLI that doesn’t fail your CI on a critical finding isn’t a gate — it’s a printout. A code-review CLI that fails your CI on any finding is a tax. Neither shape is what people actually want; the right shape is somewhere in the middle, parameterised by a severity threshold the team gets to set.

--fail-on=<severity> is that parameter in CommitBrief. The thing it does is small — map the highest finding severity in the run to an exit code. The reasons it does that small thing in exactly the way it does it are worth unpacking, because the surface is doing a few non-obvious things at once.

The mechanic

After the renderer prints whatever it was going to print, the CLI looks at the parsed findings list. If any finding meets or exceeds the requested severity, the process exits with code 1. If none do, it exits 0. The threshold takes one of: critical, high, medium, low, info, any, none, plus the empty string (which is the off-by-default).

# Fail only on critical-severity findings. Most common shape.
commitbrief --staged --fail-on=critical

# Fail on anything from medium up.
commitbrief --staged --fail-on=medium

# Strictest setting — any finding at all fails.
commitbrief --staged --fail-on=any

The severity ladder is fixed and lowercase: critical > high > medium > low > info. --fail-on=high means “fail at high or critical.” --fail-on=low means “fail at low or anything more severe.” The full enum is part of the JSON schema v1 contract, so a CI script can introspect a finding’s severity field and compute the same threshold itself if it needs custom logic — --fail-on is a convenience over that calculation, not a replacement for it.

Anything outside the accepted set is a parse error before the review fires. --fail-on=blocker (the v0.x word that some users still type by reflex) produces invalid --fail-on value "blocker" and exits 1 before a single token is spent. That’s deliberate: silently treating an unknown value as “off” is exactly the kind of behaviour that produces a CI that claims to be gating when it isn’t.

Why the rendered output prints first

The review prints to stdout in full before the exit-code decision is made. The consumer — a developer reading the CI log, or a reviewer reading the PR check output — sees the findings that caused the failure, not just the failure message. This matches how npm test works: the test output streams, then the exit code reflects the result.

The alternative — fail-fast, abort the print, just show the count — would optimise for log brevity at the cost of debuggability. When a CI build fails on a critical finding, the next question is always “which critical finding?” Putting the rendered review above the exit makes that answer one scroll away instead of one re-run away.

Graceful degrade and the skip behaviour

There’s one case where --fail-on deliberately doesn’t fire: when the LLM produced unparseable output and the one-shot retry also failed. In that path, the renderer degrades to plain-text markdown rendering and the findings list is nil — there’s nothing structured to evaluate.

When this happens, stderr prints:

ℹ --fail-on skipped: LLM produced unparseable output, no findings to evaluate.

The exit code from the review itself is 0 (the run completed) unless the pipeline had a separate failure. The reasoning is uncomfortable but defensible: failing CI on a flaky model invocation is worse than letting the run succeed and surfacing the markdown text. The user still sees the review; they can re-run; if the flakiness is persistent it’ll surface as a noisy CI signal that’s clearly different from a real finding-driven fail.

A different design would treat “unparseable LLM output” as a critical failure and exit 1. I rejected that because it conflates two very different concerns — “the model found a bug” and “the model produced gibberish today” — into one CI failure, which means the team’s response to a red build becomes ambiguous. The skip message is the unambiguous version: the gate didn’t fire because there was nothing to gate on.

Composing with other guards

--fail-on runs after a successful provider call. The pre-send guards from the previous post — secret scanner, cost preflight, .commitbrief/ check — all run earlier in the pipeline and each can independently exit 1 with their own message. The exit code is the same (1), but stderr disambiguates: a .commitbrief/ guard abort says aborted by pre-send guard, the secret scanner says aborted: pre-send secret scanner, the cost preflight says aborted: cost preflight, and --fail-on says <N> finding(s) at or above '<severity>' severity.

This means a CI step that’s gating on findings is also implicitly gating on the pre-send guards — the run can’t reach --fail-on evaluation if the secret scanner aborted earlier. That’s the right shape: a script that opts into severity-based gating is not opting out of credential checks.

Real CI shapes

The most common pattern is a single step in a workflow file:

- name: CommitBrief review
  run: commitbrief --staged --fail-on=critical --no-cost-check
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

The --no-cost-check is there because CI runners are non-interactive — if the diff happens to push past the cost threshold, the preflight would auto-abort without a human to confirm. If your team prefers a hard ceiling on cost-per-CI-run, lower the threshold (commitbrief config set cost.warn_threshold_usd 0.20) instead of bypassing.

For the artifact-plus-gate shape, where the JSON output also gets archived:

- name: CommitBrief review
  run: commitbrief --staged --fail-on=high --json --output review.json
- uses: actions/upload-artifact@v4
  with:
    name: review
    path: review.json

The JSON output is the compliance evidence shape; the --fail-on=high is the gate. They compose cleanly because the gate fires after the JSON is written — the artifact upload happens whether the build passes or fails, so you can inspect what triggered the failure even on a red build.

For pre-commit hooks, the generated body from commitbrief install-hook already embeds the gate:

commitbrief --staged --fail-on=critical --quiet --no-cost-check

The --quiet is there to suppress the per-stage progress noise that doesn’t render usefully in a hook context. The --no-cost-check is for the same reason as in CI — the hook can’t prompt.

Threshold-picking advice

Three patterns I see working in practice, in rough order of conservatism:

Pre-commit hook: --fail-on=critical. Block only on the things you’d cancel a commit for anyway. Lower thresholds at this layer produce frustration without a proportional safety win — a high-severity finding the developer wants to ship is now a hook-bypass exercise.

PR check / CI: --fail-on=high. PR-time is the right moment for “this should be discussed before merging.” High and above are the findings worth blocking a merge on; medium and below get surfaced in the review output for the human reviewer to weigh.

Nightly / scheduled audit: --fail-on=medium with notification. Run against the trailing N commits, fail at medium so noisy-but-real findings get caught, route the failure to a Slack channel rather than the build system. This is the shape that catches drift without slowing down feature work.

The wrong shape is --fail-on=any. It produces a CI that fails any time the model finds anything, which trains the team to ignore the signal. The mechanic still exists for completeness; I don’t recommend it as a default.

What this doesn’t do

--fail-on doesn’t decide what the model flags. That’s COMMITBRIEF.md’s job — if you’re getting too many critical findings, the answer is to write rules that calibrate the model’s severity assignments, not to raise the threshold. If you’re getting too few, the answer is rules that name the patterns you want flagged. The anatomy post is the right starting point for that work.

--fail-on also doesn’t replace the human reviewer. The manifesto covered this; a gate at --fail-on=critical catches the SQL-injection-shaped problems but doesn’t catch “this function answers the wrong question.” That’s still the senior’s job. The gate is the smallest layer in a stack, not the whole stack.

← all posts