How a single JSON schema survives four very different LLM providers.

Every LLM provider has its own answer to the question “how do I get structured JSON out of you reliably?” Anthropic answers with the tools API. OpenAI answers with strict response_format: json_schema. Gemini answers with ResponseMIMEType: application/json plus a ResponseSchema. Ollama answers with format: "json". These four answers don’t compose. Each one has its own quirks about required vs optional fields, schema enforcement strictness, and what the model actually does when the schema is hard to satisfy.

CommitBrief’s --json output uses the same schema v1 contract regardless of which provider is on the other end. This post is about how that contract holds together across the four implementations, what compromises the lowest-common-denominator design forced, and the retry-once-then-degrade fallback for the cases where the contract still slips.

The contract

The shape every provider has to produce:

{
  "findings": [
    {
      "severity": "critical | high | medium | low | info",
      "file": "string",
      "line": 142,
      "line_end": 156,
      "title": "string",
      "description": "string",
      "suggestion": "string",
      "language": "go",
      "snippet": "string"
    }
  ]
}

severity, file, line, title, description, and suggestion are required. line_end (for multi-line findings), language, and snippet are optional. The renderer wraps this into the public schema v1 envelope ({schema, content, findings, summary, meta}) and computes summary from the finding list. The model’s job is just the inner array.

This is the inner contract. The four implementations enforce it differently.

Anthropic: tools mode

Anthropic’s strongest schema-enforcement mechanism is the tools API. CommitBrief defines a single tool — call it report_findings — with the JSON schema as the tool’s parameter shape, and uses tool_choice: {type: "tool", name: "report_findings"} to force the model to call exactly that tool. The model’s response comes back as a tool-use block whose input is guaranteed (within the SDK’s contract) to validate against the declared schema.

The win here is that Anthropic’s schema enforcement runs server-side as part of the tool-use machinery. The model is constrained to produce the shape; if it can’t, the API returns an error rather than a non-conformant response. Required vs optional is exactly what the JSON Schema spec says; the enum constraint on severity is respected.

The catch: Anthropic counts tool-use messages slightly differently for prompt-caching purposes than plain message turns. The system prompt section that carries COMMITBRIEF.md gets cached normally (5-minute ephemeral cache), but the tool definition itself counts as part of the cached prefix, which means changes to the tool schema invalidate the cache. This matters for the engineering side — we want the tool definition stable across releases so the cache stays warm — and contributed to the JSON schema v1 freeze at v1.0.

OpenAI: strict mode

OpenAI’s response_format: { type: "json_schema", json_schema: { ..., strict: true } } does server-side schema validation similar to Anthropic’s tools mode, with one significant constraint: strict mode rejects optional properties. Every property declared in the schema has to appear in every response, even if the schema says it’s not required.

For CommitBrief that’s a problem, because the JSON contract has three optional fields: line_end, language, snippet. The model shouldn’t be forced to invent a snippet for every finding when the snippet wouldn’t materially clarify the finding; the design point in v0.9.0 was specifically that the model should omit snippets when in doubt rather than producing unhelpful ones.

The workaround: split the schema into a required-only subset for OpenAI’s strict-mode enforcement, and let the optional fields stay prompt-driven. The OpenAI integration declares severity, file, line, title, description, and suggestion as required-and-strict-enforced; line_end, language, snippet are mentioned in the prompt as “include when material” but not in the response_format schema. The downside: OpenAI’s optional-field reliability is lower than Anthropic’s, because the prompt is the only thing telling the model when to emit them. In practice GPT-4o gets it right enough of the time that this isn’t a frequent failure mode, but it’s a real asymmetry.

Gemini: ResponseSchema

Gemini takes the third path: ResponseMIMEType: "application/json" plus a ResponseSchema that mirrors JSON Schema’s structure. Optional fields are honoured (unlike OpenAI’s strict mode), and enum constraints on severity are respected. The mechanism is structurally closer to Anthropic’s tools mode than to OpenAI’s strict mode.

The Gemini-specific quirk that v1.0.0-rc.1’s gosec audit caught: the SDK takes the max-output-tokens parameter as int32. CommitBrief’s req.MaxTokens was a plain int, cast directly to the SDK’s int32 — a value above math.MaxInt32 would silently wrap to negative, which Gemini’s API interprets as “unlimited” with potentially expensive consequences. The fix bounds the cast to [1, math.MaxInt32] with a default of 4096 when the request doesn’t specify. Not exciting code; very important code.

Gemini’s context-caching API is separate from its inline prompt caching and isn’t currently wired into the CommitBrief integration. The cached-input pricing column in the verbose footer is there for accurate reporting if and when we wire it; until then, Gemini cached-token counts in the footer are always zero.

Ollama: format JSON

Ollama is the loosest of the four. format: "json" instructs the model to produce JSON; it does not validate the response against a schema. Reliability is entirely a function of the model’s instruction-following.

For a 14-billion-parameter coder model like qwen2.5-coder:14b, this works well most of the time. For smaller or older models, it works less well. CommitBrief’s prompt explicitly tells the model the JSON shape it should produce, but the model can always emit something else and Ollama won’t catch it — the response just arrives as a JSON string that fails to parse against the v1 schema.

This is where the retry-once-then-degrade fallback earns its keep.

The retry-once-then-degrade fallback

Every provider goes through the same response-handling path: try to parse the returned content as the v1 findings schema. If parsing fails, fire one retry with a tighter nudge in the prompt (“the previous response was not valid JSON; please respond with strict JSON matching this schema”). If the retry also fails, degrade to markdown rendering — emit whatever the model produced as plain text, and surface a stderr warning:

warning: LLM produced malformed JSON; falling back to plain-text view.

The cache entry records the fallback mode (format: "markdown-fallback") so subsequent replays come back through the same fallback path without re-firing the retry. --fail-on skips its threshold check entirely in this state because there are no structured findings to evaluate against the threshold; the previous post on --fail-on covers why this skip is the right shape rather than treating the malformed response as a critical failure.

The design choice is: assume the parse failure is recoverable (retry once), but don’t compound the failure (no retry-N), and degrade gracefully (still show the user something) rather than crashing the run.

In practice, the fallback rarely fires on API providers — Anthropic, OpenAI, and Gemini’s schema-enforced modes produce conformant responses near-deterministically. It fires noticeably more often on Ollama with smaller models. That asymmetry is in the design; it’s not a bug in the fallback.

Why a single contract was worth the compromise

The cost of the lowest-common-denominator approach is that no individual provider’s full capability gets surfaced. Anthropic could emit richer tool-use metadata; OpenAI’s strict mode could enforce more invariants; Gemini’s massive context window could enable larger finding sets. The lowest-common-denominator JSON schema is what works on all four — and on the fifth provider (Mistral, Cohere, whatever ships next) someone might add later.

The benefit is that everything downstream of the JSON output — the OUTPUT.md template, the cards renderer, --copy, --fail-on, the editor extensions that don’t exist yet — only has to handle one shape. Adding a provider doesn’t mean updating the renderer; updating the renderer doesn’t mean re-validating against four provider schemas. The contract is the seam.

This is the same trade-off the provider abstraction makes one layer up — narrow interface, exposed via lowest-common-denominator, cost paid in not-quite-optimal-per-provider behaviour. At both layers I think it’s worth it; a code-review tool needs to be reliable across the providers people actually have, not the perfectly-tuned best on the one provider we know best.

What this means for adding a sixth provider

If someone wants to add Mistral or Cohere or a self-hosted vLLM endpoint, the work is: implement the Provider interface, decide which of the four structured-output mechanisms is closest to the new provider’s native capability, hook that up, and verify the v1 schema validates. If the provider has no structured-output enforcement at all, fall back to prompt-only and trust the retry-once-then-degrade path to handle the noisy cases.

The schema doesn’t change. That’s the point.

The next post in this series turns from architecture to operations: commitbrief doctor, the eight-check pipeline health diagnostic that’s supposed to be the first thing you run when anything stops working — and why I wish more CLIs shipped one.

← all posts