How do you fix inconsistent LLM output in a review workflow?

Inconsistent LLM output has three separate causes: how the model is served, how loose your prompt and output format are, and how much your human reviewers disagree. Setting temperature to 0 does not remove the first. Fix the output format, write a pass-or-fail rubric with one expert, then measure agreement before you trust the workflow.

What makes LLM output inconsistent?

Three causes stack on top of each other. Each needs its own fix.

Cause	What it looks like	First fix
Serving noise	Same prompt, same settings, different words	Accept it, then constrain what matters
Loose prompt or format	Different structure, length or labels each run	Fixed schema, examples, smaller steps
Reviewer disagreement	Two experts grade the same output differently	Binary rubric, written reasons, agreement check

The first row is the one people notice. The second and third rows are the ones you can fix.

Can you make an LLM fully deterministic?

Not reliably with a hosted model. Horace He and colleagues at Thinking Machines tested a 235B-parameter model at temperature 0. They sampled 1,000 completions and got 80 unique outputs (Thinking Machines, 2025-09). The cause, they explain, is that inference kernels are not batch-invariant, and server load changes batch size from one request to the next. With batch-invariant kernels, all 1,000 completions matched.

You rarely control the server kernels of a hosted API. So treat small wording changes as normal. Design the workflow so that wording differences do not change the decision a reviewer makes.

How do you constrain the output format?

Start here, because it is cheap and it removes a whole class of variation.

1. Use schema-enforced output where the model supports it. Anthropic says its structured outputs “guarantee schema-compliant responses through constrained decoding” (Anthropic docs, accessed 2026-10-30). OpenAI says only its Structured Outputs “ensure schema adherence”, while JSON mode only gives valid JSON (OpenAI docs, accessed 2026-10-30).
2. Know the limits. Both vendors list exceptions. A refusal can override the schema. A response that hits the token limit can be cut off. Some schema features are not supported (Anthropic docs; OpenAI docs).
3. Remember what a schema does not do. It makes the shape of the answer stable. It does not make the content correct or the verdict consistent.
4. Show examples of the output you want. Anthropic’s consistency guide says examples are “more effective than abstract instructions” (Anthropic docs, accessed 2026-10-30).
5. Split complex tasks. The same guide advises chaining prompts so that each prompt covers one subtask, which reduces inconsistency errors across scaled workflows.
6. Ground answers in a fixed source. For knowledge tasks, the guide recommends retrieval so answers draw on a fixed set of information.

How do you make the review itself consistent?

A model can be stable and still be judged unevenly. If two reviewers disagree, the workflow looks inconsistent even when the model is fine.

Use binary labels. Husain and Shankar argue that “binary evaluations force clearer thinking and more consistent labeling” (Husain and Shankar, Evals FAQ, 2026-09-18). They say that on a 1-to-5 scale, adjacent points are subjective and annotators drift to the middle. For gradual progress, they suggest several binary sub-checks instead.

Anchor on one principal expert. In his judge guide, Husain says there is usually “one (maybe two) key individuals whose judgment is crucial”. The expert gives a pass or fail verdict and a written reason for each output (Hamel Husain, 2024-10).

Expect the rubric to change. Shankar and co-authors describe criteria drift: “users need criteria to grade outputs, but grading outputs helps users define criteria” (Shankar et al., arXiv, 2024-04). Re-label a sample after you revise the rubric.

Measure agreement, not just accuracy. Cohen’s kappa measures agreement between two annotators after correcting for chance. A score of 1.0 is full agreement. A score of 0.0 is no agreement beyond chance (scikit-learn docs). Husain adds that raw agreement can mislead when classes are imbalanced, and he suggests reporting true positive and true negative rates separately (Hamel Husain, 2024-10). The scikit-learn page gives no threshold for “good enough”. Agree on your own bar with your experts.

Prefer code checks where a rule works. Husain and Shankar advise favoring “assertions or other deterministic checks” over LLM judges in CI (Husain and Shankar, Evals FAQ, 2026-09-18).

What is a practical checklist?

Work through these steps in order. Each step makes the next one cheaper, and each one gives you something you can measure.

1. Collect 50 or more real outputs from the workflow. Mix easy and hard cases. (The count is a starting point for review, not a statistical rule.)
2. Have the expert label each as pass or fail, with one written reason.
3. Group the reasons into failure types. Turn each type into a yes/no rule.
4. Lock the output format with a schema. Add two or three examples to the prompt.
5. Run the workflow several times on the same inputs. Compare verdicts, not wording.
6. Have a second reviewer label a sample. Compute agreement. Fix rule wording where they split.
7. If an LLM judge scores outputs, check it against the expert labels. See how to build an LLM judge.
8. Keep the labeled set as a regression gate. Re-run it on every prompt or model change.

For the wider method, read how to evaluate an AI agent and the agent evaluation framework guide.

FAQ

Does temperature 0 make output the same every time?

No. In the Thinking Machines test, a model at temperature 0 produced 80 unique completions in 1,000 samples. Server batching changes the numeric path, so wording can differ.

Should I use JSON mode or structured outputs?

Use structured outputs where available. OpenAI states that JSON mode gives valid JSON but does not check your schema. Structured outputs enforce it.

Why do two reviewers grade the same output differently?

The rubric is usually loose, or the scale has too many points. Binary labels with a written reason reduce drift. Measure agreement and revise the rule wording where reviewers split.

How do I know when the workflow is consistent enough?

Define a bar before you test: for example, a minimum agreement score between reviewers and a maximum verdict change across repeated runs. No source gives a universal number, so set it with your experts.

How Clarity can help

Clarity by Epistemic Me is an AI-native services firm and an official partner of Parlance Labs. Clarity offers evals consulting for AI agents: error analysis, expert rubrics, golden datasets, LLM-as-judge checked against expert labels and regression gates. Lynn Chyi, AI Experience Architect at Dayforce, said the work improved “our human expert alignment scores by nearly 60%” (Dayforce case study). Engagements start at $100K+, with scope agreed in the proposal. See /evals/ and how to evaluate an AI agent.

Sources

• Thinking Machines — Defeating Nondeterminism in LLM Inference (2025-09-10)
• Anthropic — Structured outputs (docs, accessed 2026-10-30)
• Anthropic — Increase output consistency (docs, accessed 2026-10-30)
• OpenAI — Structured Outputs guide (docs, accessed 2026-10-30)
• Hamel Husain and Shreya Shankar — Frequently Asked Questions about AI Evals (2026-09-18)
• Hamel Husain — Creating a LLM-as-a-Judge That Drives Business Results (2024-10-29)
• Shankar et al. — Who Validates the Validators? (arXiv, 2024-04-18)
• scikit-learn — cohen_kappa_score (docs, accessed 2026-10-30)