How to Evaluate LLM Output: A Practical Rubric

"Does this look right" is not an evaluation process, it's a vibe check, and it stops working the moment more than one person is shipping changes to an LLM feature. A prompt edit that looks like an improvement on the five examples you tried by hand can quietly regress on the hundred cases you didn't. This post is a concrete rubric for scoring LLM output, a method for building the dataset you score it against, and a way to turn both into something you run on every change instead of when you remember to.

Why ad hoc review breaks down

Ad hoc review has three failure modes that all show up eventually. It doesn't scale: a human reading every output caps your iteration speed at however fast a person can read. It isn't consistent: the same reviewer will judge similar outputs differently depending on mood, time of day, and what they happened to notice first, and different reviewers disagree with each other even more. And it has no memory: without a fixed set of test cases, you can't tell whether a change made things better, worse, or just different, because you're not comparing against the same baseline each time.

Databricks' engineering team frames the starting point for fixing this plainly:

"The first step in evaluating an LLM is to use a dataset that is diverse, representative and unbiased." Databricks, "Best Practices and Methods for LLM Evaluation"

Everything below builds on that: a fixed dataset, a fixed rubric, and a repeatable way to run both.

A rubric that covers what actually breaks

Most LLM output failures fall into five dimensions. Score each one separately rather than reaching for a single overall quality number, because a single number hides which specific thing went wrong.

Rubric dimensions and how each is typically scored in production evaluation pipelines, following the metric categories described in Confident AI's RAG and LLM evaluation metrics guide.

A few notes on using this in practice. Format compliance and a chunk of safety checking should be deterministic wherever possible, a JSON schema validator or a regex either passes or it doesn't, and there's no reason to spend a model call deciding something a script can decide for free. Correctness, groundedness, and completeness are where judge models or human review earn their cost, because they require actual understanding of the content rather than pattern matching against a format. Score each dimension independently and record all five, not just a pass/fail on the worst one, so you can see whether a change traded one kind of failure for another instead of genuinely improving things.

Building a small golden dataset

A golden dataset is the fixed set of input/expected-output pairs you run this rubric against before shipping any meaningful change. It matters more than the choice of scoring method or tooling, because a rubric run against the wrong examples tells you nothing useful no matter how well-designed the rubric is.

Build it from three sources. Start with hand-written examples covering the cases you already know matter: the common request shapes, the tricky edge cases, the inputs that broke a previous version. Add real production samples, with any personal or sensitive data stripped out, because production traffic reliably surfaces inputs nobody thought to write by hand. Fill remaining gaps with synthetic variations, paraphrases of known-hard cases and adversarial phrasings, but review every synthetic example with the same scrutiny as a real one before it goes in, since an unreviewed bad example quietly degrades the dataset's usefulness.

Keep the dataset small enough to run often. A curated set in the low hundreds of examples, well-chosen, beats a sprawling set of thousands that takes too long to run and therefore doesn't get run before every change. Revisit it periodically: as your product changes and your users find new ways to use it, the dataset needs new examples added and stale ones reconsidered, or it slowly stops representing what your system actually faces.

Automated scoring vs. human review

Neither one replaces the other; they cover different parts of the cost-versus-fidelity tradeoff. LLM-as-a-judge is the practical way to apply your rubric at scale: a judge model scores each output against the rubric criteria above, cheaply enough to run on every candidate change against the full golden dataset. Its weakness is that a judge model has its own biases and blind spots, so it needs periodic calibration against real human judgment to confirm it's still measuring what you think it's measuring.

Reserve human review for what automated scoring can't confidently resolve: a sample of production traffic reviewed regularly regardless of whether anything looks wrong, any output the automated scorer flagged as borderline or low-confidence, and a periodic audit specifically checking whether the judge model's verdicts still track human judgment on a held-out set. This is the same layered structure that appears across most mature evaluation setups: deterministic checks first because they're free and certain, judge scoring next because it's cheap and scales, human review last because it's the most expensive and the most trustworthy.

This is also the shape of the checking that happens inline in a router rather than in a batch job. llm11's verification layer runs schema and groundedness checks on every response as it's generated, the same dimensions in the rubric above, and escalates automatically to a stronger model when a check fails, rather than waiting for an offline eval run to catch the problem after the fact. Batch evaluation and inline verification aren't competing approaches: batch eval is how you validate a change before it ships, inline verification is what catches the requests a batch eval didn't happen to sample.

Making it a repeatable process

A rubric and a dataset are only worth building if they run automatically, not when someone remembers. Wire the golden dataset run into your normal change process: on every prompt edit, model swap, or fine-tune, run the rubric across the full dataset and compare scores against the last known-good baseline, not just against an absolute threshold. Track scores over time per dimension, since a dimension that's been slowly drifting down for weeks is a different problem than one that just cratered after yesterday's change, and the trend is invisible if you only ever look at the latest run in isolation. Set an explicit bar for what counts as a regression worth blocking, most teams use a fixed threshold on any single dimension, so the process makes a decision rather than producing a report nobody acts on.

If you're doing this by hand today, the tooling landscape has moved fast enough that most of this loop, dataset versioning, judge scoring, regression tracking, is available off the shelf rather than something worth building yourself. Our overview of the current evaluation tooling landscape covers what's out there and what each platform actually automates versus what you still have to design yourself. If your system is specifically retrieval-augmented, the rubric above needs a companion set of metrics for retrieval quality; our RAG-specific evaluation guide covers that variant in depth.

Frequently asked questions

How many examples does a golden dataset need?

There's no fixed number, but a curated set in the low hundreds is a reasonable practical target for most teams: large enough to cover real variation, small enough to run before every change without becoming a chore. Coverage of edge cases and failure modes you actually know about matters more than raw size.

Can I use the same rubric for every LLM feature?

The five dimensions (correctness, groundedness, completeness, format compliance, safety) generalize across most text-generation features, but the specific criteria within each dimension need to be rewritten per feature. What counts as "complete" for a summarization task looks nothing like what counts as complete for a structured data extraction task.

How often should I re-run the evaluation?

Run it on every meaningful change to the prompt, model, or pipeline, not on a calendar schedule. A change that looks small, a one-line prompt tweak, a model version bump, can shift scores in ways that only show up when you actually measure them.

What's the difference between evaluation metrics and monitoring?

Evaluation runs against a fixed, known dataset before a change ships, so you can compare like against like. Monitoring watches live production traffic after the change is live, catching drift and edge cases the fixed dataset didn't anticipate. Mature setups need both.