Evaluating LLM Apps in Java

Introduction

Building Reliable LLM Applications in Java put it plainly: treat model output as a hypothesis to verify, not a fact to trust. Testing Best Practices in Java put the same discipline in JUnit terms: a suite only earns trust by asserting the right things at the right level, unhappy paths included. This post is where those two ideas meet — a JUnit test either passes or fails against a fixed expected value; an LLM's output is a paragraph of prose that might be right in spirit while differing token-for-token from anything you wrote down in advance. Evaluating it takes a harness, not an assertEquals.

That harness has three parts: a golden dataset of representative cases with known-good expected behavior, scoring that turns each case into a pass/fail or a number, and regression testing that runs the harness on every change and fails the build when the score drops. Making RAG Accurate in Java already gave you half of this story — recall@k, precision@k, MRR, nDCG measure whether retrieval found the right chunks. This post measures the other half: whether the generated answer built from those chunks is actually good, which is a genuinely different question a retrieval metric can't answer on its own. Everything below is illustrative, non-executed Java, grounded in the same Anthropic Java SDK shapes as posts 10/11.

The Golden Dataset: Curating Cases, Not Just Inputs

A golden dataset is a small, hand-curated set of (input, expected behavior) pairs that represents the ways your application is actually used — not a random sample, and not just the cases that already work. Each case needs enough structure to be scored automatically later:

public record EvalCase(
    String id,
    String category,          // "extraction", "qa", "summarization", ...
    String input,              // the prompt/question sent to the system under test
    String expectedExact,      // non-null only for cases scorable by exact/programmatic match
    List<String> mustContain,  // key facts a correct answer must mention (programmatic check)
    List<String> rubric        // criteria an LLM judge should apply (open-ended cases)
) {}

A single case carries expectedExact, mustContain, or rubric — never a mix — because each maps to a different scoring method below. A realistic set mixes all three:

List<EvalCase> goldenSet = List.of(
    new EvalCase("inv-001", "extraction",
        "Extract the total from: Invoice #4471, Acme Corp, Total Due: $1,240.00",
        "1240.00", List.of(), List.of()),

    new EvalCase("rag-014", "qa",
        "What is our refund window for unopened hardware?",
        null, List.of("30 days", "original packaging"), List.of()),

    new EvalCase("sum-032", "summarization",
        "Summarize this incident postmortem: <postmortem-text>",
        null, List.of(),
        List.of(
            "States the root cause in the first sentence",
            "Mentions the customer-facing impact and its duration",
            "Does not include internal Slack usernames or ticket IDs"))
);

Curate deliberately, don't just collect. A useful golden set covers: the common case, the edge cases that have actually broken before (every production incident is a candidate eval case), adversarial inputs (a retrieved chunk with injected instructions, a question with no good answer in context), and a few cases the system is expected to refuse or hedge on — a good eval set penalizes false confidence as much as it penalizes wrong answers. Keep it small enough to run in minutes (dozens to low hundreds of cases, not thousands) — a golden set you're too slow to re-run after every change stops being used.

Scoring, Method One: Exact and Programmatic Assertions

Whenever the expected output has a checkable shape, score it exactly the way you'd assert a unit test — no model needed to judge the judge:

public final class ProgrammaticScorer {

    public static boolean scoreExact(String actual, String expected) {
        return actual != null && actual.trim().equals(expected.trim());
    }

    public static boolean scoreContainsAll(String actual, List<String> mustContain) {
        String normalized = actual.toLowerCase();
        return mustContain.stream().allMatch(fact -> normalized.contains(fact.toLowerCase()));
    }
}

scoreExact fits the "extract this number" cases from Building Reliable LLM Applications in Java — structured output makes the field directly comparable. scoreContainsAll fits factual QA over retrieved context: it doesn't demand the exact wording, just that the required facts survived into the answer. Both are deterministic, free, and instant — always prefer them over a judge call when the expected behavior is checkable this way. Reach for LLM-as-judge only for what programmatic checks genuinely can't express: tone, completeness, whether a summary is faithful to its source rather than merely mentioning the right keywords.

Scoring, Method Two: LLM-as-Judge

For open-ended cases, have a second Claude call read the candidate answer against the rubric and return a structured verdict — the same "get typed output, don't parse prose" discipline post 11 applied to invoices, applied here to a scoring decision:

import com.anthropic.client.AnthropicClient