Ask an LLM judge to score your RAG agent's answer and it will hand you a number. A 7. A 4.2. A crisp green 0.85. That number feels like signal. It is actually three different failures averaged into mush, and the average is engineered to hide the one you most need to see.
I'll make the case with a concrete .NET reference implementation (maf-evals), a three-tier agent evaluation built on Microsoft Agent Framework and .NET 8. Every number below came from running it, not from reasoning about it.
The problem with a single score
A retrieval-augmented answer can be wrong in at least three independent ways:
- The retrieval was bad — wrong documents came back, or the knowledge base doesn't cover the question.
- The groundedness was bad — the retrieved documents were fine, but the answer made claims they don't support. That's a hallucination.
- The relevance was bad — the answer is perfectly grounded in real documents and still doesn't answer what the user asked.
These have nothing to do with each other. A great retrieval can feed a hallucinated answer. A flawlessly grounded answer can miss the question entirely. When you collapse them into one "quality" score, a mediocre-everywhere answer and a perfect-retrieval-but-hallucinating answer can land on the same 3.5. You cannot tell which knob to turn, because the score was designed to be indifferent to which knob is broken.
So the maf-evals RAG triad scores all three separately, and never averages them:
| Score | Isolates |
| Retrieval | a bad knowledge base or a bad query |
| Groundedness | claims the retrieved context doesn't support |
| Relevance | well-grounded answers that miss the question |
Why the average actively lies
Here's the part that turned my opinion from "nice to have" to "non-negotiable." When you calibrate these judges against human labels — and you must calibrate them, a threshold picked without calibration is just taste — the Groundedness judge produces this:
| Score | Exact | Within 1 | MAE | Bias | Same band |
| Groundedness | 42% | 67% | 1.17 | -0.17 | 75% |
Look at that bias: -0.17. Nearly zero. If you were tracking a single blended quality score, that -0.17 would tell you the judge is essentially unbiased and calibrated. Ship it.
It is not calibrated. The Groundedness judge fails in two opposite directions at once. It scores outright fabrication at exactly 3.0 every time — a hallucination sails through as a mild warning — and it penalises well-grounded answers for being slightly off topic. The two errors point opposite ways, so they cancel. The near-zero bias is the average of a systematic over-score and a systematic under-score, and it looks healthy precisely because both are broken.
The metrics that expose it are the ones the average destroys: a mean absolute error of 1.17 (over a 1–5 scale, that's enormous) and band agreement of only 75%. You only see those if you refuse to blend.
That single insight — that fabrication was being scored 3.0 — is why the Groundedness floor moved from 3.0 to 3.5. At 3.0, every hallucination slipped through as a warning. Moving the floor to 3.5 lifted band agreement from 50% to 75%. A blended score would never have surfaced the problem to move the floor over.
Deterministic first, judge last
The triad isn't the first thing that runs. In the pull-request gate (Tier 2), five checks run cheapest-first, and the judge is last:
- Rules — the same rule engine the live agent uses, so a rule can't drift between production and CI.
- Retrieval — did the expected document IDs come back? Exact, free, no model involved.
- Tool calls — right tool, right arguments? Compared, not judged.
- Meaning — embeddings, for cases where wording is free to vary. Deterministic, ~1000x cheaper than a judge.
- RAG triad — the judge, and only now.
The ordering is doctrine, not convenience. Evidence ranks on an independence axis — independent to corruptible — not a cost axis. A document-ID match is independent: it means the same thing every run. A judge score is corruptible: it moves between runs. So the exact checks run first and always block, while the judge scores get two thresholds — a floor that blocks and a target that warns — because a number that wobbles has no business being a hard gate on its own.
This is why Retrieval scoring is advisory in maf-evals, even though it's part of the triad. Calibration caught the judge returning 5, 2, 4, 5, 2 for the same input — 17% of cases would flip a merge decision at random. So expectedChunkIds, an exact and free check, does the actual gating. The judge only annotates.
What it looks like in code
A golden case declares the deterministic expectation right next to the judged one. Gating lives in the exact fields; the triad adds color:
{
"id": "refund-within-limit",
"query": "Order A-31905 arrived damaged. Please refund me 120 for it.",
"expectedChunkIds": ["refunds#3"],
"expectedToolCalls": [
{ "name": "issue_refund", "arguments": { "orderId": "A-31905", "amount": 120 } }
],
"semanticExpectations": [
{ "name": "confirms_refund", "anyOf": ["Your refund of 120 has been issued."], "minSimilarity": 0.55 }
]