When RAG Says Duplicate but the LLM Disagrees: Building an Adjudication Layer

The platform I work on generates training data through AI agents. Before each new task is created, a deduplication gate checks whether something similar already exists. The gate uses RAG — retrieval augmented generation — to embed the candidate task, search a vector index of existing tasks, and block generation when cosine similarity crosses a threshold. It is a sensible first line of defense against redundant training data.

It was also blocking valid, distinct tasks. Two tasks could share the same persona and task type but differ in subdomain context — for example, a retail softlines scenario versus a retail hardlines scenario — and still score above the duplicate threshold because their embeddings were nearly identical. The semantic overlap was real. The domain-specific distinction was not. A human reviewer would immediately see these as different tasks. The embedding model did not.

I added an LLM adjudication layer that gives RAG a second opinion on ambiguous matches. The change shipped behind a feature flag across ten files and roughly 1,300 lines of TypeScript, with 109 tests. This post walks through the false-positive problem, the two-threshold band architecture, and why I chose fail-closed semantics when the LLM is unavailable.

The false positive problem

RAG dedup works by embedding similarity. When a candidate task arrives at the pipeline gate, the system computes its vector representation, queries the index for top-K nearest neighbors, and compares the highest similarity score against a fixed crossThreshold (0.80 in our configuration). Above that line, the task is treated as a duplicate and generation stops.

The approach is fast and cheap. It also conflates "structurally similar" with "actually duplicate." Tasks that share a persona ("store manager"), a task type ("inventory reconciliation"), and broad domain framing ("retail operations") produce embeddings that cluster tightly in vector space. The subdomain context that makes them meaningfully different — softlines versus hardlines, apparel versus electronics — contributes less signal than the shared scaffolding.

Embedding similarity captures semantic overlap but misses domain-specific distinctions that humans recognize instantly. That gap is where false positives live.

We were seeing legitimate new tasks rejected at the gate. Operators would retry with slightly different wording and hit the same wall. The dedup system was doing its job too aggressively, and lowering the global threshold would have let real duplicates through. We needed a way to resolve ambiguity without abandoning the fast path for obvious non-matches.

Two-threshold band architecture

The fix is not "call an LLM on every dedup check." That would add 10–30 seconds of latency to every task submission and burn inference budget on cases where the answer is already obvious. Instead, I introduced a review band between two thresholds:

• Below llmClearThreshold (default 0.75): auto-clear. No LLM call. The match is weak enough that we trust RAG's negative signal.
• Between 0.75 and crossThreshold (0.75–0.80): LLM review required. RAG is uncertain — this is the ambiguous zone where embeddings lie.
• At or above crossThreshold (≥ 0.80): RAG says duplicate, but the LLM gets an override opportunity. A strong embedding match is not automatically a true duplicate.

The band means LLM latency is only paid for cases that actually need judgment. Clear non-matches and (after adjudication) clear non-duplicates never touch the model. The architecture looks like this in the dedup flow:

type DedupClearReason = 'below_threshold' | 'no_matches' | 'llm_cleared';

interface DedupVerdict {
  isDuplicate: boolean;
  topScore: number;
  clearReason?: DedupClearReason;
  llmAdjudication?: LlmAdjudication;
}

async function checkForDuplicate(
  candidate: TaskCandidate,
  config: DedupConfig,
): Promise<DedupVerdict> {
  const matches = await ragSearch(candidate, { topK: 5 });
  const topScore = matches[0]?.similarity ?? 0;

  if (matches.length === 0) {
    return { isDuplicate: false, topScore, clearReason: 'no_matches' };
  }

  if (!config.llmEnabled) {
    return {
      isDuplicate: topScore >= config.crossThreshold,
      topScore,
      clearReason: topScore < config.crossThreshold ? 'below_threshold' : undefined,
    };
  }

  if (topScore < config.llmClearThreshold) {
    return { isDuplicate: false, topScore, clearReason: 'below_threshold' };
  }

  // Review band or above crossThreshold — consult the LLM
  const adjudication = await adjudicateWithLlm(candidate, matches, config);
  return {
    isDuplicate: adjudication.isDuplicate,
    topScore,
    clearReason: adjudication.isDuplicate ? undefined : 'llm_cleared',
    llmAdjudication: adjudication,
  };
}

Notice the early returns. The LLM path is the exception, not the default. In production, the vast majority of candidates score below 0.75 and clear in a single RAG round-trip with zero added latency.

The LLM adjudicator module

Rather than build a new inference endpoint, I integrated with an already-deployed auto-rater service. That service exposes POST /api/v1/review with support for a custom_evaluation_prompt, which is exactly what structured dedup judgment needs. The adjudicator submits the candidate task plus the top-K RAG matches, then polls for the result — submit-and-poll, not synchronous blocking.

interface LlmAdjudication {
  isDuplicate: boolean;
  reason: string;
  confidence: number;
  durationMs: number;
}

async function adjudicateWithLlm(
  candidate: TaskCandidate,
  matches: RagMatch[],
  config: DedupConfig,
): Promise<LlmAdjudication> {
  const start = Date.now();

  const reviewId = await submitReview({
    endpoint: `${config.autoRaterBaseUrl}/api/v1/review`,
    custom_evaluation_prompt: buildDedupPrompt(candidate, matches),
    payload: { candidate, matches },
  });

  const result = await pollForResult(reviewId, {
    timeoutMs: config