The Cold-Start Problem for Agent Evals: What to Gate on Day One With Zero Labeled Data

You just shipped an agent. It works in the demo. Now someone asks the reasonable question: "How do we know it keeps working?" And you reach for evals — and hit a wall. You have no labeled dataset. No golden outputs. No historical traces. Nothing to grade against.

So the team stalls. "We'll add evals once we collect data." Meanwhile the agent runs in production, ungated, and the first time it silently breaks is the first time anyone notices.

This is the cold-start problem, and the usual response — "just get an LLM to score the output 1-10" — is exactly the wrong instinct. You do not need labels to start gating. You need to understand which evidence you can trust on day one, and which you can't trust ever.

Independence, not cost

Most eval discussions rank checks on a cost axis: cheap string matches at the bottom, expensive model-as-judge at the top, as if spending more buys you more truth. That's backwards. The axis that matters is independence: can the agent forge this signal, or not?

That reframing is the whole game for cold-start, because independent evidence needs zero labels. It's true or false about the world regardless of what your agent intended.

Three tiers, ranked independent to corruptible:

• Tier 1 — externally observable proof the agent can't forge. Did it produce valid JSON? Does the file it claims to have written exist? Did the code compile? Did the tests pass? Did it finish inside the timeout? Is the output non-empty?
• Tier 2 — statistical signal against a baseline the agent didn't author. Is the output embedding-similar to the task it was given? Is the length sane, or did it collapse into repetition? Did the diff actually change anything?
• Tier 3 — model-as-judge. A shared-substrate opinion. A signal, never a verdict.

On day one you have Tier 1 and Tier 2 completely for free. Neither needs a single labeled example, because neither asks "is this good?" — they ask "is this real?"

The day-one gate

Here's a starter gate for an agent that's supposed to produce a code patch. No dataset required:

type GateResult = { pass: boolean; tier: 1 | 2; reason: string };

async function coldStartGate(
  task: string,
  output: { patch: string; targetFile: string },
  runtime: { durationMs: number; timeoutMs: number },
  embed: (s: string) => Promise<number[]>,
): Promise<GateResult[]> {
  const results: GateResult[] = [];

  // Tier 1 — unforgeable facts about the world
  results.push({
    tier: 1, reason: "non-empty output",
    pass: output.patch.trim().length > 0,
  });
  results.push({
    tier: 1, reason: "target file exists",
    pass: await fileExists(output.targetFile),
  });
  results.push({
    tier: 1, reason: "patch applies + compiles