I benchmarked a vision-language model and scored it at 0.31.
The real number was 0.70. Same model, same weights, same hardware, same 100 questions. The only thing that changed was how I read its output.
I had already written up the 0.31 as a capability finding and concluded the model was unsuitable. That conclusion was wrong, and the failure was entirely in my harness. Here is the mistake, because I doubt I am the only one making it.
The setup
I was evaluating a batch of open-weight and frontier models on a multiple-choice benchmark: multi-view driving scenes, four options per question, one correct answer. Standard stuff. The prompt asked for reasoning followed by a final line, Answer: X.
My scoring code did the obvious thing:
m = re.search(r"Answer:\s*([A-D])", output)
pred = m.group(1) if m else None # None scores as wrong
That last comment is the bug.
What actually happened
The model I was testing is a "thinking" model. It emits a long internal reasoning trace before it commits to an answer. I had a generation budget of 1024 tokens.
On easy questions it reasoned briefly, emitted Answer: B, and scored fine. On hard questions it reasoned at length, hit the token cap mid-thought, and never emitted the answer line at all.
So the harness scored every one of those as wrong.
64 of 100 questions returned no parseable answer. Zero of those were image-loading errors or crashes. They were all truncation. And the truncation was not random:
Uncertainty 0/8 answered
Counterfactual 0/3 answered
Safety-critical Planning 1/11 answered
Safety-critical Prediction 3/12 answered
Look at that distribution. The questions the model failed to answer were precisely the questions that required the most reasoning. My harness was systematically discarding the model's performance on exactly the hard subset I was trying to measure, and reporting the result as a capability ceiling.
Of the 36 it did answer, it got 86% right. The model was fine. My measurement was garbage.
The fix
Stop parsing free text. Constrain the decoding to a schema.
With Ollama:
resp = ollama.chat(
model="my-vlm:9b",
messages=[{"role": "user", "content": prompt, "images": imgs}],
format={ # enforced at decode time
"type": "object",
"properties": {"answer": {"type": "string", "enum": ["A","B","C","D"]}},
"required": ["answer"],
},
think