Local LLM Labs

lm-eval scored a good model at 13% on MMLU-Pro. The model was fine.

dgx-spark

I ran MMLU-Pro against a locally served Qwen3.6-27B and it scored 13.3%. The model card says 86.2.

There is an obvious story to tell about that. It’s the story I nearly told: the model is served FP8-quantized, quantization degrades quality, and here is dramatic evidence of the cost. That would have been a good post. It would also have been completely wrong.

Nothing was wrong with the model. The benchmark harness was truncating every answer before the model could give it, and reporting the result as a wrong answer rather than as an error.

What the score looked like

Nothing about the output suggests a bug. lm-evaluation-harness prints a perfectly ordinary table:

|  Tasks   |Version|    Filter    |n-shot|  Metric   |   |Value |
|----------|------:|--------------|-----:|-----------|---|-----:|
|philosophy|    3.1|custom-extract|     5|exact_match|↑  |0.1333|

A number, a standard error, a metric name. If you were benchmarking an unknown model you would write it down and move on.

What was actually happening

I only found it because the number was too wrong. A four-point gap I would have believed. A seventy-point gap on a model that answers ordinary questions correctly when you talk to it directly meant something was broken.

Logging the raw responses showed six of eight completions ending at exactly 67 characters, all with identical text:

Here's a thinking process:

1.  **Analyze the User Input:**
   - **

Identical truncation across different questions is not a model behaviour. It’s a stop sequence.

lm-eval’s MMLU-Pro task config sets:

generation_kwargs:
  until:
    - "Question:"
  max_gen_toks: 2048

until: ["Question:"] is sensible for a completion-style model. Few-shot prompts are a run-on sequence of Question: ... Answer: ... blocks, and without a stop the model happily invents the next question and answers it too.

Qwen3.6 reasons before it answers, and it structures that reasoning with markdown headings. One of the first things it writes is:

1.  **Analyze the User Input:**
   - **Question:** Select the best translation into predicate logic...

There it is. The model writes **Question:** as a heading inside its own chain-of-thought, the stop sequence fires, and generation ends about sixty characters in — long before the the answer is (X) line the grader is looking for. The extraction regex finds nothing, scores the item zero, and the harness reports a confident 13.3%.

The two responses that survived were the ones whose reasoning happened not to use that heading.

The fix

Override the stop sequence with a string the model will never emit, and raise the token budget so long reasoning traces are not cut off either:

lm-eval run --model local-chat-completions \
  --model_args base_url=http://localhost:8004/v1/chat/completions,model=Qwen/Qwen3.6-27B \
  --apply_chat_template \
  --tasks mmlu_pro \
  --gen_kwargs until=ZZZNEVERZZZ,max_gen_toks=3072

Same model, same 15 questions, same everything else:

Configuration mmlu_pro_philosophy
Stock task config 13.3%
until overridden 86.7%

Extending the run to 280 questions across all fourteen MMLU-Pro categories put the model at 82.1% ± 2.3, which is in the neighbourhood of the published 86.2 — close enough that the remaining gap is explained by prompt scaffolding and sampling rather than anything alarming.

Why this is worth your attention

It is silent. There is no warning, no error, no finish_reason that looks unusual from the grader’s perspective. A truncated response is indistinguishable from a wrong answer once it reaches the scoring function.

It looks like a real finding. A 70-point drop invites explanation, and several plausible ones are available: quantization damage, a bad conversion, a broken chat template, an overrated model. Each of those is a post someone would read and believe. I was two paragraphs into drafting one.

It scales with how reasonable your model is. The better a model’s habit of structuring its reasoning with headings, the more often it trips the stop. A terser model would have scored higher on the same broken configuration. The harness effectively penalises models for writing organised chain-of-thought.

It is not specific to this task or this model. Any harness written for completion-style models can carry stop sequences that reasoning models trip over. until lists containing "Question:", "\n\n", "Q:" or similar are common, and chain-of-thought output is full of exactly those patterns.

What I’d do differently, generally

Read the raw responses before believing a surprising score. Not a sample of one — enough to see whether failures share a shape. Identical truncation points across different inputs is the tell here, and it is invisible in any aggregate metric.

--log_samples exists for this, and I now treat it as mandatory on the first run against any new model or task rather than something to reach for when results look odd. The cost is a few megabytes. The alternative, in this case, would have been publishing a confident and entirely fictional finding about FP8 quantization.

That instinct generalises past benchmarking. Any measurement pipeline can report a plausible number for an implausible reason, and aggregate metrics are specifically designed to hide the individual cases that would tell you.

Details, for anyone hitting this