A 35B MoE runs 3× faster than a 27B dense model on the DGX Spark

If you have a DGX Spark and you want a capable model on it, you have a choice to make. A mixture-of-experts model activates a fraction of its parameters per token. A dense model of similar size activates all of them. On paper the MoE should win on speed and lose on quality-per-parameter.

The question is how big that speed gap actually is on this specific machine — because the GB10’s defining characteristic is 121 GB of unified memory, and unified memory changes which bottleneck you hit first.

The answer turned out to be larger than I expected, and the more interesting finding wasn’t the headline number at all.

What’s being compared

Both models are served by the same vLLM build on the same box, one at a time. They share a VRAM group and are mutually exclusive on the GPU, so a lifecycle sidecar swaps them on demand — which means every comparison here is genuinely sequential rather than two models contending for the same memory.

The 27B is served FP8 but advertised under its bf16 model name, so clients need no change. Both run MTP speculative decoding, which drafts tokens and has the full model verify them — lossless, since any rejected draft is simply discarded.

Hardware and software, recorded automatically with every run:

• NVIDIA GB10, 121 GB unified memory, driver 580.159.03, compute capability 12.1
• Ubuntu 24.04, kernel 6.17.0-1018-nvidia, aarch64
• vLLM 0.19.0+6bc3197f.nv26.04.48680843, image nvcr.io/nvidia/vllm:26.04-py3

Four workloads, five measured repetitions each after a discarded warmup, greedy decoding, medians reported. Actual prompt lengths were 111, 364, 5,378 and 10,714 tokens.

Results

Decode throughput barely moves with context length for either model:

Standard deviation across repetitions was between 0.01 and 0.09 tok/s. These are not noisy measurements — the machine was idle, and the harness refuses to run at all if another project has work in flight.

Why the gap is 3× and not 4.5×

Decode is memory-bandwidth-bound. Producing one output token means reading the active weights out of memory, and on a machine like this that read dominates everything else.

Count the bytes. The 35B-A3B activates roughly 3B parameters per token in its routed feed-forward layers; at bf16 that’s about 6 GB. The dense 27B reads all 27B parameters; at FP8 that’s about 27 GB. Naively that predicts a 4.5× advantage for the MoE.

Observed was 2.9×. The difference is where the interesting detail lives:

“3B active” undercounts. In a MoE, only the feed-forward experts are sparse. Attention, embeddings, layer norms and the router itself are dense and get read for every token in both models. The real bytes-per-token figure is meaningfully above 6 GB, which compresses the ratio.

Sparse reads are less efficient than dense ones. Sweeping 27 GB of contiguous weights is close to the best case for a memory subsystem. Gathering a token-dependent subset of experts is a scattered access pattern that achieves lower effective bandwidth per byte moved. The MoE moves less data but moves it worse.

Both effects push in the same direction, and together they explain most of the distance between 4.5× and 2.9×.

Worth noting that MTP is active on both models and complicates the arithmetic — speculative decoding means the weight read per output token is lower than per forward pass, by a factor that depends on how often drafts are accepted. That acceptance rate is not identical between these two models, so the clean bytes-per-token calculation above should be read as an approximation that explains the shape of the result, not a precise derivation of it.

The finding I didn’t expect

Decode was the headline, but prefill is where something genuinely interesting happens. The MoE’s prefill advantage grows with prompt length.

Two things are happening at once.

First, absolute prefill throughput climbs steeply with prompt length for both models — the 27B goes from 312 to 1,695 tok/s, the MoE from 412 to 3,741. At 111 tokens there simply isn’t enough work to fill the GPU, and fixed per-request overhead dominates. By a few thousand tokens both models are doing real work and the hardware is busy.

Second, the ratio between them roughly doubles over that same range, then flattens. At short prompts the two models look almost equivalent on time-to-first-token, because you’re mostly measuring overhead that neither architecture affects. Once genuine prefill compute dominates, the MoE’s sparsity shows up and the gap settles around 2.2×.

The practical consequence: if you benchmark with short prompts, you will conclude these models are far closer than they are. A single 100-token test prompt reports a 1.3× difference on TTFT. Long-document and RAG workloads see 2.2×. This is the specific reason the scenario set here spans 111 to 10,714 tokens rather than picking one convenient prompt.

What running these actually costs

Throughput numbers hide three operational facts that matter more day to day.

They cannot co-reside. Both models are configured at --gpu-memory-utilization 0.65, so only one is resident at a time. A lifecycle sidecar swaps them on demand. Measured time from “a client asks for this model” to “the endpoint answers”:

Two things there surprised me.

Eviction is nearly free. I expected tearing down a resident 70 GB model to dominate a swap. The difference between loading the 27B into an empty GPU and loading it while the 35B is resident is about five seconds — inside the noise of a single measurement. Whatever the sidecar’s docker stop costs, it is not what you wait for.

The smaller model loads slower. The FP8 27B is roughly a third the size of the bf16 35B on disk and takes nearly two minutes longer to become servable. Weight loading is evidently not the bottleneck; engine initialisation is, and FP8 apparently has more of it to do. I have not pinned down which stage accounts for the difference, and I would rather say that than guess.

Either way the practical advice holds, just for a different reason than I first assumed: the cost of switching is the cost of loading whatever you are switching to, not the cost of clearing what was there. Seven to nine minutes per swap dwarfs the throughput difference between these two. Pick one per session.

These are single measurements, not distributions. Treat them as the right order of magnitude rather than precise constants.

The 27B is not the model its name implies. It is served as Qwen/Qwen3.6-27B-FP8 but advertised under the bf16 name, so clients need no change — and so nothing in an API response tells you which precision you are talking to. Worth knowing before you compare your numbers to anyone else’s.

Benchmark harnesses can lie about reasoning models. While measuring quality for the section below, lm-evaluation-harness scored this 27B at 13% on MMLU-Pro against a published 86.2. Nothing was wrong with the model. The task config stops generation at the string "Question:", which is harmless for completion-style models — but Qwen3.6 writes a **Question:** heading inside its own chain-of-thought, tripping the stop about sixty characters in, long before the the answer is (X) line the grader needs. Every response scored zero, silently.

Overriding the stop sequence: 13.3% → 86.7%.

That failure mode looks exactly like a bad model or a botched quantization, and there is nothing in the output to suggest otherwise. It deserves its own post; for now, treat any surprising benchmark result as a bug in your setup until you have read the raw responses.

Does the 27B earn its 3× slowdown?

The 27B is documented as the stronger model on coding and reasoning. That is the entire case for running it, since it loses on every speed axis. So I measured it two ways.

A custom 30-item suite (math, programming, verbal, objectively graded) could not separate them:

A 4B model landing within two items of a 27B means the suite is saturated, not that the models are equivalent. The 27B’s one-item lead over the 35B is noise. I am reporting this because a null result from an underpowered test is still worth stating — and because it is the honest reason I went looking for a harder measurement.

MMLU-Pro, via lm-evaluation-harness, 5-shot chain-of-thought, greedy, 280 questions sampled evenly across all fourteen categories — the same harness and the same questions for both models:

A 0.7-point gap, with a 95% confidence interval on the difference of [−5.6, +7.0] points. These two models are statistically indistinguishable on this benchmark — and one of them is three times faster.

That is the answer to the question this post opened with. The dense 27B is documented as the stronger model, and on a general reasoning benchmark run under identical conditions, that advantage does not show up at all.

Two limits on how far that conclusion stretches. The interval is wide: at 280 questions per model, a real difference of up to about six points in either direction cannot be ruled out. Narrowing it to something like two points would need roughly 700 questions per arm — about four hours per model at these speeds. And MMLU-Pro is not a coding benchmark, while coding is precisely where the 27B’s advantage is claimed. This measures general knowledge and reasoning, so it cannot rule out a real edge on SWE-bench-shaped work.

Both models also landed near 82%, below the 86.2 published for the 27B. That gap is almost certainly the harness rather than the models: vendor numbers come from their own scaffolding at temperature 1.0, while this is lm-eval’s prompt at temperature 0. The useful signal here is the comparison between two models measured identically, not either number against a model card.

One footnote worth recording, because it cuts against the headline. These evaluations ran with eight concurrent requests, and the 35B finished 280 questions in 72.7 minutes against the 27B’s 86.1 — only about 1.2× faster, not 3×. Single-stream decode and batched throughput are different quantities, and vLLM’s batching narrows the gap considerably. The two runs did not generate identical token volumes, so treat that ratio as suggestive rather than measured. But if your workload is batch rather than interactive, the speed case for the MoE is weaker than the headline suggests.

What I’d actually run

Default to the 35B-A3B. Three times the decode throughput, better prefill at every prompt length, and it holds that lead across short chat, sustained code generation, and long-context work alike. There is no workload in this set where the dense model is faster.

The case for the 27B rests on output quality, and measured under identical conditions it did not appear. Same harness, same 280 MMLU-Pro questions: 82.1% against 81.4%, a gap far inside the noise. The dense model costs you three times the decode speed and returns nothing this benchmark can detect.

That does not prove the 27B is never better. The confidence interval is wide enough to hide a few points, and MMLU-Pro does not test the coding work where its advantage is claimed. But it does move the burden of proof. If you are going to run at a third the speed, you should be able to point at a specific task where you have watched the 27B win — not at a model card.

The trade is steeper than the raw ratio suggests, too. Three times slower is the difference between a coding agent that keeps pace with you and one you wait on, and the seven-to-nine-minute swap cost means you cannot cheaply switch per task either. You are choosing for a session, not a prompt.

My default is the 35B-A3B. I will reach for the 27B when I have a coding workload where I have measured it winning, and not before.

Methodology, and one honest caveat

Every number here comes from a recorded run in this site’s repository; the tables above are rendered from those files at build time rather than typed in by hand.

The obvious objection to this comparison is that it changes two variables at once: architecture and quantization. The MoE is bf16, the dense model is FP8. That’s a fair criticism of it as a controlled experiment.

It doesn’t undermine the conclusion, though, and it’s worth being precise about why. FP8 is the faster format on this hardware — roughly half the bytes to move per parameter. So the confound works against the model that won. The 35B-A3B is three times faster while carrying the slower numeric format. A like-for-like comparison would widen the gap, not close it.

What this is, then, is not a clean architecture experiment. It’s a measurement of the two capable-tier options as they are actually served — which is the question you have if you’re deciding what to run tonight.

Two limits worth stating plainly. The speed numbers are single-stream: one request at a time, which is the realistic local-inference case, but not how the machine behaves under load. The quality runs, which used eight concurrent requests, are the hint at that difference — a 3× single-stream decode advantage showed up as roughly 1.2× in wall-clock there. If you serve several users at once, benchmark that case yourself; this post does not measure it.

And these reflect specific model, driver and vLLM versions, all recorded in the run files. Driver 580.159.03 in particular — a rerun after the next update can legitimately produce different numbers, which is exactly why the version is recorded rather than assumed.

Source data: e52e7b7f0b92f9ab6c7d994b