Here’s a funFor some non-universal definition of fun. interview question. Imagine you’re running Qwen3.6-35B-A3B at batch size $1$, on a single GPU, and you decide you want to increase throughput. For whatever reason, your engine can only work on $2$ tokens at a time. You have two choicesAssume the draft model's forward pass is free. Assume you don't have to prefill anything to batch. Assume that the sequences are short enough that the KV cache movement doesn't come into play.:

1. Run at batch size $2$ by batching $2$ random user sequences together.
2. Run at batch size $1$, speculating $1$ token ahead — so the verify works on $2$ positions, the token you just sampled plus one draft — with a per-token acceptance rate $\alpha$.

Which is better, assuming you only care about the total number of tokens being output per second?

Here’s a sensible answer:

Assuming that everything is memory bound, batching is always better for $\alpha<1$

But here’s something that comes out when you do the modelling:

Spending your 2 positions on one speculating sequence produces, globally, more output tokens per second than spending them on a batch of 2, even with $\alpha=0.9$

How come?

To find out, we ought to look at the dataCollected from our last post: half a million draft rounds for each of our two draft models, recording the drafter's own per-depth confidence and the number of tokens actually committed, plus separate captures of which experts every token routed through -- and the same routing captures for DeepSeek-V4-Flash (all published as specdec-calibration).. The answer lives in MoE routing. MoE routing is a weird part of performance analysis of LLMs: one of the places where the semantic content of the data affects what work gets done. In principle, it can do confusing things, like make benchmarks on random data unrepresentative.

First, the empirical distribution of routed expertsPreviously, we've just assumed uniform routing, which gives us coupon-collector maths:

Surprisingly non-uniform! Fitting the rank-vs-share curve, it decays roughly exponentially with rank: the busiest expert pulls several times its fair shareI wonder if there's a proxy metric in here for the data distribution: given pretraining with expert load balancing loss, can we determine the distribution of the training data by how balanced the experts are on different types of data?. It varies by domain, by model and by layer.

This doesn’t by itself explain anything. But — let’s look at the difference between the two choices on the table when we decide between width and depth: work on two randomly chosen tokens, or on two tokens that follow one from the other:

The distinct experts one verify forward touches as $N$ grows, three ways: $N$ separate sequences (width), one sequence running $N$ consecutive positions (depth), and the uniform coupon-collector.

So there’s the answer. At batch size 1 we’re memory-bound, and verifying a two-position speculative run moves less expert weight than adding a second sequence would. It does so by co-activation — speculated runs are more similar than randomly chosen data, so they activate more of the same expertsJosh did some great work on trying to recover co-activation for batching here.. Even throwing away 10% of speculated tokens at $\alpha = 0.9$

This is a toy problem, though it’s interesting to think about how we could make use of the insightIn real engines, it gets washed out by the cost of the draft model. It comes back if you do long sequences, where you're bound on KV read, but then controlling across depth and width in order to tell the story here gets hard.. In practice, we run at larger batch sizes, expert activations saturate across the batch, and the effect ought to wash out. But the $\alpha$ tax doesn’t: every speculated token still risks rejection. Do we have to pay it on every sequence equally — or can we spend depth only where it’s likely to pay off?

Batches formed over the life of the inference engine are not homogeneous: some decode requests are much easier to speculate from than others. If we can get some sense of how confident the drafter is, we can spend depth only where we need it, saving both drafterFor drafter systems that propose tokens sequentially, like EAGLE. With DFlash you get all the tokens regardless. and verifier compute — savings we can spend speculating further elsewhere, or on adding more sequences to the batch.

The first question is how much variation there is across sequences in how many tokens get accepted — if there’s none, this isn’t worth doing at all. It turns out there’s a lot. The number of committed tokens isn’t clustered around the mean: on a given round the drafter tends to be right about almost everything it drafts, or wrong almost immediately. Rounds split between ‘mostly misses’ and ‘clean sweeps’One question to explore: is this spike at the maximum draft length representative of the fact that the model could have gone further, but wasn't able to?:

A single fixed depth across the batch has to compromise between those two populations, deep enough to cash in the easy rounds, shallow enough not to waste too much verification time on requests that will be almost all rejections.

In fact, the drafter already has an opinion about which kind of round it is before the verifier runs. Below, we plot the accept length the drafter’s own per-depth confidences imply — the sum of their cumulative products — against the total length that actually committed. The diagonal is ‘perfectly calibrated’: i.e. the empirical accepted tokens are the same as the drafter’s confidences predicted. Below the diagonal, the drafter was overconfident; above, not confident enough.

There’s lots of blue along the diagonal, and that’s free signal we could be using to improve our drafting policy.

How can we do that?

Write round $i$‘s expected committed tokens at depth $\gamma$ from its own per-depth confidences $a_k^i$