Compute budgets are not training budgets. A scaling law says how much compute a target loss needs, not how long the hardware takes to deliver it or what that costs. This study puts both on the same bill. It trains three shapes at the same parameter budget and on the same data, then measures two things separately: how much each shape learns per token, and how quickly the hardware processes those tokens. What happens when the shape that learns more from each token is not the shape that processes tokens fastest?
In these runs the deepest model learns the most per token, but the widest runs 20% faster. That throughput advantage outweighs the token advantage, making the deep model 14% more expensive to train to the same target loss.
Three metrics, two winners
Three shapes matched at roughly 400M block parameters, three seeds each, 8 billion tokens each, learning rate swept per shape. Then the same three models timed on the same single machine that trained them.
Same three models, same cluster, three ways of ranking them. Bars are scaled to the largest value in each panel.
The chart below shows the same runs on the same cluster three ways. With tokens on the horizontal axis the deep model is ahead. With hours or dollars on it, the wide model is.
Same three shapes on the same machine, drawn against tokens, wall-clock hours, or money. The shaded band around each line is the range across its three seeds, and the ticks on the target line mark where each seed crossed it. Nothing about the models or the cluster changes between the three views. Only the unit on the axis does. Curves stop at the target.
The models here are roughly 400M parameters, which may look small. Small runs are what scaling laws are for: their whole purpose is to predict what happens at far more compute from what happens at less, and the effect of shape on loss is part of what Kaplan fitted. The loss side of this comparison should carry to much larger models. The speed side is what changes, and it changes toward shape mattering more: past one node, choosing a shape becomes a parallelism problem (see What to take from it).
With three seeds per shape, every deep run reached the target before every wide run, and every wide run before every balanced run. For scale, the gap between deep and wide is about six times the seed spread, whether measured in tokens to target or in loss at equal tokens. A gap that size cannot be put down to run-to-run variation. How closely the three shapes are matched is shown below, and the loss comparison rests on it.
| shape | width×layers | blocks* | attention | embedding | total |
|---|---|---|---|---|---|
| deep | 1024×32 | 398.5M | 134.2M | 51.5M | 450.0M |
| balanced | 1536×12 | 396.4M | 113.2M | 77.3M | 473.6M |
| wide | 2048×8 | 398.5M | 134.2M | 103.0M | 501.5M |
* Block parameters are the attention and feed-forward weights, the count Kaplan's scaling laws use. The tied embedding table is left out because at a fixed vocabulary it grows with width and cannot be matched alongside the blocks, and because Kaplan's data suggest it buys less loss than block parameters do, though that came from one setup and need not hold everywhere. Counting the table, the models are 450M, 474M and 501M. The extra parameters sit in the wide model, the shape that loses on loss per token, so the exclusion did not manufacture the ordering.
Why the bill disagrees
Loss falls fast early and slowly late, so near the end a small difference in loss is a large difference in tokens. That is how a loss gap under half a percent becomes a token gap you can see on the bill. But the token gap is still smaller than the speed gap, and the speed gap decides it.
- Exchange rate
- near the target, 1% of loss is worth about 12% of tokens
- Loss gap
- 0.39% at equal tokens, read where deep reaches 2.85
- Token gap
- 0.39% × 12 ≈ 4.7% implied, 4.4% measured
- Speed gap
- 20%, wide over deep
- Bill
- the speed gap wins, so deep costs 14% more
The whole inversion is one of those two gaps beating the other, which is why measuring only one of them gets the answer wrong.
Both halves of this have been reported before, separately. Kaplan and colleagues found that loss changes only a few percent across width-to-depth ratios at a fixed block parameter count, and Levine and colleagues find the best ratio drifting wider as models grow. Hardware studies and vendor guidance (Anthony and colleagues and NVIDIA) report large throughput differences between shapes. On one bill, a small loss effect meets a large speed effect, and here the speed effect wins.
Why is the wide model faster? The gap is 20%. Arithmetic explains 2.7% of it and utilization the rest, 16.4%. Attention scales with layers times width, and deep has twice that product, but wide's wider logits matmul costs it nearly as much back, so deep ends up doing only 2.7% more arithmetic per token. Utilization is the larger term because the memory-bound parts of a layer, the normalizations, residual additions and activations, also scale with layers times width and run far below matrix-multiply speed, and deep launches about four times as many kernels per step. Deep's 1024-wide matrices also tile less efficiently on an H100 than 2048-wide ones. That last effect could shrink at larger widths. The memory-traffic one travels to any accelerator.
The same comparison across the full usable range of targets:
Cost to reach each target loss, for all three shapes, read off the same curves. The deep model is the most expensive everywhere in the usable range. The hatched band covers targets below 2.80, where the target stops being measurable: the last shape to arrive is inside the final tenth of its learning-rate schedule, where the rate has decayed toward zero, the curve has gone flat, and a token count there is mostly noise. Curves are dashed inside it.
The back-of-envelope mistakes people make
Compute-optimal scaling tells you how many FLOPs to spend, not how many hours that takes. Getting from FLOPs to hours takes two more numbers, and the standard guess for each one ignores shape.
- FLOPs per token. The shortcut is six per parameter and it leaves attention out. Attention work, counted the way PaLM and Chinchilla count it, scales with layers times width, so the shortcut undercounts the deep model by 30% and the wide one by 13%, a 15% blind spot between shapes.
- Utilization, or MFU: the fraction of the chip's peak you actually sustain. The stock guess is 40%. Measured, deep reaches 42.0% and wide 48.9%, so wide sustains 16% more of peak. That is on one node with no parallelism in the way. Past one node, tensor parallelism splits width and pipeline parallelism splits depth, each with its own overhead, and the gap starts to depend on the layout as well as the model.
Both guesses are wrong by a different amount for each shape, and both are wrong in the wide model's favor. The FLOP shortcut skips attention, and the deep model does twice the attention work, so its cost is undercounted the most. The 40% utilization guess is close for the deep model but well under what the wide one actually reaches. A budget built on those two numbers therefore predicts the deep run 24% too fast, the balanced one 2% too fast and the wide one 7% too slow. Two of the three land near the budget because their errors nearly cancel, not because the budget accounted for them. Same formula, three shapes, a 33% spread it cannot see. The correction needs no loss measurements.
Kaplan's approximation treats attention as negligible when width is large relative to context. The ratio that measures it, context over twelve times width, is 0.17 for the deep model here. That is a sixth by Kaplan's own formula, which counts the attention term at half the rate used on this page, and about 30% by the full count. Either way it is not nothing. The table below shows the error growing with context and growing fastest for the deep shape: the longer the sequence, the more shape-blind the shortcut becomes.
| context | deep | balanced | wide | gap in the error | gap in true FLOPs |
|---|---|---|---|---|---|
| 2,048 | 1.30× | 1.16× | 1.13× | 15% | 2.7% |
| 8,192 | 2.19× | 1.64× | 1.54× | 43% | 28% |
| 32,768 | 5.77× | 3.55× | 3.14× | 84% | 65% |
| shape | width×layers | block params | final loss | tokens to 2.85 | tok/s | MFU | time | cost |
|---|---|---|---|---|---|---|---|---|
| deep | 1024×32 | 398M | 2.773 ± 0.002 | 5.15 ± 0.04B | 0.95M | 42.0% | 1.51 ± 0.01 h | $95.56 |
| balanced | 1536×12 | 396M | 2.793 ± 0.002 | 5.57 ± 0.04B | 1.09M | 45.5% | 1.42 ± 0.01 h | $89.70 |
| wide | 2048×8 | 398M | 2.791 ± 0.002 | 5.38 ± 0.03B | 1.13M | 48.9% | 1.32 ± 0.01 h | $83.48 |
What to take from it
A parameter-constrained model can learn more from every token and still cost more to train, because shape sets both the tokens a run needs and the rate the hardware delivers them, and a budget that tracks only one of those is incomplete. Here the wide shape is cheaper. At another scale, context length or target loss the balance can move. What decides it is the cost of reaching the target, which neither loss per token nor throughput gives on its own.
The back-of-the-envelope estimates get worse as models grow. The shortcut's blind spot between these two shapes is 15% at 2k context and 84% at 32k, by arithmetic alone. Past one node the same thing happens to utilization. Tensor parallelism splits width and pipeline parallelism splits depth, each with its own communication cost, so the rate at which the hardware delivers tokens depends on the shape in ways a flat guess cannot express. At that scale the parallelism plan stops being a problem for performance engineers to solve after the shape is chosen. It becomes part of the formula that decides how the model scales and what model architecture to choose.
What this does not settle
- Experimental scope
- One model size, one node, three shapes, three seeds, with learning rates from a 1B-token sweep that may not be optimal at 8B. A more principled way to set them at scale would be μP, which transfers a learning rate tuned on a small proxy model to larger ones instead of re-sweeping. The point is that the flip exists, not which shape is best for any configuration, including this one.
- Architectural confounds
- Blocks are matched but two things are not. The tied embedding table grows with width, so wide carries 103M embedding parameters to deep's 52M and more parameters in total. And training in bf16 with no fp32 master copy freezes the RMSNorm gains at 1.0, which handicaps deep, the shape with the most norm layers. The first cuts in the result's favor: wide carried the extra parameters, and the extra work per token they cost, and still won on throughput. The second cuts against it: a deep model without the handicap would learn a little more per token and close some of the bill gap. Only a little, since the next linear layer absorbs the missing scale and no expressive capacity is lost.
Method
- Models
- Three decoder-only transformers at roughly 400M block parameters: deep 1024×32, balanced 1536×12, wide 2048×8. Head dimension 64, multi-head attention, gated feed-forward, so a block is 4d² + 3·d·ffn. Context 2,048, vocabulary 50,304, tied embeddings.
- Training
- 8B tokens per run, three seeds per shape, global batch 128 sequences, AdamW with a cosine schedule, bf16, replicated data parallelism on one 8×H100 node with
torch.compile. - Data
- Batches are keyed on seed and step only, so every shape sees identical data.
- Learning rates
- From a shared six-rate grid at 1B tokens: 1.25e-03 deep, 1.12e-03 balanced, 8.60e-04 wide.
- Throughput
- 40 timed steps after 10 warmup, three repeats per shape.
- Cost
- SageMaker on-demand training on one ml.p5.48xlarge in us-west-2 at $63.30 an hour, the September 2026 price from the AWS Pricing API.
- Target
- 2.85, chosen where all three curves still have slope.
- Code
- The code and the run journals are at github.com/manojtumu/wide-or-deep, MIT licensed.