Purpose
Google designed TPUs from the ground up to be exceptionally efficient for machine learning workloads, delivering top-tier performance per total cost of ownership (TCO). Because TPUs trade general-purpose hardware scheduling for wide systolic arrays, dedicated SparseCore engines, and custom interconnect fabrics, adhering to specific architectural best practices is essential to achieve maximum performance. To help developers get the most out of TPUs, we are launching the open-source AI-Hypercomputer/tpu-codesign guide. This post highlights its key principles—spanning shape alignment, architectural choices like LatentMoE, and parallelism strategies—to accelerate LLMs, Mixture-of-Experts (MoE) models, and other hardware-intensive workloads on TPUs.
Codesign models for Ironwood (TPU v7x) and future TPU generations: training and inference
Performance tuning starts with model definition. Architectural choices made before the first training step set permanent ceilings on MFU (Model FLOP Utilization), cost, and serving latency. This guide gives concrete, hardware-grounded design rules for teams building or adapting models for TPUs, organized into three groups:
- Shape rules (Principles 1–2): hardware-aligned dimensions and static shapes; near-zero model-quality risk. Adopt always.
- Architecture choices (Principles 3–7): structural design choices; most involve quality tradeoffs. Validate with ablations.
- Expressing parallelism (Principle 8): parallelism, kernel, and compiler tuning; no model changes required.
How to use this guide. The shape rules apply to anyone designing a new model. If you are adapting an existing model rather than designing one, the Expressing Parallelism section applies without touching the checkpoint. The general rules hold for any large-matmul architecture (dense LLMs, multimodal, and world models), and the companion AI-Hypercomputer/tpu-codesign repository grows model- and workload-specific sections over time.
Eighth-generation TPUs. These principles carry forward to TPU 8t and TPU 8i and to the newly announced TPU 8tx and TPU 8i Max. The reasoning stays the same and the thresholds move with the hardware. TPU 8t and TPU 8i both add native FP4. For training, TPU 8t also brings more balanced Vector Processing Unit (VPU) scaling. For serving, TPU 8i adds the Boardfly topology, a Collectives Acceleration Engine (CAE), and 3x more on-chip SRAM. We note the effects inline in Principles 5–8, and follow-up posts will cover eighth-generation designs in depth.
Measured evidence. Alongside the published sources, notes marked Measured carry results from our own Ironwood profiling program: open 20–30B MoEs (a Qwen3-30B-derived LatentMoE variant; GPT-OSS-20B) plus dense controls, single slice (except for the 512-chip TPU v7x and 1,024-chip TPU v4 ablations in Principle 4), BF16, training shapes up to 32K sequence, all against tuned-systems baselines. Treat the trends and rankings as transferable, the exact percentages as config-specific.
Foundational Performance Parameters (Ironwood / TPU v7x)
TPUs are purpose-built custom ML ASICs designed from the ground up to deliver state-of-the-art performance per total cost of ownership (TCO). While their architectural choices—such as wide 256×256 systolic arrays, software-managed VMEM, dedicated SparseCore engines, and direct multi-dimensional ICI fabrics—enable exceptional efficiency, they inherently create specific structural constraints and programming patterns that must be carefully followed to unlock maximal performance. Reaching peak efficiency requires aligning model shapes, routing, and parallelism directly with these underlying hardware characteristics:
Parallelism shorthand: DP = data, TP = tensor, EP = expert, CP = context, PP = pipeline parallelism; FSDP = fully sharded data parallel.
Shape rules (adopt always)
1. Align every reduction and output dimension to 256 (especially head_dim = 256)
Make d_model, FFN and expert intermediate widths (d_ff), latent widths, attention projections, and padded vocabulary sizes (used as contracting or output dimensions) multiples of 256 to match the 256×256 MXU systolic array. Token/batch axes and counts such as heads, layers, and experts are exempt. The most frequently missed dimension is head_dim: at 64 or 128, flash-attention QK products leave the MXU ≥ 50% idle. Reshape to head_dim = 256 with proportionally fewer Q and KV heads, preserving total projection widths and logical KV cache bytes (iso-quality in our ablations).
Measured: Reshaping Qwen3-30B MoE attention from 32 × 128 to 16 × 256 (identical parameters and FLOPs) sped up end-to-end training, with gains widening at longer context:
These rules apply for non-quadratic attention as well, notably GDN ablations from Yang et al. 2025 also noted at par or better quality for head dimension 256 for their Gated DeltaNet Attention.
2. Static and Dynamic Shapes
XLA compiles each program for exact tensor shapes, so every buffer has a fixed address and every loop a known trip count, and a new shape means a recompilation and a host stall. What about dynamic shapes? Dynamic workloads fit this model: the trick is to separate the shape of a buffer from the extent of the data inside it.
- Bucketing and masking: At the program level, pad requests to a small set of precompiled sequence-length and batch-slot buckets (warmed up at server startup), replace data-dependent branches with masks, and pack training sequences (packing=True in MaxText) so padding never reaches the MXU.
- Scalar Prefetch: Another common pattern of dynamic shapes appears in dropless MoE layers, which route a different number of tokens to each expert every step, and in paged attention, which gives every sequence its own KV length, its own set of pages scattered through the cache, and in a mixed batch its own phase (prefill or decode).
Instead of padding to the largest group or the longest sequence, we keep the buffers at fixed shapes (tokens sorted by expert and padded only at the tail, the KV cache as a fixed pool of pages) and describe the ragged structure with a few small integer arrays (group sizes and offsets; KV lengths, the page table, cumulative query lengths). Pallas copies these into the TensorCore's scalar memory before the grid starts (PrefetchScalarGridSpec), and the kernel reads them to choose the next tile and bound its loops, with a mask covering a tile that straddles a group boundary or a partly filled last page. The data dependence becomes an address and a trip count instead of a shape, so the compiled program is the same for every routing and every batch composition and the kernel runs at the unpadded FLOP count.
Architecture choices (validate with ablations)
3. Compress routed token traffic with LatentMoE
Standard EP dispatches every routed token at full d_model width, making EP + FSDP slower than plain FSDP in our Ironwood benchmarks. LatentMoE projects routed tokens from d_model down to latent width ℓ (d/ℓ ≈ 2–4×) before dispatch, executes experts in latent space, and projects back up after combine (introduced by Nemotron 3 and adopted by 962B MAI-Base-1 and 2.8T Kimi K3):
- Shrink both MoE bottlenecks and reinvest: With expert width
m = d_ff, latent compression cuts small-batch expert weight reads (∝ d × m) and throughput-bound all-to-all dispatch (∝ K × d) byd/ℓ. Reinvest the saved bandwidth into more active experts while keepingK_new × ℓ ≤ K_old × d_model, raising the nonlinearity budgetK × mat constant traffic (for example, Nemotron 3'sℓ = 1024, N = 512, top-22beatd = 4096, N = 128, top-6at matched active and total parameters). - TPU rules: Route on full
d_modelbefore compressing; keep ℓ a multiple of 256; dispatch in FP8 while keeping lightweight up/down projections in BF16; and place EP on one short 3D torus axis.
Measured: On a 30B / 128-expert MoE, an ℓ = 768 (2.67×) LatentMoE cut on-device permute time 2.8×, lifted end-to-end training throughput +14% tokens/s (322 vs 270 TFLOP/s/chip, or +19%, since the latent projections add ~5% FLOPs per token; tokens stay ℓ-wide past the down-projection, so on-device permute and combine shrink alongside EP all-to-all), and freed ~25 GB/device of activation HBM when remat checkpoints were anchored on the compressed latent. While current MoE recipes in tpu-recipes run FSDP (+ TP), LatentMoE + FP8 dispatch makes EP roofline-viable when per-device batch is too small to hide FSDP weight all-gathers. Under both FSDP and EP, avoid over-fragmenting experts: small tokens × K / N group sizes starve MXU tiles.
4. Prefer higher compute intensity when quality is neutral
When candidate MoE layouts match on quality, pick the one with higher compute intensity at a similar active-parameter budget. A larger shared expert (wider d_shared) shifts work into dense, communication-free GEMMs that overlap EP dispatch/combine in training and amortize weight reads across the decode batch. Fewer, wider routed experts (halving N and K while doubling d_ff at fixed sparsity K/N) cut dispatch traffic in half and double per-expert GEMM fill. Optimal (N, K) grows with model size, and over-coarsening hurts routing diversity. Ablate expert granularity and the shared-to-routed split at your target scale rather than copying GPU co-designed models, since the TPU optimum is more shared-heavy.
Measured: On a 63B LatentMoE (ℓ = 768, K/N = 1/32, 8K context, 512 TPU v7x chips at EP = 128), halving (N, K) from (512, 16) to (256, 8) while doubling d_ff and the shared expert lifted training throughput +57% tok/s and held lower loss across 120.8B tokens:
5. Overcome the decode bandwidth wall: topology scale-out (5.1), speculative decoding (5.2), and concurrency sizing
Low-concurrency decode streams model weights every token (~300 concurrent tokens per weight shard are needed in dense FP8 to reach compute roofline). Combine two architectural levers with explicit concurrency sizing:
- 5.1 Low-diameter scale-out: Scaling decode across high-connectivity, low-diameter topologies (such as Boardfly on TPU 8i) aggregates multi-chip HBM bandwidth while keeping collective latency low enough to beat single-chip bandwidth scaling in latency-sensitive regimes.TPU 8i pairs Boardfly with a Collectives Acceleration Engine (CAE), which replaces Ironwood's four SparseCores and cuts on-chip collective latency by up to 5x. It also triples on-chip SRAM, which keeps more of the KV cache on silicon. TPU 8i Max doubles compute, SRAM, and HBM bandwidth per package. The sizing rule below still applies with each system's own HBM numbers.
- 5.2 Design-time speculative decoding (MTP): Multi-token prediction (MTP) heads kept in BF16 provide a denser training signal and a built-in drafter that amortizes each weight read across multiple verified tokens (DeepSeek-V3 85–90% acceptance; Qwen3-Next; GLM-4.5; Nemotron 3 +2.4% benchmark gain and ~97% two-token acceptance). Measure net committed tokens/s across the full draft, verify, and rollback cycle of speculative decoding at target concurrency.
- Size to real serving concurrency and MoE batch occupancy: Per-token decode latency is bounded below by
per-chip weight bytes ÷ HBM bandwidth(for example, when the batch touches all experts, Qwen3.5-397B in FP8 across 4 chips streams ~100 GB/chip, setting a ~13.5 ms weight-read floor before KV), while max batch is capped by(remaining HBM) ÷ (allocated KV/state per sequence). For MoEs, weight traffic scales with the union of experts touched across the batch (E × [1 - (1 - K/E)^B]under uniform routing); profile busiest-rank skew and replicate hot experts (DeepSeek-V3) when HBM permits. If real concurrency will be small, design fewer active bytes (fewer active parameters or a smaller model) rather than planning around a batch size you will not have.
6. Minimize exposed VPU time
Norms, activations, masking, routing math and quantization scale applies run on the Vector Processing Unit (VPU), which is roughly two orders of magnitude below the MXU in throughput and executes in sequence with the matmuls unless XLA fuses it into an adjacent GEMM or we move it off the TensorCore. Therefore we drop reshapes and transposes that carry no quality, standardize on fusable RMSNorm and SwiGLU so the elementwise work folds into the neighbouring GEMM, and offload the MoE sort, gather and scatter to the SparseCore.
Note that fused VPU work is free only while its cycle count fits inside the MXU window it is fused into. A quick illustration of this concept is block-scaled fp8 GEMM.
With a scale per block of B elements along the contraction axis, the MXU emits one partial sum per output element per block and the VPU rescales each one (about three vector ops), so per output element the MXU does 2K flops and the VPU does 3·ceil(K/B) ops. The rescale hides when B ≥ 1.5 × r_mxu / r_vpu, which on v7x (fp8 peak 4,614 TFLOPS against a 19.2 TFLOPS vector peak, a ratio near 240) puts the threshold near 340. In simulation on a DeepSeek-V3 expert shape, block 512 runs at the per-tensor time, block 256 is VPU-bound at 1.2x and block 128 at 2.4x. Achieved MXU efficiency is lower than peak, which lowers the effective ratio and can make the smaller blocks workable; the way to decide is to measure the GEMM at each block instead of using peak ratio. This threshold is specific to Ironwood. TPU 8t is designed with specialized support for scaling so quantization for MXFP4 and MXFP8 formats has no overhead. Re-measure the block-size threshold there instead of reusing 340. The rule itself carries forward: keep vector work hidden inside a matmul window.
7. Use multi-pass BF16 for high-precision-bound compute
For workloads dominated by BF16-sensitive matmuls (most notably long-context quadratic attention where direct FP8 degrades accuracy), native single-pass BF16 leaves MXU capacity unused whenever hardware low-precision MXU throughput exceeds native BF16 throughput by more than the pass count. Use the multi-pass BF16 algorithm to execute BF16-accurate matmuls via multiple passes over the higher-throughput lower-precision MXU pipeline without losing numerical fidelity. This matters more as low-precision throughput pulls ahead of BF16. TPU 8t adds native FP4, which doubles MXU throughput. On Ironwood the FP8 to BF16 ratio is exactly 2:1, so two or more passes cannot beat native BF16.
Expressing Parallelism
8. Map parallel axes onto physical links (D2D → ICI torus → DCN)
Assign communication-heavy parallel axes to the fastest physical links and keep logical mesh factors aligned with torus dimensions:
- TP = 2 max for training (D2D), wider TP for decode: Training TP = 4 requires MLP > ~46K to hide behind ICI; map TP = 2 to the dual chiplets. Decode is weight-read bound, so wider TP (TP = 8 + EP on Qwen3-Coder-480B-A35B) cleanly shards HBM reads.
- FSDP default, EP on one short torus axis, DP/PP over DCN: When tiny per-device batches expose FSDP all-gathers inside a pod, verify intra-pod PP (which moves only stage activations) via roofline and short profiles. On eighth-generation systems the same approach applies with new link speeds and hop counts. TPU 8t keeps the 3D torus with 2x ICI and up to 4x DCN bandwidth. Boardfly on TPU 8i puts every chip in a 1,024-chip pod within seven hops, versus 16 on a torus of the same size, so EP dispatch and other all-to-all traffic cross fewer hops. TPU 8tx adopts the same Boardfly network.
Minimum-degree CP matched to kernel trade-offs: All-gather CP is simplest (though its all-gathers can sit exposed) when paired with zigzag load balancing (context_parallel_load_balance in MaxText) so causal masking does not idle tail ranks; Ring CP overlaps collectives but trades off SplashAttention chunk efficiency; Ulysses A2A CP avoids causal skew but exposes all-to-alls and caps at head count.
- SparseCore offload: Enabling XLA SparseCore async collective and sparse-gather offload cuts step time by 15–22% in tpu-recipes benchmarks with zero model changes (confirm the win with a short profile; SparseCore is also the native engine for large recsys-style embedding tables).
Putting it together: ~100B Ironwood LatentMoE
Start every project with the quality-safe foundation: 256-aligned dimensions (1), static shapes with dropless routing (2), one parallelism plan mapped to physical links (8), wall-clock ablations on a tuned stack, and published reference targets (both in Measuring what matters below). For MoEs, add LatentMoE (3) after an ablation to cut routing/dispatch traffic and decode weight reads. Then prioritize by bottleneck regime:
- High expert count (> 100 experts): Principles 3, 4, 6, 8 (latent dispatch compression, large shared expert, wider
d_ff, SparseCore offload, 1D torus EP). - Long context (> 64K): Principles 1, 7, 8 (head_dim = 256, multi-pass BF16, load-balanced CP).
- Multi-pod scale (> 100B): Principles 8, 3, 6, 5 (DP/PP over DCN, roofline-checked EP, block-scaled FP8, concurrency-sized serving).
- Latency & goodput serving: Principles 5, 6, 8 (concurrency-sized active bytes, Boardfly scale-out, MTP, block-scaled FP8, wider decode TP).
Reference Config
Measuring what matters: wall-clock ablations on a tuned baseline
Google engineered TPUs from the ground up as custom ML ASICs to deliver top-tier performance per total cost of ownership (TCO). Unlocking that potential requires bridging the gap between hardware architecture and model design. By standardizing on 256-aligned shapes, leveraging LatentMoE, and mapping parallel axes directly to physical links, engineering teams can maximize TPU efficiency across both training and serving workloads.
A reference blueprint is only as good as the loop used to validate it. Always evaluate architectural changes in wall-clock time on a tuned software stack rather than paper FLOPs: in our Ironwood benchmarks, pure systems tuning (SparseCore offload, MXU-aligned tiles, and remat tuning) lifted an open 30B MoE from 12% to 23% MFU before any model change, and published Ironwood baselines show 256-aligned MoEs (DeepSeek-V3, Qwen3-235B at 608–630 BF16 TFLOP/s/chip) running at roughly 2× the throughput of unaligned GPT-OSS-120B (330 TFLOP/s/chip). For the 6-signal XProf (profiler=xplane) triage checklist, roofline and HBM budgeting rules, wall-clock ablation methodology, published Ironwood reference targets, and ongoing updates as new TPU generations and kernels land, explore the living diagnostics guide and companion patterns in the open-source AI-Hypercomputer/tpu-codesign repository.
References
- Cloud TPU & JAX references: AI-Hypercomputer/tpu-codesign (living guide and companion configs), TPU7x (Ironwood) performance optimizations, TPU 8t and TPU 8i technical deep dive, How to Scale Your Model, AI-Hypercomputer/tpu-recipes, AI-Hypercomputer/maxtext, Qwen3.5-397B on Ironwood playbook, and TPU Ragged Paged Attention.
- Frontier model design papers: NVIDIA Nemotron 3 (arXiv:2512.20856), Microsoft MAI-Thinking-1, DeepSeek-V3 (arXiv:2412.19437), Kimi K3 (arXiv:2607.24653), and Gemma 3 (arXiv:2503.19786).
- Serving systems & profiling tools: Sarathi-Serve (arXiv:2403.02310), DistServe (arXiv:2401.09670), speculative decoding (arXiv:2211.17192), XProf / TensorBoard profiler, Pallas TPU pipelining, tune-jax, and jax.experimental.roofline.
- AI
- Cloud
- How-To Guides
- Best Practices
- Problem-Solving
- Learn
- Influence
- Solve
- Model Optimization
- Mixture of Experts
- LLM Training
- AI Infrastructure
- JAX
- MaxText
- LLM Inference
- Ironwood
- Machine Learning
- TPU
- Resources
AI
Cloud
Best Practices
Problem-Solving
Agents are sessions, not requests: size GKE node pools for AI agent sessions
NOV. 16, 2026
AI
Cloud
Tutorials
Case Studies
Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch
NOV. 16, 2026
AI
Cloud
Tutorials
How-To Guides
Securing Your Agents: A Hands-On Guide to Agent Anomaly Detection
NOV. 16, 2026