Novita AI 优化 GLM4-MoE 生产部署：TTFT 降低 65%，TPOT 提升 22%

Contents

TL;DR; How We Implemented Core Production Optimizations for GLM-MoE; 1. Shared Experts Fusion; 2. Qknorm Fusion; 3. Async Transfer; Production Benchmark Results; Benchmark configuration; Results; Suffix decoding; Background: The Inference Bottleneck in Agentic Coding; How Suffix Decoding Works; Data Validation: Output Pattern Repetition Analysis; Performance Comparison; Conclusion; How to Reproduce; Core Optimization Flags (SGLang Runtime); Speculative decoding configuration (agentic coding workload); Suffix Decoding configuration (optional); References

TL;DR

A suite of production-tested, high-impact optimizations has been developed by Novita AI for deploying GLM4-MOE models based on SGLANG. We introduce an end-to-end performance optimization strategy that addresses bottlenecks across the entire inference pipeline — from kernel execution efficiency to cross-node data transfer scheduling. Through the integration of Shared Experts Fusion and Suffix Decoding, we observe substantial gains in key production metrics, including:

up to 65% reduction in Time-to-First-Token (TTFT); 22% improvement in Time-Per-Output-Token (TPOT)

under agentic coding workloads.

All results were validated on H200 clusters under TP8 and FP8 configurations, providing a battle-tested blueprint for achieving both optimal throughput and low latency in demanding production environments.

1. Shared Experts Fusion

SGLang PR #13873: Shared Experts Fusion

Full credit for this optimization belongs to the original work on Deepseek model. As illustrated in the figure above, MoE models such as GLM4.7 route all input tokens through a shared expert, while each token is also individually routed to its own set of top‑k routed experts as selected by the model’s router. The outputs from all experts are then weighted and aggregated. GLM4.7, for instance, employs 160 routed experts alongside a single shared expert, selecting the top 8 routed experts per token. In earlier implementations, these two components were handled separately. Given that they share identical tensor shapes and computational procedures, it is natural to unify them by merging the shared expert into the routed MoE structure—selecting the top 9 out of the total 161 experts, with the shared expert consistently assigned the 9th position.

As documented in PR, this optimization achieves performance gains of up to 23.7% in TTFT and 20.8% in ITL. These gains are expected because, under TP8 and FP8 configurations—where the intermediate size is only 192, which is relatively small for H200 hardware—the fusion operation substantially boosts Streaming Multiprocessor (SM) utilization and significantly reduces memory I/O overhead.

2. Qknorm Fusion

This migration builds upon the optimization from Qwen-MOE. The underlying idea is straightforward. Since both operators perform head-wise computations, it is a natural approach to fuse them into a single kernel. Our contribution lies in adapting this fused kernel to accommodate the GLM4-MOE variant's specific case, where only half of the dimensions within a head are rotated.

3. Async Transfer

SGLang PR #14782: Async Transfer

In scenarios where PD disaggregation with overlapping schedules is applied, although throughput can gain about 10%, TTFT drops significantly. We observed that in the current implementation of prefill, the data transfer process is delayed until after the kernel launch for the next batch. For a model like GLM4.7, which consists of 92 layers, kernel launch without CUDA Graph can be time‑consuming (often taking hundreds of milliseconds even more than 1 second).

To address this, in our modification we advance the transfer step slightly, scheduling it right after its corresponding GPU operations complete. Additionally, the transfer is placed in a separate thread. By carefully handling potential data‑race structures, it can proceed without blocking the main thread.

Production Benchmark Results

After implementing the approaches described above, we observed significant performance improvements for GLM-MOE models, as clearly demonstrated by the benchmark results below.

Benchmark configuration

Input length: 4096; Output length: 1000; Request rate: 14 req/s; Model: GLM-4.7 FP8 (TP8)

Results

These optimizations are not just experimental — they have already been deployed and validated in Novita AI's production inference service.

Suffix decoding

Agentic coding scenarios (like Cursor and Claude Code) exhibit a high volume of reusable code patterns, allowing for targeted performance optimizations such as Suffix Decoding.

LLM Agents excel at code generation tasks, but latency remains a significant challenge. Traditional Speculative Decoding accelerates inference by predicting multiple tokens in advance, but common approaches require training additional draft models, introducing engineering complexity.

How Suffix Decoding Works

Suffix Decoding takes a fundamentally different approach—it is completely model-free:

No dependency on additional model weights; Leverages patterns from previously generated output sequences to predict upcoming tokens; When the current request's suffix matches a historical pattern, it continues along that historical sequence for speculation

By analyzing 22 Claude Code sessions (17,487 conversation turns), we discovered:

39.3% output pattern repetition: High frequency of similar tool calls and response patterns; Highly structured agentic behaviors: Fixed phrases like "Let me...", "Now let me..." appear frequently

To support further research, we have open-sourced the evaluation dataset on Hugging Face : Agentic Code Dataset on Hugging Face

Performance Comparison

With built-in MTP acceleration, Suffix Decoding further reduces TPOT by 22% (from 25.13ms to 19.63ms):

Conclusion

The combination of these optimizations provides comprehensive performance improvements for SGLANG deployments:

Most components are already merged upstream or undergoing integration; feel free to check them out on the SGLang repo.

How to Reproduce

Only the key performance-relevant parameters are shown here. Full launch scripts (baseline vs optimized), benchmark harness, and profiling traces are published in GitHub: novitalabs/sglang (glm_suffix branch).

Core Optimization Flags (SGLang Runtime)

Speculative decoding configuration (agentic coding workload)

Suffix Decoding configuration (optional)

References