- Overview
- Measurement Setup
- 1. Prefill CP and Decode DP
- 2. FP8 DeepGEMM for the Prefill Indexer
- 3. Breakable CUDA Graphs for Prefill
- 4. Tensor-Parallel MoE
- 5. Larger Indexer Score Tiles
- 6. Fuse Prefill Selection and Writeback
- 7. Reuse Paged Indexer K Tiles During Verification
- Follow-up Measurements
- 8. Colocate Serving Optimization
- Reproduction
- Disaggregated baseline
- Disaggregated optimized
- PD router
- PD ablations
- Colocated baseline
- Colocated optimized
- Benchmark
- GSM8K accuracy
Overview
We optimized disaggregated DeepSeek-V4.1-Flash serving on H20-3e by tuning prefill and decode parallelism, supporting BCG with prefill CP and bounded SWA replay, and speeding up Indexer kernels. Compared with TP4, the final configuration cut mean end-to-end latency by 65.5% and delivered 2.91× the output throughput.
Our intra-node PD deployment used four GPUs each for prefill and decode, with KV cache transfers over NVLink. Mean end-to-end latency fell from 13.63 s to 4.70 s, and output throughput rose from 847 to 2,465 tokens/s. Prefill context parallelism paired with decode data parallelism gave the largest gain, followed by FP8 DeepGEMM scoring.
| Cumulative configuration | Mean E2E (ms) | Output (tok/s) | Mean TTFT (ms) | Mean TPOT (ms) | P99 E2E (ms) |
|---|---|---|---|---|---|
| TP4 baseline | 13,633.80 | 846.76 | 12,037.73 | 3.20 | 21,306.70 |
| 1. Prefill CP + decode DP | 7,547.46 (−44.64%) | 1,537.90 | 5,156.84 (-57%) | 4.79 | 14,012.91 |
| 2. FP8 DeepGEMM | 5,294.89 (−29.85%) | 2,200.82 | 2,662.13 (-48%) | 5.28 | 11,016.82 |
| 3. Breakable CUDA Graphs | 5,100.87 (−3.66%) | 2,290.05 | 2,390.32 (-10%) | 5.43 | 10,641.21 |
| 4. TP MoE | 4,811.81 (−5.67%) | 2,425.96 | 1,877.23 (-22%) | 5.88 | 10,224.78 |
| 5. Larger prefill score tiles | 4,773.97 (−0.79%) | 2,450.67 | 1,838.15 (-2%) | 5.88 | 10,284.72 |
| 6. Prefill selection fusion | 4,774.47 (+0.01%) | 2,440.68 | 1,802.77 (-2%) | 5.96 | 10,221.00 |
| 7. Paged K-tile reuse | 4,698.65 (−1.59%) | 2,464.90 | 1,894.54 | 5.62 (-6%) | 10,073.00 |
Most of the latency gain came before the first token. Mean time to first token (TTFT) fell from 12.04 s to 1.89 s. Mean time per output token after the first (TPOT) rose from 3.20 to 5.62 ms. At fixed concurrency, faster prefill can keep decode busier, and faster decode can free client slots for new prompts. That can shift queueing between the two workers.
In the original sweep, the median prefill queue fell from 15 requests at baseline to 9 after CP and DP, then to 4 after DeepGEMM and 3 after breakable graphs. It reached zero with TP MoE. The median decode queue stayed at zero.
Measurement Setup
The router sends each request through prefill and decode. The workers transfer the KV state directly over intra-node Mooncake NVLink. The baseline runs TP4 attention and EP4 MoE on both workers with eager BF16 Torch prefill scoring. All runs used the same DeepSeek-V4.1-Flash checkpoint, Humming W4A8 MoE, a DSpark block size of 5, and 8,192-token prefill chunks.
| Workload | Value |
|---|---|
| Requests and concurrency | 256 requests, 24 in flight |
| Nominal input | 61,440 shared tokens + 4,096 request-specific tokens |
| Output | 500 tokens per request |
| Cache protocol | 10 warmup requests, then a cache flush before measurement |
| Observed cache hit rate | 93.21% |
All 256 requests completed and produced 128,000 output tokens in each run. The TP4 baseline and Steps 1–3 are single runs from the original sweep. Steps 4–7 show the mean of all completed follow-up runs for each configuration. The E2E percentages compare adjacent rows. P99 is calculated per run and then averaged.
1. Prefill CP and Decode DP
Prefill handles many token rows from a single long request. Interleaved context parallelism assigns those rows to different ranks so the four GPUs can work on the same request. Decode has many active requests with a small number of query rows each. Data-parallel attention and a data-parallel LM head assign independent requests to different ranks.
| Worker | Baseline attention | Step 1 attention | MoE in both |
|---|---|---|---|
| Prefill | TP4 | Interleaved CP4 | EP4 |
| Decode | TP4 | DP4 attention and LM head | EP4 |
Mean TTFT fell from 12.04 s to 5.16 s in this step. Mean TPOT rose from 3.20 to 4.79 ms.
Serving result. Mean E2E went from 13,633.80 to 7,547.46 ms (−44.64%). Output throughput rose from 846.76 to 1,537.90 tokens/s (+81.62%).
2. FP8 DeepGEMM for the Prefill Indexer
The Indexer scores visible keys before selecting a small subset for attention. On Hopper, we gather each request’s packed FP4 keys once, convert queries and keys to FP8 E4M3, and call DeepGEMM’s fp8_mqa_logits in each prefill Indexer layer. Each query can still see only the keys allowed by its causal mask and candidate set.
The queries are already FP4 fake-quantized, so we cast them to E4M3 without applying another scale. For keys, we apply the FP4 block’s power-of-two scale in FP32 and clamp to −448 to +448 before casting. This keeps out-of-range values finite. DeepGEMM gets unit key scales because the block scales are already applied.
The power-of-two scaling preserves exactly representable values. Out-of-range keys saturate, very small values can round or underflow, and the change in scoring precision can affect rankings. In a separate prefill probe on a 64K prompt, we sampled 64 query rows across 8 Indexer layers. None of the 16,774,528 sampled key values needed clamping. The BF16 and DeepGEMM scorers shared 99.08% of the selected top-k entries on those same rows.
Serving result. Mean E2E went from 7,547.46 to 5,294.89 ms (−29.85%). Output throughput rose from 1,537.90 to 2,200.82 tokens/s (+43.11%). SGLang PR #49066
3. Breakable CUDA Graphs for Prefill
Long prefill runs launch many small kernels. Breakable CUDA Graphs capture the graph-safe stretches and replay them together, while operations that need live metadata stay eager. Bounded sliding-window attention (SWA) replay runs the late layers on the shorter decoder tail and stays enabled throughout the sweep. The pinned experiment branch supports interleaved prefill CP with breakable graphs and bounded replay by capturing the tail in separate graph buckets. This combination requires the experiment branch. Experiment branch implementation, SGLang PR #40333, SGLang PR #43089.
At step 3, with an 8,192-token capture size and a 262,144-token graph context cap, startup logs reported 14.53–14.59 GB of capture memory per prefill rank and about 388 s to capture. The context cap limits the size of capture buffers. SGLang PR #36534
Serving result. Mean E2E went from 5,294.89 to 5,100.87 ms (−3.66%). Output throughput rose from 2,200.82 to 2,290.05 tokens/s (+4.05%).
4. Tensor-Parallel MoE
Expert parallelism assigns different experts to different ranks. When routing favors a few experts, their ranks do more work and the other ranks wait at the collective. Tensor-parallel MoE shards each expert across all four ranks, so uneven routing no longer puts more expert work on one GPU than another.
In a separate decode profile, an early EP rank spent about 100 µs waiting for a late rank at an all-reduce. TP made the expert GEMMs slightly slower in that profile and left total verification time roughly unchanged. In prefill, the two profiled batches below took 771 ms with EP and 657 ms with TP despite slower expert GEMMs with TP.
Tensor parallel
We compared against EP with --moe-a2a-backend none. This setup did not support DeepEP or MegaMoE all-to-all. In future work, we plan to test EP with all-to-all and expert parallel load balancing (EPLB), aiming to keep the faster GEMMs while balancing workload across GPUs.
Serving result. Mean E2E went from 5,100.87 to 4,811.81 ms (−5.67%). Output throughput rose from 2,290.05 to 2,425.96 tokens/s (+5.93%).
5. Larger Indexer Score Tiles
The Indexer limits the size of its temporary score matrix. With 16K compressed keys, the 64 MiB budget fits about 1,000 query rows, so a larger batch repeats scoring and selection across tiles. Raising the prefill limit to 2 GiB lets more rows share a tile. The limit is per rank and the scorer uses the space while processing the tile. Prefill KV capacity stayed at 1,945,088 tokens with --mem-fraction-static 0.8.
Serving result. Mean E2E went from 4,811.81 to 4,773.97 ms (−0.79%), within the run-to-run variation. SGLang PR #49067
6. Fuse Prefill Selection and Writeback
The Indexer needs both candidate block IDs for later layers and physical KV slots for attention. Triton selects blocks from their maximum scores. It reduces and selects in one launch on smaller inputs, and reduces in parallel before selecting on wider inputs. A separate fused writeback filters invalid picks, sorts the remaining positions, and translates them to KV slots in one launch. The selector uses the same scores and can pick different block IDs at ties. Its tie-break favors lower block IDs, while Torch topk does not promise a stable tie order.
Serving result. Mean TTFT fell from 1,838.15 to 1,802.77 ms (−1.92%). Mean E2E stayed around 4.77 s across 45 runs with fusion enabled. SGLang PR #49069
7. Reuse Paged Indexer K Tiles During Verification
DSpark verifies several query rows from the same request in one forward pass. Those rows look at nearly the same keys. The paged scorer loads and dequantizes one FP4 K tile into BF16, then reuses it for the request’s query rows. Each row still applies its own causal limit.
Serving result. Mean TPOT fell from 5.96 to 5.62 ms, and mean E2E fell from 4,774.47 to 4,698.65 ms (−1.59%). Output throughput rose by 0.99%. SGLang PR #49068
Follow-up Measurements
For Steps 4–7, in order to rule out the influence of steadiness the table reports the mean and sample standard deviation across all completed runs at concurrency 24. A single run’s mean E2E varied by roughly 1–1.6% at these steps.
| Step | Runs | Mean E2E (ms) | Output (tok/s) | Mean TTFT (ms) | Mean TPOT (ms) |
|---|---|---|---|---|---|
| 4. TP MoE | 3 | 4,811.81 ± 77.13 | 2,425.96 ± 38.96 | 1,877.23 ± 84.20 | 5.88 ± 0.31 |
| 5. Larger prefill score tiles | 6 | 4,773.97 ± 53.94 | 2,450.67 ± 26.56 | 1,838.15 ± 23.77 | 5.88 ± 0.14 |
| 6. Prefill selection fusion | 45 | 4,774.47 ± 48.24 | 2,440.68 ± 30.41 | 1,802.77 ± 40.21 | 5.96 ± 0.14 |
| 7. Paged K-tile reuse | 6 | 4,698.65 ± 47.73 | 2,464.90 ± 43.53 | 1,894.54 ± 58.59 | 5.62 ± 0.18 |
We also varied concurrency on the same 256-request workload and verified the effectiveness of our opt on other types of workload.
| Concurrency | TP4 E2E (ms) | Final E2E (ms) | TP4 output (tok/s) | Final output (tok/s) |
|---|---|---|---|---|
| 8 | 4,798.86 | 2,690.09 | 826.73 | 1,462.65 |
| 16 | 9,178.42 | 3,678.99 | 854.19 | 2,121.03 |
| 24 | 13,563.03 | 4,698.65 | 853.98 | 2,464.90 |
| 32 | 17,843.62 | 6,097.87 | 852.97 | 2,502.19 |
8. Colocate Serving Optimization
We ran the same request workload at concurrency 24 on a single four-GPU worker with DSpark. Compared with the BF16 TP4 baseline, mean E2E fell from 18.62 s to 7.96 s (57.26% lower), and output throughput rose from 642 to 1,483 tokens/s (130.79% higher).
| Cumulative configuration | Mean E2E (ms) | Output (tok/s) |
|---|---|---|
| TP baseline (BF16 prefill Indexer) | 18,619.53 | 642.46 |
| Prefill CP, decode TP (BF16 prefill Indexer) | 12,499.25 (-33%) | 949.97 |
| FP8 DeepGEMM | 9,727.21 (-22%) | 1,222.38 |
| Breakable CUDA Graphs | 9,418.85 (-3%) | 1,256.71 |
| Paged K-tile reuse | 8,694.89 (-8%) | 1,367.96 |
| Tensor-parallel MoE | 8,632.46 (-1%) | 1,377.49 |
| Batch-wise prefill CP and decode DP | 7,957.13 (-8%) | 1,482.72 |
We added batch-wise prefill CP and decode DP to the same worker. Prefill still splits prompt tokens across four ranks. During decode and target verification, each rank instead handles a slice of the batches with all attention heads, and the worker gathers the results back into batch order.
Adding decode DP to the prefill CP configuration cut mean E2E by 7.82% and raised output throughput by 7.64%. Decode DP requires replicated KV and Indexer keys. With the same --max-running-requests 64, the startup log reported 955,136 KV token slots with decode TP and 953,088 with decode DP. Those counts measure the KV pool. The supported concurrency also depends on the length of live sequences. SGLang PR #45729
For this workload, the colocated configuration produced more output tokens per second per GPU (371 vs. 308). The eight-GPU PD configuration delivered lower mean latency (4.70 vs. 7.96 s) and higher total throughput.
Reproduction
Use commit b07c70112c175eb7ca3c157c7bbc7847cc398d7d on the public SYChen123/sglang:v4.1/h20-dev-pd-blog branch. The commands use the DeepSeek-V4.1-Flash checkpoint. The H20 follow-up environment used PyTorch 2.14.1, Triton 3.8.0, sgl-deep-gemm==0.2.2, humming-kernels==0.1.12, mooncake-transfer-engine-cuda13==0.3.10.post2, and sglang-router==0.3.2. On the experiment branch, SGLANG_DSV41_TORCH_PREFILL_INDEXER=1 selects BF16 Torch scoring. To use FP8 DeepGEMM on Hopper, set it to 0 and select SGLANG_DSV41_SM90_PREFILL_INDEXER=deepgemm.
Steps 2 and 5–7 and colocated decode DP are still under upstream review. The four Indexer PR branches use the same names for their respective switches as the ablation table below. Run these commands on the pinned experiment revision, which also supports prefill CP with breakable graphs and bounded SWA replay. FP8 DeepGEMM, score budget, selection fusion, K-tile reuse, colocated decode DP.
Disaggregated baseline
Run prefill and decode in separate terminals on the same eight-GPU host.
Start baseline prefill (GPUs 0–3)CUDA_VISIBLE_DEVICES=0,1,2,3 SGLANG_DSV41_TORCH_PREFILL_INDEXER=1 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_HOST_IP=127.0.0.1 SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true MC_INTRANODE_NVLINK=true SGLANG_SET_CPU_AFFINITY=1 \
SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=0 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
--host 127.0.0.1 --port 9091 --trust-remote-code --language-model-only \
--attention-backend dsv4 --tp 4 --ep-size 4 --moe-runner-backend humming --moe-a2a-backend none \
--speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
--chunked-prefill-size 8192 --max-prefill-tokens 8192 --disable-prefill-cuda-graph \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.1 --mem-fraction-static 0.8 \
--tokenizer-worker-num 8 --detokenizer-worker-num 8 \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998
Start baseline decode (GPUs 4–7)
CUDA_VISIBLE_DEVICES=4,5,6,7 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_HOST_IP=127.0.0.1 SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true MC_INTRANODE_NVLINK=true SGLANG_SET_CPU_AFFINITY=1 \
SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=0 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
--host 127.0.0.1 --port 9092 --trust-remote-code --attention-backend dsv4 \
--tp 4 --ep-size 4 --moe-runner-backend humming --moe-a2a-backend none \
--speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
--mem-fraction-static 0.85 --max-running-requests 256 \
--tokenizer-worker-num 8 --detokenizer-worker-num 8 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake
Disaggregated optimized
Start optimized prefill (GPUs 0–3)CUDA_VISIBLE_DEVICES=0,1,2,3 SGLANG_DSV41_TORCH_PREFILL_INDEXER=0 \
SGLANG_DSV41_SM90_PREFILL_INDEXER=deepgemm \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_HOST_IP=127.0.0.1 SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true MC_INTRANODE_NVLINK=true SGLANG_SET_CPU_AFFINITY=1 \
SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=1 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
SGLANG_OPT_DSV41_LARGE_PREFILL_SCORE_BUDGET=1 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
--host 127.0.0.1 --port 9091 --trust-remote-code --language-model-only \
--attention-backend dsv4 --tp 4 --ep-size 1 --moe-runner-backend humming --moe-a2a-backend none \
--speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
--chunked-prefill-size 8192 --max-prefill-tokens 8192 --enable-prefill-cp --cp-strategy interleave \
--cuda-graph-backend-prefill breakable --cuda-graph-max-bs-prefill 8192 --cuda-graph-prefill-max-context 262144 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.1 --mem-fraction-static 0.8 \
--tokenizer-worker-num 8 --detokenizer-worker-num 8 --disaggregation-mode prefill \
--disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 8998
Start optimized decode (GPUs 4–7)
CUDA_VISIBLE_DEVICES=4,5,6,7 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_HOST_IP=127.0.0.1 SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true MC_INTRANODE_NVLINK=true SGLANG_SET_CPU_AFFINITY=1 \
SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=1 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=1 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
--host 127.0.0.1 --port 9092 --trust-remote-code --attention-backend dsv4 \
--tp 4 --dp 4 --enable-dp-attention --enable-dp-lm-head \
--ep-size 1 --moe-runner-backend humming --moe-a2a-backend none \
--speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
--mem-fraction-static 0.85 --max-running-requests 256 \
--tokenizer-worker-num 8 --detokenizer-worker-num 8 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake
PD router
Start the router once both workers are healthy.
Start the PD routerpython3 -m sglang_router.launch_router --pd-disaggregation \
--prefill http://127.0.0.1:9091 8998 --decode http://127.0.0.1:9092 \
--host 127.0.0.1 --port 9090 --worker-startup-timeout-secs 3600 \
--disable-circuit-breaker --prefill-policy round_robin --decode-policy round_robin
PD ablations
The ablation experiments in this blog start with the PD baseline and apply the changes in order. Keep the baseline settings unless a row changes them. The decode budget changes in Step 5 only on the experiment branch.
PD ablation guide| Step | Worker | Change from the previous step |
|---|---|---|
| 1. CP + DP | Both | Prefill adds --enable-prefill-cp --cp-strategy interleave. Decode adds --dp 4 --enable-dp-attention --enable-dp-lm-head. |
| 2. DeepGEMM | Prefill | Set SGLANG_DSV41_TORCH_PREFILL_INDEXER=0 and SGLANG_DSV41_SM90_PREFILL_INDEXER=deepgemm. |
| 3. BCG | Prefill | Replace --disable-prefill-cuda-graph with --cuda-graph-backend-prefill breakable --cuda-graph-max-bs-prefill 8192 --cuda-graph-prefill-max-context 262144. |
| 4. TP MoE | Both | Set --ep-size 1. |
| 5. Larger score tiles | Both | On prefill, set SGLANG_OPT_DSV41_LARGE_PREFILL_SCORE_BUDGET=1. On decode, set SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=1 to move from 128 to 512 MiB. |
| 6. Selection fusion | Prefill | Set SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=1. |
| 7. Paged K-tile reuse | Decode | Set SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=1. |
Colocated baseline
The four-GPU worker listens on the benchmark port directly.
Start the colocated baselineCUDA_VISIBLE_DEVICES=0,1,2,3 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=0 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_DSV41_TORCH_PREFILL_INDEXER=1 SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=0 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
--host 127.0.0.1 --port 9090 --trust-remote-code --attention-backend dsv4 \
--tp 4 --ep-size 4 --moe-runner-backend humming --moe-a2a-backend none \
--speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
--chunked-prefill-size 8192 --max-prefill-tokens 8192 --disable-prefill-cuda-graph \
--enable-decoder-swa-bounded-replay --cuda-graph-max-bs-decode 64 \
--mem-fraction-static 0.8 --max-running-requests 64 \
--enable-hierarchical-cache --hicache-ratio 2 --hicache-io-backend kernel --hicache-write-policy write_through
Colocated optimized
Start the optimized colocated workerCUDA_VISIBLE_DEVICES=0,1,2,3 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=0 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_DSV41_TORCH_PREFILL_INDEXER=0 SGLANG_DSV41_SM90_PREFILL_INDEXER=deepgemm \
SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=1 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
--host 127.0.0.1 --port 9090 --trust-remote-code --attention-backend dsv4 \
--tp 4 --ep-size 1 --moe-runner-backend humming --moe-a2a-backend none \
--speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
--chunked-prefill-size 8192 --max-prefill-tokens 8192 --enable-prefill-cp --cp-strategy interleave \
--enable-cp-decode-attn-tp --cp-decode-attn-mode dp --enable-decoder-swa-bounded-replay \
--cuda-graph-backend-prefill breakable --cuda-graph-max-bs-prefill 8192 --cuda-graph-prefill-max-context 262144 \
--cuda-graph-max-bs-decode 64 --mem-fraction-static 0.8 --max-running-requests 64 \
--enable-hierarchical-cache --hicache-ratio 2 --hicache-io-backend kernel --hicache-write-policy write_through
Benchmark
Run the serving benchmarkpython3 -m sglang.bench_serving --model deepseek-ai/DeepSeek-V4.1-Flash \
--served-model-name llm --host 127.0.0.1 --port 9090 --backend sglang-oai \
--dataset-name generated-shared-prefix \
--num-prompts 256 --gsp-num-groups 1 --gsp-prompts-per-group 256 \
--gsp-system-prompt-len 61440 --gsp-question-len 4096 --gsp-output-len 500 \
--max-concurrency 24 --warmup-requests 10 --seed 42 --flush-cache --cache-report
GSM8K accuracy
Use the chat endpoint on the PD router for both configurations. Enable --reasoning-parser deepseek-v41 --enable-strict-thinking on the workers. The evaluator takes five questions for few-shot examples and scores the remaining 1,314.
python3 -m sglang.test.run_eval --eval-name gsm8k --api chat --num-examples 1319 \
--host 127.0.0.1 --port 9090 --num-threads 256 --reasoning-effort high
| PD configuration | GSM8K | 64K retrieval | DSpark tokens / verify |
|---|---|---|---|
| TP4 baseline | 96.50% (1,268/1,314) | 10/10 | 5.368 |
| Final | 96.42% (1,267/1,314) | 10/10 | 5.460 |
AI Trends