‹ Back to BlogOn this page

Overview

We optimized disaggregated DeepSeek-V4.1-Flash serving on H20-3e by tuning prefill and decode parallelism, supporting BCG with prefill CP and bounded SWA replay, and speeding up Indexer kernels. Compared with TP4, the final configuration cut mean end-to-end latency by 65.5% and delivered 2.91× the output throughput.

Our intra-node PD deployment used four GPUs each for prefill and decode, with KV cache transfers over NVLink. Mean end-to-end latency fell from 13.63 s to 4.70 s, and output throughput rose from 847 to 2,465 tokens/s. Prefill context parallelism paired with decode data parallelism gave the largest gain, followed by FP8 DeepGEMM scoring.

Compared with TP4, the final PD configuration has 0.34× mean end-to-end latency, 0.16× mean time to first token, and 2.91× output throughput.

Cumulative configurationMean E2E (ms)Output (tok/s)Mean TTFT (ms)Mean TPOT (ms)P99 E2E (ms)
TP4 baseline13,633.80846.7612,037.733.2021,306.70
1. Prefill CP + decode DP7,547.46 (−44.64%)1,537.905,156.84 (-57%)4.7914,012.91
2. FP8 DeepGEMM5,294.89 (−29.85%)2,200.822,662.13 (-48%)5.2811,016.82
3. Breakable CUDA Graphs5,100.87 (−3.66%)2,290.052,390.32 (-10%)5.4310,641.21
4. TP MoE4,811.81 (−5.67%)2,425.961,877.23 (-22%)5.8810,224.78
5. Larger prefill score tiles4,773.97 (−0.79%)2,450.671,838.15 (-2%)5.8810,284.72
6. Prefill selection fusion4,774.47 (+0.01%)2,440.681,802.77 (-2%)5.9610,221.00
7. Paged K-tile reuse4,698.65 (−1.59%)2,464.901,894.545.62 (-6%)10,073.00

Most of the latency gain came before the first token. Mean time to first token (TTFT) fell from 12.04 s to 1.89 s. Mean time per output token after the first (TPOT) rose from 3.20 to 5.62 ms. At fixed concurrency, faster prefill can keep decode busier, and faster decode can free client slots for new prompts. That can shift queueing between the two workers.

In the original sweep, the median prefill queue fell from 15 requests at baseline to 9 after CP and DP, then to 4 after DeepGEMM and 3 after breakable graphs. It reached zero with TP MoE. The median decode queue stayed at zero.

Measurement Setup

The router sends each request through prefill and decode. The workers transfer the KV state directly over intra-node Mooncake NVLink. The baseline runs TP4 attention and EP4 MoE on both workers with eager BF16 Torch prefill scoring. All runs used the same DeepSeek-V4.1-Flash checkpoint, Humming W4A8 MoE, a DSpark block size of 5, and 8,192-token prefill chunks.

WorkloadValue
Requests and concurrency256 requests, 24 in flight
Nominal input61,440 shared tokens + 4,096 request-specific tokens
Output500 tokens per request
Cache protocol10 warmup requests, then a cache flush before measurement
Observed cache hit rate93.21%

All 256 requests completed and produced 128,000 output tokens in each run. The TP4 baseline and Steps 1–3 are single runs from the original sweep. Steps 4–7 show the mean of all completed follow-up runs for each configuration. The E2E percentages compare adjacent rows. P99 is calculated per run and then averaged.

1. Prefill CP and Decode DP

Prefill handles many token rows from a single long request. Interleaved context parallelism assigns those rows to different ranks so the four GPUs can work on the same request. Decode has many active requests with a small number of query rows each. Data-parallel attention and a data-parallel LM head assign independent requests to different ranks.

WorkerBaseline attentionStep 1 attentionMoE in both
PrefillTP4Interleaved CP4EP4
DecodeTP4DP4 attention and LM headEP4

Mean TTFT fell from 12.04 s to 5.16 s in this step. Mean TPOT rose from 3.20 to 4.79 ms.

Serving result. Mean E2E went from 13,633.80 to 7,547.46 ms (−44.64%). Output throughput rose from 846.76 to 1,537.90 tokens/s (+81.62%).

2. FP8 DeepGEMM for the Prefill Indexer

The Indexer scores visible keys before selecting a small subset for attention. On Hopper, we gather each request’s packed FP4 keys once, convert queries and keys to FP8 E4M3, and call DeepGEMM’s fp8_mqa_logits in each prefill Indexer layer. Each query can still see only the keys allowed by its causal mask and candidate set.

The queries are already FP4 fake-quantized, so we cast them to E4M3 without applying another scale. For keys, we apply the FP4 block’s power-of-two scale in FP32 and clamp to −448 to +448 before casting. This keeps out-of-range values finite. DeepGEMM gets unit key scales because the block scales are already applied.

The power-of-two scaling preserves exactly representable values. Out-of-range keys saturate, very small values can round or underflow, and the change in scoring precision can affect rankings. In a separate prefill probe on a 64K prompt, we sampled 64 query rows across 8 Indexer layers. None of the 16,774,528 sampled key values needed clamping. The BF16 and DeepGEMM scorers shared 99.08% of the selected top-k entries on those same rows.

Torch scores in BF16; DeepGEMM scores in E4M3 after scaling and clamping the FP4 keys

Serving result. Mean E2E went from 7,547.46 to 5,294.89 ms (−29.85%). Output throughput rose from 1,537.90 to 2,200.82 tokens/s (+43.11%). SGLang PR #49066

3. Breakable CUDA Graphs for Prefill

Long prefill runs launch many small kernels. Breakable CUDA Graphs capture the graph-safe stretches and replay them together, while operations that need live metadata stay eager. Bounded sliding-window attention (SWA) replay runs the late layers on the shorter decoder tail and stays enabled throughout the sweep. The pinned experiment branch supports interleaved prefill CP with breakable graphs and bounded replay by capturing the tail in separate graph buckets. This combination requires the experiment branch. Experiment branch implementation, SGLang PR #40333, SGLang PR #43089.

At step 3, with an 8,192-token capture size and a 262,144-token graph context cap, startup logs reported 14.53–14.59 GB of capture memory per prefill rank and about 388 s to capture. The context cap limits the size of capture buffers. SGLang PR #36534

Serving result. Mean E2E went from 5,294.89 to 5,100.87 ms (−3.66%). Output throughput rose from 2,200.82 to 2,290.05 tokens/s (+4.05%).

4. Tensor-Parallel MoE

Expert parallelism assigns different experts to different ranks. When routing favors a few experts, their ranks do more work and the other ranks wait at the collective. Tensor-parallel MoE shards each expert across all four ranks, so uneven routing no longer puts more expert work on one GPU than another.

In a separate decode profile, an early EP rank spent about 100 µs waiting for a late rank at an all-reduce. TP made the expert GEMMs slightly slower in that profile and left total verification time roughly unchanged. In prefill, the two profiled batches below took 771 ms with EP and 657 ms with TP despite slower expert GEMMs with TP.

Expert parallelExpert-parallel prefill timeline showing 770.7 ms for five requests and 16,119 tokens. Tensor parallelTensor-parallel prefill timeline showing 656.5 ms for five requests and 16,104 tokens.
Separate prefill trace samples, both with five requests and about 16K tokens.

We compared against EP with --moe-a2a-backend none. This setup did not support DeepEP or MegaMoE all-to-all. In future work, we plan to test EP with all-to-all and expert parallel load balancing (EPLB), aiming to keep the faster GEMMs while balancing workload across GPUs.

Serving result. Mean E2E went from 5,100.87 to 4,811.81 ms (−5.67%). Output throughput rose from 2,290.05 to 2,425.96 tokens/s (+5.93%).

5. Larger Indexer Score Tiles

The Indexer limits the size of its temporary score matrix. With 16K compressed keys, the 64 MiB budget fits about 1,000 query rows, so a larger batch repeats scoring and selection across tiles. Raising the prefill limit to 2 GiB lets more rows share a tile. The limit is per rank and the scorer uses the space while processing the tile. Prefill KV capacity stayed at 1,945,088 tokens with --mem-fraction-static 0.8.

Serving result. Mean E2E went from 4,811.81 to 4,773.97 ms (−0.79%), within the run-to-run variation. SGLang PR #49067

6. Fuse Prefill Selection and Writeback

The Indexer needs both candidate block IDs for later layers and physical KV slots for attention. Triton selects blocks from their maximum scores. It reduces and selects in one launch on smaller inputs, and reduces in parallel before selecting on wider inputs. A separate fused writeback filters invalid picks, sorts the remaining positions, and translates them to KV slots in one launch. The selector uses the same scores and can pick different block IDs at ties. Its tie-break favors lower block IDs, while Torch topk does not promise a stable tie order.

Triton reduces the work for candidate publishing and fuses KV-slot writeback into one launch

Serving result. Mean TTFT fell from 1,838.15 to 1,802.77 ms (−1.92%). Mean E2E stayed around 4.77 s across 45 runs with fusion enabled. SGLang PR #49069

7. Reuse Paged Indexer K Tiles During Verification

DSpark verifies several query rows from the same request in one forward pass. Those rows look at nearly the same keys. The paged scorer loads and dequantizes one FP4 K tile into BF16, then reuses it for the request’s query rows. Each row still applies its own causal limit.

Verify queries switch from one load and dequantization each to one shared BF16 key tile

Serving result. Mean TPOT fell from 5.96 to 5.62 ms, and mean E2E fell from 4,774.47 to 4,698.65 ms (−1.59%). Output throughput rose by 0.99%. SGLang PR #49068

Follow-up Measurements

For Steps 4–7, in order to rule out the influence of steadiness the table reports the mean and sample standard deviation across all completed runs at concurrency 24. A single run’s mean E2E varied by roughly 1–1.6% at these steps.

StepRunsMean E2E (ms)Output (tok/s)Mean TTFT (ms)Mean TPOT (ms)
4. TP MoE34,811.81 ± 77.132,425.96 ± 38.961,877.23 ± 84.205.88 ± 0.31
5. Larger prefill score tiles64,773.97 ± 53.942,450.67 ± 26.561,838.15 ± 23.775.88 ± 0.14
6. Prefill selection fusion454,774.47 ± 48.242,440.68 ± 30.411,802.77 ± 40.215.96 ± 0.14
7. Paged K-tile reuse64,698.65 ± 47.732,464.90 ± 43.531,894.54 ± 58.595.62 ± 0.18

We also varied concurrency on the same 256-request workload and verified the effectiveness of our opt on other types of workload.

ConcurrencyTP4 E2E (ms)Final E2E (ms)TP4 output (tok/s)Final output (tok/s)
84,798.862,690.09826.731,462.65
169,178.423,678.99854.192,121.03
2413,563.034,698.65853.982,464.90
3217,843.626,097.87852.972,502.19

8. Colocate Serving Optimization

We ran the same request workload at concurrency 24 on a single four-GPU worker with DSpark. Compared with the BF16 TP4 baseline, mean E2E fell from 18.62 s to 7.96 s (57.26% lower), and output throughput rose from 642 to 1,483 tokens/s (130.79% higher).

Cumulative configurationMean E2E (ms)Output (tok/s)
TP baseline (BF16 prefill Indexer)18,619.53642.46
Prefill CP, decode TP (BF16 prefill Indexer)12,499.25 (-33%)949.97
FP8 DeepGEMM9,727.21 (-22%)1,222.38
Breakable CUDA Graphs9,418.85 (-3%)1,256.71
Paged K-tile reuse8,694.89 (-8%)1,367.96
Tensor-parallel MoE8,632.46 (-1%)1,377.49
Batch-wise prefill CP and decode DP7,957.13 (-8%)1,482.72

We added batch-wise prefill CP and decode DP to the same worker. Prefill still splits prompt tokens across four ranks. During decode and target verification, each rank instead handles a slice of the batches with all attention heads, and the worker gathers the results back into batch order.

Both colocated configurations split prefill tokens; decode switches from splitting heads to splitting query rows

Adding decode DP to the prefill CP configuration cut mean E2E by 7.82% and raised output throughput by 7.64%. Decode DP requires replicated KV and Indexer keys. With the same --max-running-requests 64, the startup log reported 955,136 KV token slots with decode TP and 953,088 with decode DP. Those counts measure the KV pool. The supported concurrency also depends on the length of live sequences. SGLang PR #45729

For this workload, the colocated configuration produced more output tokens per second per GPU (371 vs. 308). The eight-GPU PD configuration delivered lower mean latency (4.70 vs. 7.96 s) and higher total throughput.

Reproduction

Use commit b07c70112c175eb7ca3c157c7bbc7847cc398d7d on the public SYChen123/sglang:v4.1/h20-dev-pd-blog branch. The commands use the DeepSeek-V4.1-Flash checkpoint. The H20 follow-up environment used PyTorch 2.14.1, Triton 3.8.0, sgl-deep-gemm==0.2.2, humming-kernels==0.1.12, mooncake-transfer-engine-cuda13==0.3.10.post2, and sglang-router==0.3.2. On the experiment branch, SGLANG_DSV41_TORCH_PREFILL_INDEXER=1 selects BF16 Torch scoring. To use FP8 DeepGEMM on Hopper, set it to 0 and select SGLANG_DSV41_SM90_PREFILL_INDEXER=deepgemm.

Steps 2 and 5–7 and colocated decode DP are still under upstream review. The four Indexer PR branches use the same names for their respective switches as the ablation table below. Run these commands on the pinned experiment revision, which also supports prefill CP with breakable graphs and bounded SWA replay. FP8 DeepGEMM, score budget, selection fusion, K-tile reuse, colocated decode DP.

Disaggregated baseline

Run prefill and decode in separate terminals on the same eight-GPU host.

Start baseline prefill (GPUs 0–3)
CUDA_VISIBLE_DEVICES=0,1,2,3 SGLANG_DSV41_TORCH_PREFILL_INDEXER=1 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_HOST_IP=127.0.0.1 SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true MC_INTRANODE_NVLINK=true SGLANG_SET_CPU_AFFINITY=1 \
SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=0 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
  --host 127.0.0.1 --port 9091 --trust-remote-code --language-model-only \
  --attention-backend dsv4 --tp 4 --ep-size 4 --moe-runner-backend humming --moe-a2a-backend none \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
  --chunked-prefill-size 8192 --max-prefill-tokens 8192 --disable-prefill-cuda-graph \
  --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.1 --mem-fraction-static 0.8 \
  --tokenizer-worker-num 8 --detokenizer-worker-num 8 \
  --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
  --disaggregation-bootstrap-port 8998
Start baseline decode (GPUs 4–7)
CUDA_VISIBLE_DEVICES=4,5,6,7 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_HOST_IP=127.0.0.1 SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true MC_INTRANODE_NVLINK=true SGLANG_SET_CPU_AFFINITY=1 \
SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=0 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
  --host 127.0.0.1 --port 9092 --trust-remote-code --attention-backend dsv4 \
  --tp 4 --ep-size 4 --moe-runner-backend humming --moe-a2a-backend none \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
  --mem-fraction-static 0.85 --max-running-requests 256 \
  --tokenizer-worker-num 8 --detokenizer-worker-num 8 \
  --disaggregation-mode decode --disaggregation-transfer-backend mooncake

Disaggregated optimized

Start optimized prefill (GPUs 0–3)
CUDA_VISIBLE_DEVICES=0,1,2,3 SGLANG_DSV41_TORCH_PREFILL_INDEXER=0 \
SGLANG_DSV41_SM90_PREFILL_INDEXER=deepgemm \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_HOST_IP=127.0.0.1 SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true MC_INTRANODE_NVLINK=true SGLANG_SET_CPU_AFFINITY=1 \
SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=1 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
SGLANG_OPT_DSV41_LARGE_PREFILL_SCORE_BUDGET=1 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
  --host 127.0.0.1 --port 9091 --trust-remote-code --language-model-only \
  --attention-backend dsv4 --tp 4 --ep-size 1 --moe-runner-backend humming --moe-a2a-backend none \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
  --chunked-prefill-size 8192 --max-prefill-tokens 8192 --enable-prefill-cp --cp-strategy interleave \
  --cuda-graph-backend-prefill breakable --cuda-graph-max-bs-prefill 8192 --cuda-graph-prefill-max-context 262144 \
  --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.1 --mem-fraction-static 0.8 \
  --tokenizer-worker-num 8 --detokenizer-worker-num 8 --disaggregation-mode prefill \
  --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 8998
Start optimized decode (GPUs 4–7)
CUDA_VISIBLE_DEVICES=4,5,6,7 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_HOST_IP=127.0.0.1 SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK \
MC_INTRA_NVLINK=true MC_INTRANODE_NVLINK=true SGLANG_SET_CPU_AFFINITY=1 \
SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1 \
SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=1 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=1 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
  --host 127.0.0.1 --port 9092 --trust-remote-code --attention-backend dsv4 \
  --tp 4 --dp 4 --enable-dp-attention --enable-dp-lm-head \
  --ep-size 1 --moe-runner-backend humming --moe-a2a-backend none \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
  --mem-fraction-static 0.85 --max-running-requests 256 \
  --tokenizer-worker-num 8 --detokenizer-worker-num 8 \
  --disaggregation-mode decode --disaggregation-transfer-backend mooncake

PD router

Start the router once both workers are healthy.

Start the PD router
python3 -m sglang_router.launch_router --pd-disaggregation \
  --prefill http://127.0.0.1:9091 8998 --decode http://127.0.0.1:9092 \
  --host 127.0.0.1 --port 9090 --worker-startup-timeout-secs 3600 \
  --disable-circuit-breaker --prefill-policy round_robin --decode-policy round_robin

PD ablations

The ablation experiments in this blog start with the PD baseline and apply the changes in order. Keep the baseline settings unless a row changes them. The decode budget changes in Step 5 only on the experiment branch.

PD ablation guide
StepWorkerChange from the previous step
1. CP + DPBothPrefill adds --enable-prefill-cp --cp-strategy interleave. Decode adds --dp 4 --enable-dp-attention --enable-dp-lm-head.
2. DeepGEMMPrefillSet SGLANG_DSV41_TORCH_PREFILL_INDEXER=0 and SGLANG_DSV41_SM90_PREFILL_INDEXER=deepgemm.
3. BCGPrefillReplace --disable-prefill-cuda-graph with --cuda-graph-backend-prefill breakable --cuda-graph-max-bs-prefill 8192 --cuda-graph-prefill-max-context 262144.
4. TP MoEBothSet --ep-size 1.
5. Larger score tilesBothOn prefill, set SGLANG_OPT_DSV41_LARGE_PREFILL_SCORE_BUDGET=1. On decode, set SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=1 to move from 128 to 512 MiB.
6. Selection fusionPrefillSet SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=1.
7. Paged K-tile reuseDecodeSet SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=1.

Colocated baseline

The four-GPU worker listens on the benchmark port directly.

Start the colocated baseline
CUDA_VISIBLE_DEVICES=0,1,2,3 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=0 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_DSV41_TORCH_PREFILL_INDEXER=1 SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=0 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
  --host 127.0.0.1 --port 9090 --trust-remote-code --attention-backend dsv4 \
  --tp 4 --ep-size 4 --moe-runner-backend humming --moe-a2a-backend none \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
  --chunked-prefill-size 8192 --max-prefill-tokens 8192 --disable-prefill-cuda-graph \
  --enable-decoder-swa-bounded-replay --cuda-graph-max-bs-decode 64 \
  --mem-fraction-static 0.8 --max-running-requests 64 \
  --enable-hierarchical-cache --hicache-ratio 2 --hicache-io-backend kernel --hicache-write-policy write_through

Colocated optimized

Start the optimized colocated worker
CUDA_VISIBLE_DEVICES=0,1,2,3 \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_OPT_DSV41_TRITON_PREFILL_SELECTION=0 SGLANG_OPT_DSV41_LARGE_INDEXER_SCORE_BUDGET=0 \
SGLANG_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"float8e4m3"}' \
SGLANG_DSV41_TORCH_PREFILL_INDEXER=0 SGLANG_DSV41_SM90_PREFILL_INDEXER=deepgemm \
SGLANG_OPT_DSV41_VERIFY_INDEXER_BF16_K_REUSE=1 \
sglang serve --model-path deepseek-ai/DeepSeek-V4.1-Flash --served-model-name llm \
  --host 127.0.0.1 --port 9090 --trust-remote-code --attention-backend dsv4 \
  --tp 4 --ep-size 1 --moe-runner-backend humming --moe-a2a-backend none \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 --enable-cache-report \
  --chunked-prefill-size 8192 --max-prefill-tokens 8192 --enable-prefill-cp --cp-strategy interleave \
  --enable-cp-decode-attn-tp --cp-decode-attn-mode dp --enable-decoder-swa-bounded-replay \
  --cuda-graph-backend-prefill breakable --cuda-graph-max-bs-prefill 8192 --cuda-graph-prefill-max-context 262144 \
  --cuda-graph-max-bs-decode 64 --mem-fraction-static 0.8 --max-running-requests 64 \
  --enable-hierarchical-cache --hicache-ratio 2 --hicache-io-backend kernel --hicache-write-policy write_through

Benchmark

Run the serving benchmark
python3 -m sglang.bench_serving --model deepseek-ai/DeepSeek-V4.1-Flash \
  --served-model-name llm --host 127.0.0.1 --port 9090 --backend sglang-oai \
  --dataset-name generated-shared-prefix \
  --num-prompts 256 --gsp-num-groups 1 --gsp-prompts-per-group 256 \
  --gsp-system-prompt-len 61440 --gsp-question-len 4096 --gsp-output-len 500 \
  --max-concurrency 24 --warmup-requests 10 --seed 42 --flush-cache --cache-report

GSM8K accuracy

Use the chat endpoint on the PD router for both configurations. Enable --reasoning-parser deepseek-v41 --enable-strict-thinking on the workers. The evaluator takes five questions for few-shot examples and scores the remaining 1,314.

Run GSM8K
python3 -m sglang.test.run_eval --eval-name gsm8k --api chat --num-examples 1319 \
  --host 127.0.0.1 --port 9090 --num-threads 256 --reasoning-effort high
PD configurationGSM8K64K retrievalDSpark tokens / verify
TP4 baseline96.50% (1,268/1,314)10/105.368
Final96.42% (1,267/1,314)10/105.460