SarvamSarvam105BAlibabaQwen 3.827BGoogleEmbeddingGemma 2DeepSeekDeepSeekV4.1 FlashDeepSeekDeepSeekV4 ProGoogleGemma 431BSarvamSarvamPrismSarvamSarvam 105BConversationsMoonshot AIKimiK3.1 ProCloudflareClefZ.aiGLM 5.3Z.aiGLM 5.3FlashMoonshot AIKimiK3.1 ProCloudflareClefZ.aiGLM 5.3Z.aiGLM 5.3FlashSarvamSarvam105BAlibabaQwen 3.827BGoogleEmbeddingGemma 2DeepSeekDeepSeekV4.1 FlashDeepSeekDeepSeekV4 ProGoogleGemma 431BSarvamSarvamPrismSarvamSarvam 105BConversationsDeepSeekDeepSeekV4 ProGoogleGemma 431BSarvamSarvamPrismSarvamSarvam 105BConversationsMoonshot AIKimiK3.1 ProCloudflareClefZ.aiGLM 5.3Z.aiGLM 5.3FlashSarvamSarvam105BAlibabaQwen 3.827BGoogleEmbeddingGemma 2DeepSeekDeepSeekV4.1 FlashSarvamSarvam105BAlibabaQwen 3.827BGoogleEmbeddingGemma 2DeepSeekDeepSeekV4.1 Flash

Models served on Sarvam Token Factory include Sarvam 105B, Gemma 4 31B, GLM 5.3, DeepSeek V4.1 Flash, Kimi K3.1 Pro, Qwen 3.8 27B, Sarvam Prism, GLM 5.3 Flash, DeepSeek V4 Pro, Clef, EmbeddingGemma 2, Sarvam 105B Conversations.

Nov 7, 2026·11 min read

Today, we are launching Sarvam Token Factory for everyone. Token Factory is our inference stack to serve our own models and leading open-weight models from GPU clusters located in India. On most of the models we serve, we match or beat the leading inference providers globally on both speed and cost.

Along with Sarvam 105B, we serve GLM 5.3, GLM 5.3 Flash, DeepSeek V4.1 Flash 1017, DeepSeek V4 Pro, Kimi K3, Kimi K3.1 Pro and Gemma 4 31B through one OpenAI-compatible API. See the full model catalogue.

Sarvam Token Factory at a glance

  • Cost: We offer globally competitive pricing on every model we serve. DeepSeek V4.1 Flash launches with introductory pricing of ₹14 per million input tokens and ₹55 per million output tokens.
  • Performance:
  • Latency: Gemma 4 31B returns its first token in 0.40 seconds on a 10K-token prompt, 5× faster than the only other provider serving it.
  • Throughput: Kimi K3.1 Pro delivers up to 1.9× tokens per GPU at the same per-user speed as InferenceX’s published results, measured on coding tasks.
  • Caching: We built a three-tier cache with cache-aware routing and admission for the newest attention architectures. On DeepSeek V4.1 Flash, we serve 450B+ tokens a day across diverse use cases, with 95.7% of prompt tokens served from cache.
  • Scale: We run India’s largest B200 cluster and are expanding our fleet to serve more models at population scale.
  • Deployment: Apart from Sarvam Token Factory, we also run this inference optimization stack on-premise for critical institutions.

Performance: Throughput & Latency

Gemma 4 31B is built for real-time conversation and chat in Indian languages. Measured from Mumbai on a 10K-token prompt, Gemma 4 31B returns its first token in 0.40 seconds and streams at 308 tokens per second.

Gemma 4 31B from Mumbai, 10K-token prompt

Time to first token, lower is better

Output speed, higher is better

Median over 72 hours from a client in AWS Mumbai. The other provider is the only other one serving the model; each of its bars is the best of its tiers on that metric.

Gemma 4 31B from Mumbai, 10K-token prompt. Time to first token: Sarvam Token Factory 0.40 s, Other provider 2.00 s. Output speed: Sarvam Token Factory 308 tokens per second, Other provider 193 tokens per second.

Gemma 4 31B from Mumbai, 10K-token prompt
ProviderTime to first token (s)Output speed (tokens/s)
Sarvam Token Factory0.40308
Other provider2.00193

DeepSeek V4.1 Flash 1017, one of the most used models on OpenRouter, is available through two endpoints. The Interactive at about 200 tokens per second per user and is built for coding agents. The Standard endpoint is built for background and batch work, with price and speed comparable to GPT-6 Luna and DeepSeek’s own API. Both endpoints serve the model at its native precision. Per million tokens, Interactive is priced at ₹25 for input and ₹105 for output, and Standard at ₹14 for input and ₹55 for output.

DeepSeek V4.1 Flash: price vs output speed

* Introductory pricing. Hover any point for exact values.

DeepSeek V4.1 Flash: price vs output speed. Sarvam Standard: blended price $0.092 per million tokens, output speed 130 tok/s. Sarvam Interactive: blended price $0.168 per million tokens, output speed 200 tok/s. DeepSeek API: blended price $0.092 per million tokens, output speed 60 tok/s. Together: blended price $0.184 per million tokens, output speed 278 tok/s. Baseten: blended price $0.185 per million tokens, output speed 165 tok/s. Fireworks: blended price $0.184 per million tokens, output speed 193 tok/s. Fireworks US: blended price $0.276 per million tokens, output speed 169 tok/s. Luna (OpenAI), GPT-6 Luna for reference: blended price $0.077 per million tokens, output speed 67 tok/s.

DeepSeek V4.1 Flash: price vs output speed
EndpointBlended price (USD per M tokens)Output tokens per second
Sarvam Standard0.092130
Sarvam Interactive0.168200
DeepSeek API0.09260
Together0.184278
Baseten0.185165
Fireworks0.184193
Fireworks US0.276169
Luna (OpenAI)0.07767

Measured from India, Token Factory matches the fastest provider on time to first token for GLM 5.3 with 10K-token prompts. We measured every provider the same way, using a recreation of Artificial Analysis’s method. Each test sends one prompt at a time, at 1K and 10K tokens, from a client in AWS Mumbai. We report the median over 72 hours.

(this image needs to be updated**)

Time to first token from Mumbai, lower is better

Sarvam Token FactoryTogetherFireworksDeepInfra

GLM 5.3: time to first token

DeepSeek V4.1 Flash: time to first token

Median, one request at a time from AWS Mumbai, using a recreation of Artificial Analysis’s method. Each provider at its fastest tier (Fireworks on GLM 5.3: Priority at 1K, default at 10K; DeepInfra on DeepSeek V4.1 Flash: Priority). Sarvam DeepSeek V4.1 Flash: Standard endpoint. Sarvam GLM 5.3 measured 6 Nov 2026; all other figures 3 to 5 Nov 2026.

Time to first token from Mumbai, lower is better. GLM 5.3: 1K-token prompt, Sarvam Token Factory 0.68 s, Together 0.82 s, Fireworks 0.84 s, DeepInfra 2.17 s; 10K-token prompt, Sarvam Token Factory 1.19 s, Together 1.19 s, Fireworks 1.57 s, DeepInfra 2.18 s. DeepSeek V4.1 Flash: 1K-token prompt, Sarvam Token Factory 0.51 s, Together 0.61 s, Fireworks 1.01 s, DeepInfra 1.08 s; 10K-token prompt, Sarvam Token Factory 0.90 s, Together 0.71 s, Fireworks 1.55 s, DeepInfra 1.75 s.

Time to first token from Mumbai, lower is better
Model and promptSarvam Token Factory (s)Together (s)Fireworks (s)DeepInfra (s)
GLM 5.3, 1K-token prompt0.680.820.842.17
GLM 5.3, 10K-token prompt1.191.191.572.18
DeepSeek V4.1 Flash, 1K-token prompt0.510.611.011.08
DeepSeek V4.1 Flash, 10K-token prompt0.900.711.551.75

We ran our stack against upstream vLLM and Dynamo, without our changes, on GLM 5.3 Flash, with both stacks disaggregated on the same B200 nodes, and against InferenceX’s published results on Kimi K3.1 Pro.

On GLM 5.3 Flash, running SemiAnalysis's AgentX coding-agent traces on the same two B200 nodes, our stack returns the first token up to 2.1× sooner than stock vLLM and Dynamo, while serving at the same throughput per GPU. At 64 concurrent sessions, the median first token takes 667 ms against 1,428 ms, and the slowest 1% wait 3.2 s against 14.1 s. Both stacks stream at about the same speed per user, because the gain comes from prefill, not decode.

At the same tokens per GPU, GLM 5.3 Flash answers up to 2.1× sooner

Sarvam's optimised stackStock open-source stack

1,428 ms → 667 ms

Median first token at 64 sessions, 2.1× sooner

14.1 s → 3.2 s

Slowest 1% of first tokens (p99) at 64 sessions, 4.4× sooner

190 → 192 tok/s

Median output speed per user at 64 sessions

Same AgentX coding-agent benchmark and the same test topology for both stacks. Measured with vLLM 0.31.0, Dynamo 1.6.0, NIXL 1.4.1 and Mooncake 0.3.13 for the stock open-source stack, against Sarvam's optimised stack.

At the same tokens per GPU, GLM 5.3 Flash answers up to 2.1× sooner. Sarvam's optimised stack at 1 sessions: median first token 699 ms, p99 2.3 s, 1,097 tokens per second per GPU, 248 per user. Sarvam's optimised stack at 4 sessions: median first token 503 ms, p99 1.7 s, 1,131 tokens per second per GPU, 234 per user. Sarvam's optimised stack at 16 sessions: median first token 551 ms, p99 2.7 s, 1,725 tokens per second per GPU, 229 per user. Sarvam's optimised stack at 32 sessions: median first token 646 ms, p99 2.2 s, 6,455 tokens per second per GPU, 220 per user. Sarvam's optimised stack at 64 sessions: median first token 667 ms, p99 3.2 s, 10,768 tokens per second per GPU, 192 per user. Stock open-source stack at 1 sessions: median first token 1,043 ms, p99 3.8 s, 1,038 tokens per second per GPU, 241 per user. Stock open-source stack at 4 sessions: median first token 912 ms, p99 9.7 s, 1,087 tokens per second per GPU, 232 per user. Stock open-source stack at 16 sessions: median first token 959 ms, p99 4.2 s, 1,692 tokens per second per GPU, 227 per user. Stock open-source stack at 32 sessions: median first token 1,168 ms, p99 14.2 s, 6,370 tokens per second per GPU, 215 per user. Stock open-source stack at 64 sessions: median first token 1,428 ms, p99 14.1 s, 10,193 tokens per second per GPU, 190 per user.

At the same tokens per GPU, GLM 5.3 Flash answers up to 2.1× sooner
StackSessionsFirst token p50 (ms)First token p99 (s)Tokens per GPU (tok/s)Output per user (tok/s)
Sarvam's optimised stack16992.31,097248
Sarvam's optimised stack45031.71,131234
Sarvam's optimised stack165512.71,725229
Sarvam's optimised stack326462.26,455220
Sarvam's optimised stack646673.210,768192
Stock open-source stack11,0433.81,038241
Stock open-source stack49129.71,087232
Stock open-source stack169594.21,692227
Stock open-source stack321,16814.26,370215
Stock open-source stack641,42814.110,193190

On SemiAnalysis’s agent traces, we serve up to 1.9× InferenceX’s tokens per GPU at the same per-user speed, from 14 to 48 concurrent sessions, against their published Kimi K3.1 Pro results.

At the same speed per user, Kimi K3.1 Pro serves 1.9× the tokens per GPU

Sarvam Inference StackInferenceX Stack

1.9× throughput

Same speed per user: 70 tok/s per user (median), our c24 against the InferenceX curve

9.1K → 13.4K tok/s

Peak tokens per GPU, best point on each curve (1.5×)

Kimi K3.1 Pro on 16 B200s, AgentX coding-agent traces. Labels are concurrent sessions. InferenceX figures are its published B200 vLLM results, run without speculative decoding at 96 sessions.

At the same speed per user, Kimi K3.1 Pro serves 1.9× the tokens per GPU. Sarvam Inference Stack at 1 sessions: 667 tokens per second per GPU at 112.2 per user. Sarvam Inference Stack at 4 sessions: 1,282 tokens per second per GPU at 91.2 per user. Sarvam Inference Stack at 8 sessions: 2,189 tokens per second per GPU at 86.1 per user. Sarvam Inference Stack at 14 sessions: 3,411 tokens per second per GPU at 83.3 per user. Sarvam Inference Stack at 24 sessions: 6,177 tokens per second per GPU at 69.5 per user. Sarvam Inference Stack at 48 sessions: 10,909 tokens per second per GPU at 48.1 per user. Sarvam Inference Stack at 72 sessions: 12,807 tokens per second per GPU at 31.2 per user. Sarvam Inference Stack at 96 sessions: 13,387 tokens per second per GPU at 21.9 per user. InferenceX Stack at 1 sessions: 711 tokens per second per GPU at 164.5 per user. InferenceX Stack at 4 sessions: 1,437 tokens per second per GPU at 107.2 per user. InferenceX Stack at 8 sessions: 2,271 tokens per second per GPU at 84.0 per user. InferenceX Stack at 14 sessions: 3,472 tokens per second per GPU at 65.4 per user. InferenceX Stack at 24 sessions: 6,106 tokens per second per GPU at 51.5 per user. InferenceX Stack at 48 sessions: 9,066 tokens per second per GPU at 32.8 per user. InferenceX Stack at 72 sessions: 4,523 tokens per second per GPU at 7.1 per user. InferenceX Stack at 96 sessions: 6,244 tokens per second per GPU at 8.0 per user. At 70 tokens per second per user, ours serves 6,177 tokens per second per GPU against 3,213 on the InferenceX curve.

At the same speed per user, Kimi K3.1 Pro serves 1.9× the tokens per GPU
StackSessionsMedian output per user (tok/s)Tokens per GPU (tok/s)
Sarvam Inference Stack1112.2667
Sarvam Inference Stack491.21,282
Sarvam Inference Stack886.12,189
Sarvam Inference Stack1483.33,411
Sarvam Inference Stack2469.56,177
Sarvam Inference Stack4848.110,909
Sarvam Inference Stack7231.212,807
Sarvam Inference Stack9621.913,387
InferenceX Stack1164.5711
InferenceX Stack4107.21,437
InferenceX Stack884.02,271
InferenceX Stack1465.43,472
InferenceX Stack2451.56,106
InferenceX Stack4832.89,066
InferenceX Stack727.14,523
InferenceX Stack968.06,244

At 72 concurrent sessions, InferenceX’s KV cache reaches 89% and starts evicting prefixes. Those prefixes get re-prefilled, and p95 TTFT climbs to 77 seconds. We use Moonshot’s Mooncake (G3), a shared cache that keeps prefixes alive across prefill servers instead of evicting them. Combined with per-engine session caps, p95 TTFT stays at 3.1 seconds under the same load.

Under heavy agent load, the shared cache keeps first-token time steady

72 concurrent sessions

Median (p50)

Slowest 5% (p95)

96 concurrent sessions

Median (p50)

Slowest 5% (p95)

12.8K vs 4.5K

72 sessions, tok/s per GPU, Sarvam vs InferenceX

13.4K vs 6.2K

96 sessions, tok/s per GPU, Sarvam vs InferenceX

Kimi K3.1 Pro on 16 B200s, AgentX coding-agent traces. InferenceX turns speculative decoding off at 96 sessions to free its KV cache. G3 keeps our evicted prefixes, so ours stays on.

Under heavy agent load, the shared cache keeps first-token time steady. 72 sessions (spec decode on for both; their KV 89% full): median first token Sarvam 0.8 s, InferenceX 11.6 s; 95th percentile Sarvam 3.1 s, InferenceX 76.6 s; throughput 12.8K vs 4.5K tokens per second per GPU. 96 sessions (InferenceX turns spec decode off): median first token Sarvam 1.1 s, InferenceX 0.9 s; 95th percentile Sarvam 9.3 s, InferenceX 18.8 s; throughput 13.4K vs 6.2K tokens per second per GPU.

Under heavy agent load, the shared cache keeps first-token time steady
SessionsConditionSarvam p50 (s)InferenceX p50 (s)Sarvam p95 (s)InferenceX p95 (s)Tokens per GPU, Sarvam vs InferenceX
72spec decode on for both; their KV 89% full0.7911.583.1376.6212.8K vs 4.5K
96InferenceX turns spec decode off1.080.879.2918.8313.4K vs 6.2K

Technical Deep Dive

Disaggregation, speculative decoding, quantisation and prefix caching are established techniques. Each new model tests the assumptions behind their implementation, from kernel correctness and attention-state storage to scheduling under real traffic. Each issue affects speed, cost or correctness. These issues need to be found and fixed within hours of a model’s release.

Our work spans three layers:

  • The inference engine decides how efficiently a GPU turns a request into tokens.
  • The cache decides how much of that work is done only once.
  • The control plane decides which GPU gets each request, when it gets it, and what is already in that GPU’s memory.

Token Factory: three layers around a shared cache

Requests

Control plane

Cache-aware router

sends each turn to where its prompt sits

Admission

counts only uncached tokens

Planner

scales on rejections and backpressure

Inference engine

Prefill pool

reads the whole prompt in one pass

tuned all-to-all, MXFP8 and NVFP4

Decode pool

generates one token at a time

fused clamp kernel, suffix-tree drafts

Shared cache

GPU memory

holds a prompt ~4 s

~2 s under load

G3 shared store (Mooncake)

read by every prefill server

holds prompts for hours or days, depending on traffic

The control plane routes, admits and scales. Prefill and decode run on separate GPU pools and hand over KV and model state at aligned checkpoints. Every prefill server reads the shared G3 store, which holds a prompt far longer than GPU memory.

Token Factory: three layers around a shared cache. Requests enter the control plane: Cache-aware router: sends each turn to where its prompt sits; Admission: counts only uncached tokens; Planner: scales on rejections and backpressure. The router sends work to the inference engine's Prefill pool: reads the whole prompt in one pass; tuned all-to-all, MXFP8 and NVFP4, which passes KV and model state at aligned checkpoints to the Decode pool: generates one token at a time; fused clamp kernel, suffix-tree drafts. The planner scales the decode pool. The prefill pool reads two cache tiers: GPU memory: holds a prompt ~4 s; ~2 s under load; G3 shared store (Mooncake): read by every prefill server; holds prompts for hours or days, depending on traffic.

Token Factory: three layers around a shared cache
ComponentLayerWhat it does
Cache-aware routerControl planesends each turn to where its prompt sits
AdmissionControl planecounts only uncached tokens
PlannerControl planescales on rejections and backpressure
Prefill poolInference enginereads the whole prompt in one pass; tuned all-to-all, MXFP8 and NVFP4
Decode poolInference enginegenerates one token at a time; fused clamp kernel, suffix-tree drafts
GPU memoryShared cacheholds a prompt ~4 s; ~2 s under load
G3 shared store (Mooncake)Shared cacheread by every prefill server; holds prompts for hours or days, depending on traffic

Long agent sessions needed prefill and decode on separate GPUs

Every request goes through two phases. Prefill reads the whole prompt in one compute-heavy pass, and decode then generates tokens one at a time, limited by memory bandwidth. When both share a GPU, each new 100K-token coding prompt stalls every session already decoding there. We run the two phases on separate pools, each configured for its own work, and added EFA support to the KV transfer layer alongside InfiniBand so the split works across all our clusters.

Splitting prefill and decode is now standard. Further gains depend on transferring the cache, attention state and draft model state correctly between GPUs. We keep real-time voice workloads on the same GPU instead of using a disaggregated architecture because they have shorter input tokens.

Kernel tuning across models helps fix correctness and performance both

GLM 5.3 Flash clamps expert activations to a fixed range. vLLM’s fused decode kernel skipped this step, causing deviations of up to 165% from the reference and stray characters in place of code indentation. We fused clamping, activation, multiplication and quantisation into one kernel. It matches the reference bit for bit on in-range inputs and runs about 40% faster. On AgentX, the missing clamp showed up as stray Chinese characters in 32 of 299 long code responses. With the clamp in place, there were none.

Two other bugs caused garbled output from the first token. The kernel that initialises the indexer cache lacked a position input, so request tails wrote to the same cache block and raced with each other. Rejected draft tokens could also overwrite committed keys. After we fixed both, our test bench returned without garbled output, and draft acceptance rose from 38% to 84%, improving speed alongside correctness.On GLM 5.3 Flash, we traced prefill delays to the all-to-all exchange between GPUs. vLLM allocates 20 GPU compute units to this exchange. Increasing that to 64 : 96 and tuning chunk sizes cut median time to first token from 28.7 s to 4.7 s at a 90% cache hit rate. With two additional kernel changes, uncached prefill throughput rose from about 14K to 22K tokens per second per GPU.

On DeepSeek V4.1 Flash 1017, DeepGEMM’s MegaMoE kernels increased decode throughput by 28% at 128 concurrent sessions. Upgrading to vLLM v0.32 raised uncached prefill throughput from about 19K to 41K tokens per second per GPU, measured at an 84% cache hit rate.

Moving to MXFP8

We measure every format end to end on our own traffic before it ships. GLM 5.3 moved from FP8 weights on SGLang to MXFP8 weights on vLLM. MXFP8 is an 8-bit format with a shared scale for every block of 32 values, and B200’s tensor cores run it natively. Uncached prefill per GPU doubled, from about 3.7K to 7.6K tokens per second, decode rose 25%, and p95 first-token time on real chat halved, with accuracy unchanged against the FP8 baselines. MXFP8 has served GLM 5.3 in production since mid-October.

Moving GLM 5.3 to MXFP8 on vLLM doubled uncached prefill per GPU

Uncached prefill

2.1×

Decode

+25%

Real chat, TTFT p95

2.1× lower

GLM 5.3 with FP8 weights on SGLang against MXFP8 weights on vLLM, on the same nodes (1 prefill + 1 decode, 16 B200) and the same tests.

Moving GLM 5.3 to MXFP8 on vLLM doubled uncached prefill per GPU. Uncached prefill: SGLang FP8 3,700, vLLM MXFP8 7,600 input tokens per second per GPU. Decode: SGLang FP8 5,226, vLLM MXFP8 6,519 output tokens per second. Real chat p95 time to first token: SGLang FP8 19.6 s, vLLM MXFP8 9.2 s.

Moving GLM 5.3 to MXFP8 on vLLM doubled uncached prefill per GPU
MeasureSGLang FP8vLLM MXFP8
Uncached prefill (input tokens per second per GPU)3,7007,600
Decode (output tokens per second)5,2266,519
Real chat, p95 time to first token (s)19.69.2

Multi-tier cache for context reuse across agent turns

Agent prompts on GLM 5.3 average 131K tokens. After a tool call, the agent returns tens of seconds or minutes later with almost the same prompt. GPU memory retained that context for only about 4 seconds, falling to 2s under load. Nearly every returning turn required a full prefill, repeating the compute and adding latency. We added G3 to every prefill server to retain context so agents can reuse it when they return.

Over a week in production On DeepSeek, the combined cache hit rate rose from 91.4% to 95.7%, halving the tokens that needed prefill. During peak traffic on GLM 5.3, G3 reduced the required prefill nodes from 58 to 34.

Where prompt tokens were served in production

Share of prompt tokens by where they were served

GPU cacheG3Recomputed
Production prefill counters over seven days, 30 October to 6 November. Prefill compute scales with the share of prompt tokens that miss every cache, so taking DeepSeek V4.1 Flash from 91.4% to 95.7% served halves the prefill it runs.

Where prompt tokens were served in production. DeepSeek V4.1 Flash, 872B prompt tokens: 91.4% from the GPU cache, 4.3% from G3, 4.3% recomputed. GLM 5.3, 164B prompt tokens: 88.6% from the GPU cache, 3.4% from G3, 8.1% recomputed.

Where prompt tokens were served in production
ModelPrompt tokensGPU cache (%)G3 (%)Recomputed (%)
DeepSeek V4.1 Flash872B prompt tokens91.44.34.3
GLM 5.3164B prompt tokens88.63.48.1

Aligning the multi-tier cache with new attention architectures

On DeepSeek V4.1 Flash 1017, sliding-window cache lookups and writes used different positions, preventing G3 hits for 74% of prompt lengths. The context was cached but couldn’t be found, so the GPUs processed it again. We aligned the positions, raising the cold cache hit rate from 37.1% to 99.8% and cutting latency from 2.80 s to 0.64 s, with no additional storage. For other hybrid models, we save sliding-window state at every turn boundary in both GPU memory and G3. This reduced returning conversations with no reusable context from 78% to 8.5%.

DeepSeek V4.1 Flash: which prompt lengths hit the cache

Miss: full re-prefillHitHit, after the fix

37.1% → to 99.8%

G3 cold hit rate

2.80 s → to 0.64 s

Latency

1,878 B/token

Storage cost, same as the GPU cache, no extra blocks

Not fixed

Upstream: GPU-cache fix #57613 is still open and stores an extra block per boundary

A DeepSeek V4.1 Flash cache hit is the minimum across five KV groups, so one missing sliding-window group forces a full re-prefill while the request still returns success. Lookup and store used different anchors for the sliding-window KV; we anchored both on the same rule. Each cell is one value of prompt length modulo 128.

DeepSeek V4.1 Flash: which prompt lengths hit the cache. Each cell is one value of prompt length modulo 128. GPU cache, before, misses at remainders 1 to 32: 25% of lengths miss. G3 store, before, misses at remainders 33 to 127: 74.2% of lengths miss. Both, after, every remainder hits: 0% miss. G3 cold hit rate rose from 37.1% to 99.8%. Latency fell from 2.80 s to 0.64 s. Storage cost is 1,878 bytes per token, the same as the GPU cache. Upstream: not fixed.

DeepSeek V4.1 Flash: which prompt lengths hit the cache
CacheRemainders that missShare of lengths that miss
GPU cache, before1 to 3225%
G3 store, before33 to 12774.2%
Both, afternone0%

Making the control plane cache-aware

Dynamo’s router accounts for context cached on GPU, host memory and disk, but with vLLM and G3, it missed shared Mooncake cache hits and counted cached tokens as new prefill work. Admission limits counted those tokens too, separately per frontend, rejecting 1,381 of 1,400 requests in a cache-heavy test while GPUs sat mostly idle. The planner couldn’t see the rejections, so short queues masked unmet demand. We updated routing to account for G3 hits and congestion, balancing context reuse with capacity. Admission now counts only uncached tokens under one shared in-flight limit across frontends, and the same test completed without rejections.

Router cache-hit accounting and tool-call grammar, before and after

Requests sent to the wrong worker

17.6% → to 3.5%

Time to first token, p95

24 s → to 9 s

Tool calls leaked as raw text

7.7% → to 0.08%

Router figures are at 128 concurrent sessions. Tool-call leaks are DeepSeek V4.1 Flash calls returned as raw DSML text, measured on 31 replayed production conversations.

Router cache-hit accounting and tool-call grammar, before and after. At 128 concurrent sessions, requests sent to the wrong worker fell from 17.6% with stock accounting to 3.5% once fixed, and p95 time to first token fell from 24 s to 9 s. DeepSeek V4.1 Flash tool calls leaked as raw text fell from 7.7% with no grammar to 0.08% with a DSML grammar.

Router cache-hit accounting and tool-call grammar, before and after
MeasureBeforeAfter
Requests sent to the wrong worker (stock accounting vs fixed)17.6%3.5%
Time to first token, p95 (stock accounting vs fixed)24 s9 s
Tool calls leaked as raw text (no grammar vs DSML grammar)7.7%0.08%

Stock planner and our planner on the same replayed load

Prefill GPU-hours, 104 minutes from load start

47% fewer

Worst minute, p95 time to first token

Inside the SLA

1,050 vs 1,054

requests completed

78.5%

cache hit rate in both runs

0.3 → 0.5 → 1

requests per second over the 30-minute ramp

40 (about 4%)

briefly turned away by ours at the peak

The same synthetic agentic load, replayed on one production GLM 5.3 deployment under each planner. Ours used 47% fewer prefill GPU-hours and kept p95 first token inside the 10-second SLA.

Stock planner and our planner on the same replayed load. Prefill GPU-hours over 104 minutes: Stock planner 32.3, Our planner 17.1. Worst-minute p95 time to first token: Stock planner 19.7 s, Our planner 9.96 s, against a 10 s SLA. Both runs: 1,050 vs 1,054 requests completed, 78.5% cache hit rate in both runs, load ramped 0.3, 0.5, 1 requests per second over 30 minutes; 40 (about 4%) briefly turned away by ours at the peak. Timeline, Stock planner: about 1 min, 2nd pod at 2% busy; about 21 min, p95 19.7 s, 3rd pod at 9% busy; about 105 min, still 2 pods. Our planner: about 20 min, 40 requests get a 529; about 23 min, 2nd pod serving; about 47 min, back to 1 pod.

Stock planner and our planner on the same replayed load
MeasureStock plannerOur planner
Prefill GPU-hours, 104 minutes32.317.1
Worst-minute p95 time to first token (s)19.79.96
Stock planner event: 2nd pod at 2% busy (min)1
Stock planner event: p95 19.7 s, 3rd pod at 9% busy (min)21
Stock planner event: still 2 pods (min)105
Our planner event: 40 requests get a 529 (min)20
Our planner event: 2nd pod serving (min)23
Our planner event: back to 1 pod (min)47

New architectures carry state that the handoff between GPUs was dropping

Disaggregated serving was dropping model-specific state in the prefill-to-decode handoff. The transfer code assumed a standard attention KV cache.

Kimi K3.1 Pro and GLM 5.3 Flash’s draft model both carry extra state, and it wasn’t reaching the decode GPU intact.

On Kimi K3.1 Pro, the prefill GPU, router and decode GPU each used different checkpoint positions. Aligning them raised multi-turn cache hits from 39% to 73%.

On GLM 5.3 Flash, only part of the draft model’s cache was transferred, so accepted draft tokens fell from about 5 per step to 1. Matching the cache layout on both GPUs brought acceptance back to 4.7.

Aligning state checkpoints kept the handoff from losing work

Before

Checkpoints at different positions; unmatched ones (hollow) lose state

After

All three place checkpoints at the same 7,680-token boundaries

Kimi K3.1 Pro, multi-turn conversations

39% → to 73%

GLM 5.3 Flash drafter, accepted tokens per step

4.92 → to 1.05 → to 4.69

When prefill and decode run on separate GPUs, the state built in prefill has to reach decode intact. Kimi K3.1 Pro's linear-attention layers and the GLM 5.3 Flash draft model carry state beyond a plain attention cache. Aligning checkpoint positions raised Kimi K3.1 Pro multi-turn cache hits from 39% to 73%. Matching the draft cache layout on both sides brought GLM 5.3 Flash acceptance back to 4.69 tokens per step.

Aligning state checkpoints kept the handoff from losing work. Before: the prefill GPU, router and decode GPU place state checkpoints at different positions, so only the first and last line up and the rest lose state. After: all three place checkpoints at the same 7,680-token boundaries. Kimi K3.1 Pro multi-turn prefix cache hit rate: 39% misaligned, 73% aligned. GLM 5.3 Flash drafter accepted tokens per step: 4.92 before the split, 1.05 split and misaligned, 4.69 aligned.

Aligning state checkpoints kept the handoff from losing work
MeasureBefore splitMisalignedAligned
Kimi K3.1 Pro multi-turn prefix cache hit rate (%)3973
GLM 5.3 Flash drafter, accepted tokens per step4.921.054.69

Bringing it all together to serve Sarvam 105B more efficiently

With Sarvam 105B, we control both the weights and the inference engine, so we can optimise them together for the workloads we serve. Most traffic comes from agents with long system prompts, growing context from tool results and short replies. Moving from BF16 to MXFP8 weights with DSpark speculative decoding increased output throughput by about 2.2× while reducing each replica from two B200s to one. We also fixed a vLLM bug that skipped CUDA graphs after warm-cache restarts, leaving GPU utilisation at 38%. Four kernel and precision changes added another 30% output per GPU on agent workloads. These included 4-bit NVFP4 experts calibrated on real MoE inputs and a custom Triton kernel for 105B’s 64 attention heads, replacing a FlashInfer kernel that assumed 128. When agent sessions exceeded GPU cache capacity, G3 increased throughput by 2.21× and cut the share of prompt tokens needing recomputation from 56.8% to 17.6%. None of these changes reduced accuracy.

Sarvam 105B: cumulative speedup by optimisation step

vLLM 0.32 on B200. Steps one to three are measured against the BF16 baseline on two B200s, the rest on agentic traffic against the one-B200 MXFP8 setup. The engine-kernel step is measured on the decode step.

Sarvam 105B: cumulative speedup by optimisation step. BF16 baseline (2× B200): 1.00× cumulative speedup against the BF16 baseline. Engine kernels (compiled MLA): 1.73× cumulative speedup against the BF16 baseline. torch.compile + CUDA graphs (models team): 2.20× cumulative speedup against the BF16 baseline. NVFP4 experts: 2.42× cumulative speedup, plus 9.8% over the previous step on agentic traffic. Triton K-concat (for MLA): 2.59× cumulative speedup, plus 7.2% over the previous step on agentic traffic. FP8 prefill attn (TRT-LLM ragged): 2.76× cumulative speedup, plus 6.7% over the previous step on agentic traffic. Fused FP8 K/V: 2.86× cumulative speedup, plus 3.6% over the previous step on agentic traffic. The last four steps are chained onto 2.20×.

Sarvam 105B: cumulative speedup by optimisation step
StepCumulative speedupGain over previous step
BF16 baseline (2× B200)1.00×
Engine kernels (compiled MLA)1.73×
torch.compile + CUDA graphs (models team)2.20×
NVFP4 experts2.42×+9.8%
Triton K-concat (for MLA)2.59×+7.2%
FP8 prefill attn (TRT-LLM ragged)2.76×+6.7%
Fused FP8 K/V2.86×+3.6%

Kimi K3.1 Pro shows how quickly we can optimise a model we didn’t train. At 1,454 GiB, Kimi K3.1 Pro runs across two nodes using pipeline parallelism. We started with the pre-v0.32 vLLM nightly used in InferenceX’s B200 recipe. On our EFA clusters, the two pipeline stages deadlocked during warm-up. We fixed this by connecting the stages before warm-up and giving both the same warm-up batch. Profiling showed GPUs were busy for 92–97% of each decode step. Expert GEMMs took the largest share; sparse attention accounted for less than 4%. Further gains required reducing the work per step.

Experts and communication take over half of a Kimi K3.1 Pro decode step

Median of 241 decode steps on the first pipeline stage (52 layers), 8 GPUs, 160 concurrent sessions: 31.7 ms per step. GPUs are busy 92 to 94% of the window (95 to 97% on the second stage), so the gains came from removing work. Shares are of total GPU kernel time on the stage. With sparse attention, attention is under 4% of it.

Experts and communication take over half of a Kimi K3.1 Pro decode step. One 31.7 ms decode step, share of GPU kernel time: Expert GEMMs (MXFP4 weights) 31.9%; Communication (tensor + pipeline parallel) 24.1%; Dense GEMMs (MXFP8) 17.3%; Expert routing 5.1%; Linear attention (KDA) 4.1%; Attention (gated sparse (DSA) + DCP) 3.8%; Everything else (norms, copies, sampling, other GEMMs) 13.7%.

Experts and communication take over half of a Kimi K3.1 Pro decode step
Kernel groupDetailShare of GPU time (%)
Expert GEMMsMXFP4 weights31.9
Communicationtensor + pipeline parallel24.1
Dense GEMMsMXFP817.3
Expert routing5.1
Linear attentionKDA4.1
Attentiongated sparse (DSA) + DCP3.8
Everything elsenorms, copies, sampling, other GEMMs13.7

Every change was measured: we kept the six that paid off

Each change was measured on its own on the decode stack Kimi K3.1 Pro runs on: one decode deployment, two nodes, pipeline-parallel.

Every change was measured: we kept the six that paid off. Kept: One linear-attention state copy during speculation +22%; CUDA-graph sizes 320, 384, 448 +16.7%; Skip CUDA-graph padding rows in MoE routing +12.7%; Uneven pipeline split across two nodes +9.8%; DSpark with 3 draft tokens instead of 4 +6.2%; MXFP8 for BF16 linears on decode +6%. Flat, not kept: Overlap last stage draft with next step flat; Expert load balancing (EPLB) flat.

Every change was measured: we kept the six that paid off
Change testedWhat was measuredMeasured change (%)Outcome
One linear-attention state copy during speculationKV use 40% instead of 99%+22%Kept
CUDA-graph sizes 320, 384, 448no padding a 352-request step to 512+16.7%Kept
Skip CUDA-graph padding rows in MoE routingper user at 256 sessions+12.7%Kept
Uneven pipeline split across two nodesthroughput+9.8%Kept
DSpark with 3 draft tokens instead of 4at 320 sessions+6.2%Kept
MXFP8 for BF16 linears on decodeper user on real code, 7.7 GiB freed+6%Kept
Overlap last stage draft with next stepflat once its NCCL deadlock was fixedflatFlat, not kept
Expert load balancing (EPLB)load already 0.96 to 0.98 balancedflatFlat, not kept

Agents explore the search space and engineers decide

Each new attention variant, architecture and workload adds more configurations to test. An orchestrator agent reads fleet telemetry and assigns experiments to sub-agents working on different layers of the stack. They use profilers, a simulator calibrated on our servers and our evaluation suite to test each change. Simulation filters out most candidates before live A/B testing. An engineer approves every change before deployment. This lets us run dozens of experiments per model in days, without engineers setting up each one manually. We build on the work of the teams behind vLLM, SGLang, Dynamo, Mooncake, NIXL, DeepEP, DeepGEMM and FlashInfer.

Availability

Sarvam Token Factory is available today to developers and enterprises. We’ll continue to expand the models available, supporting new open-weight models from day one and adding compute as demand grows. We’re also building more tools for developers to build and deploy agents on Token Factory.

We already run this stack on-premise for critical institutions and offer the same deployment option to enterprises. Teams can choose the models that fit their workloads and build an AI stack they own, without lock-in.

AnnouncementProductNov 7, 2026