StartLux open-sources decision model inference engine Prism, up to 5.3x faster inference on a single card

After DeepSeek, StartLux follows: open-sources the first decision model inference engine, inference up to 5.3x faster. Zhidongxi download client, exclusive first look, October 23, 2026 19:50:11 from Beijing

On October 20, DeepSeek officially open-sourced DeepInfer, the inference framework actually used in its production environment. Almost simultaneously, on October 22, the domestic AI startup team StartLux (Yuandian Xinghui) also open-sourced its self-developed inference engine Prism.

Both are self-developed inference systems, but DeepSeek's DeepInfer serves the cloud-scale business of its own general-purpose large models; Prism, by contrast, focuses on "decision models," and according to public information, it is also the first inference engine built specifically for decision models and open-sourced to the public.

Decision models are a new large-model track that has rapidly gained popularity recently. Unlike generative large models that focus on dialogue and content creation, decision models do not need to output large blocks of text; after reading the material, they directly output judgment results, giving probabilistic conclusions such as yes/no, option selection, and risk-level scores.

Since the Jev model went viral in September this year, decision models have quickly become an industry focus, with Microsoft, AWS, Cloudflare and other companies successively entering the field.

In the domestic decision model track, StartLux (Yuandian Xinghui), founded less than five months ago, has already achieved good results. On October 16, the team launched a new-generation decision model StartLux‑Decision-v1.1, which ranked first on Hugging Face's Decision Index 0.3.2 leaderboard with a score of 69.1, ranking first among 173 publicly reproducible models.

As of now, the StartLux decision model has accumulated more than 180,000 downloads, and the decision model API has cumulatively responded to more than 3 million decision requests from around the world.

Github inference engine:

Github decision model:

Github decision model leaderboard:

Huggingface decision model:

Huggingface decision model leaderboard:

However, even if model capabilities achieve breakthroughs, there are still practical bottlenecks in deployment.

Previously, StartLux-Decision-v1.1 ran on a general inference framework based on vLLM 0.31. However, mainstream frameworks such as vLLM are natively designed for token-by-token generation scenarios such as chat and text continuation, and are not suited to decision models, making it difficult to fully unleash the model's actual capabilities.

Prism is StartLux's solution: rewriting the inference engine according to the needs of decision models.

In actual tests on a single H200 GPU, Prism processes a single short question 2.8 times faster than the original solution, and when asking three questions in succession about the same material, it can be up to 5.3 times faster.

What key changes did this inference engine, rebuilt specifically for decision models, actually make? What are the measured results for each performance metric? Why are more and more AI companies choosing to develop their own inference frameworks rather than directly using general community solutions? This article will interpret the above questions.

1. StartLux uses four major innovations to solve the inference efficiency problem

To understand Prism's optimization approach, one must first clarify the operating logic of decision models.

For example, a customer complains: "I was charged twice for the order, but the App still shows unpaid." Upon receiving this complaint, the system needs to simultaneously determine: which department the ticket should go to, whether a reply is required today, and how serious the problem is. The decision model needs to read the material, score the candidate answers for each question separately, and return the probability of each option.

In contrast, general inference frameworks such as vLLM and SGLang were originally designed to serve text generation tasks. If they are used to run decision models, each question must be broken down into an independent request, separately completing network requests, tokenization, and scheduling, and repeatedly executing the computation of the huge output layer.

All of Prism's optimizations revolve around one core goal: to complete one decision while performing only the essential computation.

Optimization 1: Narrowing the scope of output computation

Taking the 27B-parameter StartLux‑Decision model as an example, its output layer contains 1.27 billion weights, and a general engine needs to fully score 248,000 candidate words, but decision tasks only use the 26 letters A‑Z as output options.

Prism compresses the scope of output layer computation to 0.01% of the original, while the model still fully completes material understanding, only skipping the computation of a large number of irrelevant candidate words at the stage of outputting judgment results.

Optimization 2: "Forest-style" computation

In business scenarios, the same material often corresponds to multiple judgment requirements. For example, a customer ticket needs to simultaneously determine the team it belongs to, whether a same-day reply is needed, and the severity level of the fault. Traditional frameworks must fully process the ticket material again for each question answered.

Prism innovatively constructs batch tasks into a "forest-style" computation structure: general instructions serve as the tree roots, shared reference material serves as the tree trunk, and each independent question serves as a branch.

The computation corresponding to the tree roots and trunk is executed only once, and different questions only process their own newly added parts. Multiple rounds of follow-up questions about the same material can also reuse existing computation results.

Optimization 3: Using integer arithmetic

When interacting with large language models, we often encounter a problem: for the same question, the large model gives different answers each time, sometimes even completely opposite ones.

This is fatal for decision models, because the probability values output by decision models directly guide business systems in executing subsequent actions.

Prism solves the result drift problem through integer arithmetic: all related matrix operations use int8 precision, with accumulation completed at int32 precision. Because integer addition is not affected by computation order, the computation results are not disturbed by batch size or scheduling order.

Optimization 4: Mixed scheduling

StartLux-Decision-v1.1 supports AutoThink automatic thinking mode: when the model is sufficiently confident, it outputs the answer directly; when confidence is insufficient, it first generates a reasoning process of up to 4096 tokens.

But this also brings scheduling challenges: if short requests and long-thinking requests enter the same task queue, simple tasks will be blocked by long-duration requests.

Prism designed a hybrid scheduling mechanism for this: prompts for thinking-type tasks are merged into direct-output batches for preprocessing, and subsequent decoding steps are precompiled into CUDA graphs, supporting up to 64 thinking tasks and direct-output tasks time-sharing GPU resources. Newly arriving simple requests wait at most 1-2 decoding steps before they can jump the queue for execution.

2. Inference up to 5.3x faster, 231 questions average only 41 milliseconds

After such optimization, how much performance improvement is there? StartLux conducted a series of comparative tests on this.

(Note: The following data come from StartLux's measured results under different model versions and test conditions. Among them, the 5.3x speedup test used the StartLux-Decision-v1.1-27B AutoThink model; the 231-question cross-engine performance comparison and throughput test used the StartLux-Decision-v1-27B model. The specific test configurations and results have all been made public on GitHub.)

Test 1: Single-request latency

The test results show that in the single short-question scenario, Prism takes 19 milliseconds, while the original solution takes 53.1 milliseconds, a 2.8x speedup. For continuously processing three questions on the same material, Prism needs only 17.4 milliseconds, while the original solution needs 92.9 milliseconds, a speedup of 5.3x.

If thinking mode is enabled on top of the three questions, Prism takes 785 milliseconds, while the original solution takes 1296 milliseconds, a 1.7x speedup. For a single long question of about 8192 tokens, the performance gap between the two narrows to 1.2x.

In a more intuitive video understanding test, StartLux-Prime is 3.38x faster than vllm, and the output probabilities are basically consistent.

Test 2: Cross-engine latency

The research team selected a total of 231 questions from the JevBench test set and benchmarked Prism horizontally against mainstream open-source inference frameworks such as vLLM, SGLang, and llama.cpp.

The results show that Prism's average latency per question is 41 milliseconds, ranking first among all tested frameworks. The next closest, the decider compiled version, is 66.8 milliseconds, SGLang is 75.6 milliseconds, vLLM reaches 112.7 milliseconds, and llama.cpp takes the longest, averaging 589 milliseconds per question. Under the same condition of 8-bit precision, Prism's latency is about 40% lower than the second-fastest framework.

Two test premises need to be explained here: the decider and JevK5 engines cannot directly run StartLux's native models, so the tests used substitute models with consistent structure, comparing only speed and not accuracy; some other frameworks do not support 8-bit quantization and could only run at higher precision.

Test 3: Throughput

In the high-concurrency throughput test, with 64-way concurrency fully loading the GPU, Prism's advantage over competitors narrows to 17% to 22%. At this stage the bottleneck is at the hardware compute level; both sides have already maxed out the performance of the H200, and software optimization can hardly bring significant improvement.

But the most common decision-type business is not the extreme full-load scenario, but small requests that occur at high frequency. Under this typical daily load, Prism can process 100 requests per second, while the best-performing competitor handles 38 per second, with Prism leading by 2.6x.

In other words, Prism's performance advantage falls exactly in the business range that enterprises actually use at high frequency, which is also the core value of a dedicated inference engine.

Test 4: Flexible adaptation for multi-card deployment

At the multi-card deployment level, Prism supports both data parallelism and tensor parallelism, and they can also be used in combination to adapt to different business needs.

Taking the 27B version model as an example, four H200s in data parallelism mode each independently receive requests, processing a total of 121 requests per second, about 3.92x the single-card throughput. If the four cards use tensor parallelism to jointly compute the same question, single-question latency can drop from 41 milliseconds to 29 milliseconds.

Two dual-card hybrid deployments can balance throughput and latency, achieving 86 requests per second and 36 milliseconds per question. Regardless of the deployment method, the output results are bit-for-bit identical to running on a single card.

Test 5: Precision and deployment threshold

Beyond the speedup, the model's inference precision has not degraded.

In a thinking-enabled test covering seven evaluation sets and a total of 12,351 decision samples, the Prism int8 version achieved an average accuracy of 92.94%, the original engine's FP8 version was 92.83%, and the unquantized BF16 original version was 92.97%. The accuracy gap between the Prism quantized version and the full original version is only 0.03 percentage points, even slightly better than the original engine's quantized version.

The deployment threshold is also lower than expected. This model has a compressed version of 16.55GB, which can run on a single consumer-grade RTX 5090 graphics card with 32GB of VRAM, with output comparable to the professional-grade H200 graphics card. On this compressed version, Prism is 3.9x faster than the mainstream open-source framework llama.cpp.

3. Inference systems pull ahead, model companies begin to develop their own inference engines

Within just two days, DeepSeek and StartLux successively open-sourced their self-developed inference engines, reflecting a common shift in the large-model industry: beyond competition in model capabilities, the importance of inference engineering is rising rapidly.

Whether it is a general-purpose large model or a vertical-domain decision model, once embedded in core business processes, the latency of each call and the compute cost per ten thousand requests will be directly reflected in business costs. The deployment cost, response speed, and operational stability of the same model have begun to become key factors determining its commercialization space.

General-purpose inference frameworks need to adapt to models of different structures and cover vastly different application scenarios, and inevitably must compromise on specialization; only model vendors themselves best understand their own models' interface characteristics and optimization space.

StartLux is taking exactly this self-developed path, while also carving out its own open style in the decision model niche.

Unlike many vendors that only open-source model weights or inference code, every StartLux product release simultaneously outputs the underlying methodology and supporting toolchain, turning its own practical experience into public assets that the industry can reuse.

When it previously released the StartLux-Decision decision model, the team simultaneously made public the RLCD² reinforcement learning training algorithm, systematically solving the training challenge of decision models' "when to start thinking and when to stop thinking," filling a gap in decision model training methodology. The DecisionPivot evaluation benchmark launched at the same time also filled the industry's gap in evaluation standards for decision models' detail recognition capability, providing a unified comparative reference for the whole industry.

The open-sourcing of the Prism inference engine this time continues this approach. The team not only released the complete engine code but also made public the full set of optimization logic for decision model inference, equivalent to fully opening the core design ideas of decision model inference engineering to the industry.

Open-source AI vendors such as DeepSeek and StartLux turn inference engineering from an internal capability of a single vendor into reusable, verifiable public infrastructure for the industry.

For the decision model track, which is still in its early stages of development, this open model of "model + method + tools" directly lowers the trial-and-error cost for the whole industry, allowing more teams to avoid building the complete chain of training, evaluation, and deployment from scratch and to focus on upper-layer scenario innovation and performance optimization.

Conclusion: In the competition over decision models, the inference layer will be the deciding factor

To truly apply decision models in real business scenarios such as risk control, content moderation, and business ticket routing, having only high-performance models is far from enough.

The essence of Prism is to recalculate the entire set of inference computation logic for decision models. When the output of large models shifts from long-form text to a set of probabilistic judgments, the various default assumptions of inference engines from the text generation era all have room to be re-examined and re-optimized.

Future competition in the decision model track will also evolve in this direction: the gap in model architecture will keep narrowing, but barriers at the inference engineering level will become more pronounced, and the latter cannot be simply closed by stacking models with larger parameter counts.

This actually makes the core of competition in decision models and the entire model field more prominent, and makes the outside world even more eager to see who can run farther and go deepest.

"Special statement: The above content (including videos, pictures, or audio if any) is uploaded and published by users of "Dafeng Hao," a self-media platform under Phoenix Net. This platform only provides information storage space services.

Notice: The content above (including the videos, pictures and audios if any) is uploaded and posted by the user of Dafeng Hao, which is a social media platform and merely provides information storage space services."