StartLux open-sources the Prism inference engine for decision models, delivering up to 5.3x faster single-card inference

On October 20, DeepSeek officially open-sourced DeepInfer, the inference framework actually used in its production environment. Almost simultaneously, on October 22, the domestic AI startup team StartLux also open-sourced its self-developed inference engine Prism.

Both are self-developed inference systems, but DeepSeek's DeepInfer serves the cloud-scale, large-scale business of its own general-purpose large models, while Prism focuses on "decision models." According to public information, it is also the first inference engine specifically built for decision models and open-sourced to the public.

Decision models are a new large-model track that has rapidly gained popularity recently. Unlike generative large models that focus on dialogue and content creation, decision models do not need to output long passages of text; after reading the materials, they directly output judgment results, giving probabilistic conclusions such as yes/no, option selection, and risk-level scores.

Since the Jev model went viral in September this year, decision models have quickly become an industry focus, with Microsoft, AWS, Cloudflare and other companies successively entering the field.

In the domestic decision-model track, StartLux, which has been established for less than five months, has already achieved impressive results. On October 16, the team launched a new-generation decision model, StartLux‑Decision-v1.1, which ranked first on Hugging Face's Decision Index 0.3.2 with a score of 69.1, ranking first among 173 publicly reproducible models.

As of now, the StartLux decision model has accumulated more than 180,000 downloads, and the decision model API has cumulatively responded to more than 3 million decision requests from around the world.

Github inference engine:

Github decision model:

Github decision model leaderboard:

Huggingface decision model:

Huggingface decision model leaderboard:

However, even if model capabilities achieve a breakthrough, there are still practical bottlenecks in deployment.

Previously, StartLux-Decision-v1.1 ran on a general-purpose inference framework based on vLLM 0.31. However, mainstream frameworks such as vLLM are natively designed for token-by-token generation scenarios such as chat and text continuation, and are not suited to decision models, making it difficult to fully unleash the model's actual capabilities.

Prism is StartLux's solution: rewriting the inference engine according to the needs of decision models.

In actual tests, on a single H200 GPU, Prism processes a single short question 2.8 times faster than the original solution, and when asking three questions in succession about the same material, it can be up to 5.3 times faster.

What key changes did this inference engine, rebuilt specifically for decision models, actually make? What are the actual measured results for each performance metric? Why are more and more AI companies choosing to develop their own inference frameworks rather than directly using general community solutions? This article will address these questions.

1. StartLux uses four major innovations to solve the inference efficiency problem

To understand Prism's optimization approach, we first need to clarify the operating logic of decision models.

For example, a customer complains: "I was charged twice for my order, but the App still shows it as unpaid." Upon receiving this complaint, the system needs to simultaneously determine: which department the ticket should go to, whether a reply is required today, and how serious the problem is. The decision model needs to read the material and score the candidate answers for each question separately, returning the probability of each option.

In contrast, general-purpose inference frameworks such as vLLM and SGLang were originally designed to serve text-generation tasks. If they are used to run decision models, each question must be broken down into an independent request, separately completing network requests, tokenization, and scheduling, while repeatedly executing the computation of the huge output layer.

All of Prism's optimizations revolve around one core goal: to complete one decision while performing only the essential computation.

Optimization 1: Narrowing the scope of output computation

Taking the 27B-parameter StartLux‑Decision model as an example, its output layer contains 1.27 billion weights, and a general-purpose engine needs to fully score 248,000 candidate tokens, but decision tasks only use the 26 letters A‑Z as output options.

Prism compresses the scope of output-layer computation to 0.01% of the original, while the model still fully completes material understanding; it simply skips the computation of a large number of irrelevant candidate tokens at the stage of outputting judgment results.

Optimization 2: "Forest-style" computation

In business scenarios, the same material often corresponds to multiple judgment requirements. For example, a customer ticket may require simultaneously determining the team responsible for handling it, whether a same-day reply is needed, and the severity level of the fault. With traditional frameworks, each question answered requires reprocessing the ticket material in full.

Prism innovatively constructs batch tasks into a "forest-style" computation structure: general instructions serve as the tree roots, shared reference material serves as the tree trunk, and individual questions serve as branches.

The computation corresponding to the roots and trunk is executed only once, and different questions only process their own newly added parts. Multiple follow-up questions about the same material can likewise reuse existing computation results.

Optimization 3: Using integer arithmetic

When interacting with large language models, we often encounter a problem: for the same question, the large model gives different answers each time, sometimes even completely opposite ones.

This is fatal for decision models, because the probability values output by decision models directly guide business systems in executing subsequent actions.

Prism solves the problem of result drift through integer arithmetic: all correlation matrix operations use int8 precision, with accumulation performed at int32 precision. Because integer addition is unaffected by computation order, the results are not disturbed by batch size or scheduling order.

Optimization 4: Mixed scheduling

StartLux‑Decision-v1.1 supports AutoThink automatic thinking mode: when the model is sufficiently confident, it outputs the answer directly; when it is not, it first generates a reasoning process of up to 4096 tokens.

But this also brings a scheduling challenge: if short requests and long-thinking requests enter the same task queue, simple tasks will be blocked by long-running requests.

Prism designed a hybrid scheduling mechanism for this: the prompts of thinking tasks are merged into the direct-output batch for preprocessing, and the subsequent decoding steps are precompiled into CUDA graphs, supporting up to 64 thinking tasks and direct-output tasks sharing GPU resources in a time-sliced manner. Newly arriving simple requests wait at most 1–2 decoding steps before they can jump the queue and execute.

II. Inference up to 5.3x faster, average of only 41 ms across 231 questions

After these optimizations, how much does performance improve? StartLux conducted a series of comparative tests on this.

(Note: The following data come from StartLux's measured results under different model versions and test conditions. Among them, the 5.3x speedup test used the StartLux-Decision-v1.1-27B AutoThink model; the 231-question cross-engine performance comparison and throughput tests used the StartLux-Decision-v1-27B model. The specific test configurations and results have all been made public on GitHub.)

Test 1: Single-request latency

The test results show that in the single short-question scenario, Prism takes 19 ms, compared with 53.1 ms for the original solution, a 2.8x speedup. When processing three consecutive questions about the same material, Prism needs only 17.4 ms, compared with 92.9 ms for the original solution, a speedup of 5.3x.

If thinking mode is enabled on top of the three questions, Prism takes 785 ms, compared with 1296 ms for the original solution, a 1.7x speedup. For a single long question of about 8192 tokens, the performance gap between the two narrows to 1.2x.

In a more intuitive video understanding test, StartLux-Prime is 3.38x faster than vllm, and the output probabilities are essentially the same.

Test 2: Cross-engine latency

The research team selected a total of 231 questions from the JevBench test set and benchmarked Prism horizontally against mainstream open-source inference frameworks such as vLLM, SGLang, and llama.cpp.

The results show that Prism has an average latency of 41 ms per question, ranking first among all tested frameworks. The next fastest, the decider compiled version, is 66.8 ms, followed by SGLang at 75.6 ms, vLLM at 112.7 ms, and llama.cpp with the longest time, averaging 589 ms per question. Under the same conditions of 8-bit precision, Prism's latency is about 40% lower than the next-fastest framework.

Two test premises need to be explained here: the decider and JevK5 engines cannot directly run StartLux's native models, so the tests used structurally consistent substitute models and compared only speed, not accuracy; some other frameworks do not support 8-bit quantization and could only run at higher precision.

Test 3: Throughput

In the high-concurrency throughput test, with 64 concurrent streams saturating the GPU, Prism's advantage over competitors narrows to 17% to 22%. At this stage the bottleneck lies in hardware compute: both sides have already maxed out the H200's performance, and software optimization can hardly bring further significant gains.

But the most common decision-making business scenario is not extreme full-load operation, but small requests that occur at high frequency. Under this typical daily load, Prism can process 100 requests per second, while the best-performing competitor handles 38 per second, giving Prism a 2.6x lead.

In other words, Prism's performance advantage falls exactly in the business range that enterprises actually use at high frequency, which is also the core value of a dedicated inference engine.

Test 4: Flexible adaptation for multi-GPU deployment

At the multi-GPU deployment level, Prism supports both data parallelism and tensor parallelism, and the two can also be combined to suit different business needs.

Taking the 27B model as an example, four H200s in data parallelism mode each receive requests independently, handling a total of 121 requests per second, about 3.92x the single-GPU throughput. If the four GPUs use tensor parallelism to jointly compute the same question, single-question latency can drop from 41 ms to 29 ms.

A two-GPU hybrid deployment of the two groups can balance throughput and latency, achieving 86 requests per second and 36 ms per question. Regardless of the deployment method, the output results are bit-for-bit identical to running on a single GPU.

Test 5: Precision and deployment threshold

Beyond the speedup, the model's inference precision has not degraded.

In thinking-enabled tests covering seven evaluation sets and a total of 12,351 decision samples, the Prism int8 version achieved an average accuracy of 92.94%, compared with 92.83% for the original engine's FP8 version and 92.97% for the unquantized BF16 original version. The accuracy gap between the Prism quantized version and the full original version is only 0.03 percentage points, and it is even slightly better than the original engine's quantized version.

The deployment threshold is also lower than expected. The model has a compressed version of 16.55GB, which can run on a single consumer-grade RTX 5090 graphics card with 32GB of VRAM, with output quality comparable to the professional-grade H200 GPU. On this compressed version, Prism is 3.9 times faster than the mainstream open-source framework llama.cpp.

III. The inference system pulls ahead, and model companies begin to develop their own inference engines

Within just two days, DeepSeek and StartLux successively open-sourced their self-developed inference engines, reflecting a common shift in the large model industry: beyond competition in model capabilities, the importance of inference engineering is rising rapidly.

Whether it is a general-purpose large model or a decision model in a vertical domain, once it is embedded in core business processes, the latency of every call and the compute cost of every ten thousand requests will be directly reflected in business costs. The deployment cost, response speed, and operational stability of the same model have begun to become key factors determining its commercialization space.

General-purpose inference frameworks need to adapt to models with different structures and cover vastly different application scenarios, so they inevitably have to compromise on specialization. Only the model vendors themselves know best the interface characteristics and optimization potential of their own models.

StartLux is taking exactly this self-development path, while also carving out its own open style in the niche track of decision models.

Unlike many vendors that only open-source model weights or inference code, every product release from StartLux simultaneously outputs the underlying methodology and supporting toolchain, turning its own practical experience into public assets that the industry can reuse.

When it previously released the StartLux-Decision decision model, the team also simultaneously disclosed the RLCD² reinforcement learning training algorithm, systematically solving the training challenge of "when to start thinking and when to stop thinking" for decision models and filling a gap in decision model training methodology. The DecisionPivot evaluation benchmark launched at the same time also filled the industry's gap in evaluation standards for the detail-recognition capability of decision models, providing a unified reference for comparison across the entire industry.

The open-sourcing of the Prism inference engine this time continues this same approach. The team not only released the complete engine code, but also disclosed the full set of optimization logic for decision model inference, which amounts to fully opening the core design ideas of decision model inference engineering to the industry.

Open-source AI vendors such as DeepSeek and StartLux are turning inference engineering from an internal capability of a single vendor into reusable, verifiable public infrastructure for the industry.

For the decision model track, which is still in its early stages of development, this open model of "model + method + tools" directly lowers the trial-and-error cost for the entire industry, allowing more teams to avoid building the complete chain of training, evaluation, and deployment from scratch and to focus on upper-layer scenario innovation and performance optimization.

Conclusion: In the competition over decision models, the inference layer will be the deciding factor

To truly apply decision models in real business scenarios such as risk control, content moderation, and business ticket routing, high-performance models alone are far from enough.

The essence of Prism is to recalculate the entire set of inference computation logic specifically for decision models. When the output of large models shifts from long-form text to a set of probabilistic judgments, the various default assumptions of inference engines from the text-generation era all have room to be re-examined and re-optimized.

In the future, competition in the decision model track will also evolve in this direction: the gap in model architecture will continue to narrow, but barriers at the inference engineering level will become more pronounced, and the latter cannot be simply closed by stacking models with larger parameter counts.

This actually makes the core of competition in decision models and the entire model field more prominent, and also makes the outside world even more eager to see who can run farther and go deepest.