AI Trends

Google Releases Gemini Decision 1, Leading Jev on Five of Seven Shared Benchmarks

Sina TechnologyUpdated Nov 19 17:56

AI summary

Google has released Gemini Decision 1, a multimodal decision model that can directly return probabilities and confidence scores for candidate options. A single request can process up to 1,000 independent judgments, with support for text, images, audio, video, and PDFs. It leads on five of the seven shared benchmarks reported by both companies, while Jev leads on When2Call and Mind2Web. In Artificial Analysis's Decision Index v2.1, Gemini Decision 1 scores 65.98, slightly below Jev's 66.23, and costs $0.10 per million input tokens, higher than Jev's $0.042.

Why read this

SelectedAI score 82

The article presents official shared benchmarks alongside the third-party Decision Index and includes pricing and latency differences, making it easier to compare the trade-offs between the two decision models.

Topics

Tags

Body · AI translation

New public beta promotional copy New public beta promotional copyend Add app-exclusive traffic redirection

Jev’s $7.5 billion valuation was once questioned as a bubble. Now, with a multimodal decision model, Google is telling the market: this is not a bubble—it is a category.

Google has just released Gemini Decision 1, formally challenging TypeSafe Jev, which ignited the decision-model category two months ago.

Like Jev, this model generates no text. It only calculates probabilities and confidence scores for candidate options that programs can use directly to drive Agent execution. It does not need to generate answers Token by Token, eliminating output Token costs as well.

The bigger breakthrough lies in parallel computation. Gemini Decision 1 can process up to 1,000 independent judgments within the same context and supports mixed inputs of text, images, audio, video, and PDFs. For Agents that need to repeatedly inspect documents, select tools, and assess task status, many judgments previously scattered across separate model calls can now be completed in a single request.

In terms of data, Gemini Decision 1 wins five of the seven shared benchmarks reported by both sides, while Jev retains its lead on When2Call tool calling and Mind2Web multimodal tasks (82.0% vs 79.9%, and 86.8% vs 86.2%).

However, Gemini charges $0.10 per million input Tokens, about 2.38 times Jev’s price. Performance has improved, but the price is also higher. Google clearly intends to open up the market with stronger multimodal and batch decision-making capabilities.

Within weeks of its September launch, Jev secured a $7.5 billion valuation, with a16z leading an $870 million investment, and about one-third of the Fortune 500 were listed as customers. But doubts never disappeared: the vendor’s self-reported claims that it was “193 times faster and 445 times cheaper” lacked third-party replication, the controversy over its originality remained unresolved, and independent evaluations pointed to the same conclusion: the speed and low cost were real, but quality was another matter. Leiphone (official WeChat account: Leiphone)

Google’s entry at this point proves at least one thing: decision models are not a farce, but a category worth betting on for tech giants.

01

How the Model Makes Decisions Without Generating Text

When an ordinary LLM handles a task such as “choose one of four tools,” it still uses autoregressive decoding: it generates a tool name or JSON Token by Token, which the program then parses and executes. The model is performing open-ended text generation, while the business actually needs a probabilistic judgment among a limited set of candidates.

Gemini Decision 1 changes the output objective to candidate scoring. Developers define the question, candidate set, and evaluation criteria in advance. After reading the input, the model directly returns a probability distribution over the candidates. The entire process is completed through a single forward pass, with no need to generate an answer Token by Token and no output Token costs.

This is clearly different from ordinary Structured Output. Structured Output still has the model generate text, merely using a Schema to constrain the fields and format of the JSON. Gemini Decision 1 instead performs scoring directly within a limited candidate space, and the returned result is itself data that a program can read.

There is an easily overlooked detail here: probabilities have different meanings for different judgment types. Noul handles binary judgments and returns the probability that a given condition is true, ranging from 0.01 to 0.99. If the question itself is ambiguous, developers can also define separately what counts as Yes and what counts as No, reducing bias caused by unclear evaluation criteria.

Choice handles multiclass problems, supporting 2–255 candidates. It returns the winning option and a normalized probability for each candidate, with all candidate probabilities summing to 1. It is suitable for tool routing, model selection, and task classification.

However, this probability allocation always occurs within the given candidate set. Even if none of the options match the input, the model will still select the relatively closest result. Developers therefore need to provide an Other option or add separate range checks.

Score has a more interesting design. Rather than simply returning a level, it first assigns a probability to each level in an ordered scoring rubric and then calculates a weighted score. For example, if an issue is divided into Level 0, Level 1, and Level 2, the model might assign probabilities of 0.1, 0.7, and 0.2, producing a final score between 1 and 2.

This effectively extends discrete classification into continuous scoring, making it easier for programs to rank results and filter them by threshold. However, the weighted result compresses information from the probability distribution: two inputs may receive the same score even though their underlying level probabilities are entirely different. Therefore, for tasks that need to distinguish among risk levels, the program should also read the probability of each level rather than using only the final score.

These three types make up Gemini Decision 1’s output system. The model is no longer responsible for generating a human-readable answer. Developers need to design clear questions and candidate criteria, while the program is responsible for converting the probabilistic results into specific execution conditions. Leiphone

02

How 1,000 Questions Are Computed Independently

A Gemini Decision 1 request consists of one Shared Input paired with a set of Questions. The former carries text, images, audio, video, and PDFs, while the latter defines the decision tasks to be completed. Each question can use a different decision type and scoring criteria, and a single request supports up to 1,000 questions.

Take equipment inspection as an example. The system receives a video of the equipment in operation, a maintenance record, and an on-site audio recording, and needs to determine whether the equipment is abnormal, the type of anomaly, its severity, and whether a manual inspection is required.

Developers can place all three materials in the same input and then define these questions separately. Anomaly detection uses Noul, category determination uses Choice, and severity uses Score. The model reads the same materials and completes each assessment separately.

This design has two technical details. First, each question has an independent evaluation path. Google’s documentation explicitly states that each question can read only the shared input and cannot read the answers to other questions.

Adding or removing a question also does not change the answers to the other questions. This means that parallel scoring does not depend on a reasoning chain from preceding questions, allowing the model to complete multiple independent judgments in the same forward pass.

Second, the shared input is billed only once. The additional cost generated by each question comes from its own instructions and criteria Tokens, without requiring the entire set of materials to be processed repeatedly. For batch review of long documents, audio and video inspection, and multidimensional content evaluation, this pricing structure affects the overall cost of API calls.

However, support for 1,000 questions does not mean that an arbitrarily complex workflow can be packed into a single request. If question B must generate new candidate options based on the specific result of question A, the two tasks have a sequential dependency. Developers need to read the result of A first and then submit B.

If B’s candidate options have already been determined and B’s answer merely needs to be ignored when A does not meet a condition, the two judgments can be submitted together, with the program deciding whether to use the result.

Input length must also be considered. Gemini Decision 1 has a context limit of 256K Tokens, and this budget is shared by the common materials and all questions; each question does not have its own 256K context. As the number of questions increases, their instructions and criteria also consume context space.

According to latency data published by Google, under test conditions where the context did not exceed 16K Tokens, increasing the number of questions from 1 to 100 left model compute latency essentially flat, with median compute latency of 50–100 ms and end-to-end latency of 150–250 ms. It is important to distinguish between the API’s support for 1,000 questions and the publicly reported near-flat latency tests, which covered 1–100 questions.

Multimodal input also involves an engineering tradeoff. Images and videos can be assigned resolution levels such as low, medium, high, and ultra_high; as the amount of visual detail increases, input costs change accordingly.

Audio and PDFs, meanwhile, enter their corresponding content-processing paths directly, without requiring developers to set up separate recognition services. Google has also explicitly included the costs associated with OCR and audio transcription in the model’s input pricing, with no separate output Token charge.

There is not enough public information to reconstruct the model’s internal parallel computation graph. What can be confirmed is that the questions share the same input and are scored independently in a single forward pass. Google has not yet provided a complete technical explanation of whether separate output heads are used internally or how computational resources are allocated among the questions.

03

PK Jev: Behind the “Lead in Five Benchmarks”

Gemini Decision 1 provides a confidence value for each answer, ranging from 0 to 1. It represents the model’s estimate of the likelihood that the current result is correct. For Noul, confidence is the higher probability of the Yes and No outcomes; for Choice, it corresponds to the probability of the winning option; and for Score, it corresponds to the level with the highest probability. This design allows programs to set different execution thresholds based on the reliability of a judgment.

This involves probability calibration. ECE (Expected Calibration Error) measures the deviation between the probabilities produced by a model and the actual proportion of correct results. Google reports an ECE of 0.0250 for Gemini Decision 1, compared with 0.0494 for Jev 1.14.

In terms of real-world product deployment, Google is already using the Decisions API for model routing in Gemini Notebook, selecting model tiers and reasoning levels based on request complexity. This has reportedly reduced latency for routine requests by 40%.

Google Search uses it for security checks on MCP tools, while Stitch uses it to accelerate action selection during the interaction-design process. Compared with TypeSafe Jev, the two models follow similar technical directions, but the publicly available information reveals different areas of emphasis.

TypeSafe has disclosed that Jev uses a specialized model architecture, Parallel Sampler, and an RLCD (Reinforcement Learning for Calibrated Decisions) training method. Jev 1.14 added image input, the Box decision type, and long-context capabilities, although these features were still placed under experimental configurations at launch.

Gemini Decision 1, by contrast, is built on Gemini’s native multimodal architecture, incorporating audio, video, and PDFs into its input range while offering a 256K context window and support for 1,000-question requests. Google has not disclosed a training approach comparable to TypeSafe’s, so the two models’ internal training paths cannot yet be directly compared.

In terms of performance, Gemini Decision 1 surpassed Jev 1.14 on five of the seven shared benchmarks published by Google, while Jev retained the lead on When2Call tool calling and multimodal Mind2Web. When2Call is a core scenario for Agent tool selection, and Jev’s lead on this metric shows that Google’s advantage does not extend across the board.

In addition, on the Decision Index v2.1 leaderboard from independent third-party evaluator Artificial Analysis, Gemini Decision 1 has an overall score of 65.98, slightly below Jev’s 66.23 and well behind the leading Drex Max v1.1 at 72.62. Its single-task latency is 211 ms, higher than Jev’s 92 ms. In terms of price, Gemini charges $0.10 per million input Tokens, compared with $0.042 for Jev.

In other words, Google charges a premium for multimodal capabilities and batch processing, but on a standardized third-party task suite, “more powerful” has not yet translated into “better.” Its lead in calibration error is real, but superior calibration does not equal superior overall decision quality.

04

The Decision Layer for Agents Is Becoming a Battleground of Its Own

Gemini Decision 1 formally turns a class of tasks that was previously scattered throughout Agent workflows into a standalone model-call capability. This involves more than scoring speed; it also encompasses the coordination of candidate-space definition, shared-input reuse, question isolation, probability calibration, and programmatic control.

General-purpose LLMs remain responsible for complex reasoning, planning, and content generation, while Decision Models handle judgment tasks whose criteria can be defined in advance. Developers need to arrange the order of calls according to task dependencies and then use probability performance to determine the scope of automated execution.

Jev ignited this category in September, and Google raised the stakes in November with multimodal and batch-processing capabilities. Whether this approach can establish a long-term advantage will depend on the models’ stable performance with complex inputs, shifts in task distributions, and real-world business environments. With Google’s entry, specialized decision models are beginning to compete in products at a larger scale, while Agent engineering will shift from relying solely on generative models toward a more refined division of labor among models.

This article is an original work by Leiphone. Reproduction without authorization is prohibited. See the reproduction notice for details.

]article_adlist Statement: Exclusive Sina.com article; reproduction without authorization is prohibited. tech_web_article_v2015_block_bottom Non-targeted 300*250 button 17/09 wenjing begin

Non-targeted 300*250 button end

Source

Sina Technology