Laya 零样本决策与校准开发者指南：在 CLINC150 上实测路由与概率

In this tutorial, we work with Laya, the open-source decision engine from Convai Innovations that became one of the most-starred machine-learning repositories of September 2026. Laya is a non-autoregressive System 1 model: instead of generating text, a 421-million-parameter encoder reads a piece of text and a set of typed questions, a choice between labels, a score on a scale, or a yes/no, and returns a probability for every option in a single forward pass with zero output tokens. Its pitch is speed and calibrated probabilities, the open answer to TypeSafe’s Jev. Rather than repeat the README’s examples, we put those promises to work on real labelled data with known answers, the banking domain of the CLINC150 intent dataset, and measure what a production router actually gets: zero-shot accuracy against a trained classifier, how much the wording and order of the options matter, how honest the shipped probabilities are, what fitting a temperature on validation data fixes and what it quietly breaks, an abstention gate fitted to an error budget, out-of-scope traffic, a yes/no question that temperature cannot repair, and typed outputs from a pydantic schema.

We install the released package, laya 0.3.27, and load the English checkpoint. Two choices here keep the run reproducible. By default laya.load follows the Hugging Face main branch, so we pin the revision the library’s own authors reviewed, which it exposes as laya.PINNED_REVISIONS. And on CUDA Laya autocasts to half precision, so we switch that off to keep every device in fp32 and let a GPU run reproduce the CPU numbers shown here. Printing the checkpoint’s shipped temperatures turns up the first finding before any prediction: the entry for choice questions with eleven or more options is 0.10, outside the valid range, so the loader clamps it to 0.5 and warns. A temperature below one sharpens probabilities, so every answer to a question with that many options will look twice as certain as the raw model is.

One call to predict answers three typed questions about a support ticket in a single forward pass: the department as a choice, the urgency as a score from 0 to 2, and the churn risk as a yes/no. The result carries a probability for every option and two confidence fields that are easy to confuse. answer_confidence is the probability of the reported answer, and it is the quantity that calibration, the abstention gate and every metric later in this tutorial use. confidence is one minus the normalized entropy, whose scale depends on how many options a question has. The usage block shows zero output tokens, because Laya scores the options it is given and never generates text.

Before building on Laya, we measure the cost of a forward pass on a single message. Each question becomes its own row, paired with the message, so sixteen yes/no questions take about eight times as long as one. All the options of a choice question share one row and its head budget, so a forty-option choice costs barely twice a three-option one and a quarter of what sixteen yes/no questions do on our CPU. That gives a design rule that shapes everything after it: ask one choice question with many options rather than many yes/no questions.

For real labeled data, we use CLINC150, a public intent-classification benchmark of 150 intents across ten domains plus a set of out-of-scope queries, read straight from the Hugging Face Hub as a parquet file. We take its banking domain, fifteen intents with 100 training, 20 validation, and 30 test queries each, and ask Laya to route the 450 test queries zero-shot, giving it each intent’s name and a one-line description of the kind a developer would write. It reaches 0.804 accuracy with no labeled examples. For scale, a TF-IDF and logistic-regression classifier reaches 0.651 with three labeled queries per intent, 0.848 with ten, and 0.904 with thirty.

Next we change only the wording of the options. Giving Laya the fifteen bare intent names, without our descriptions, lifts accuracy from 0.804 to 0.878 and halves the time, because the options take less than half the tokens. The descriptions blurred intents the names keep apart: account_blocked was routed to freeze_account ten times, and interest rate questions to balance. Reversing the order of the bare names changes 4.2 percent of individual answers even though overall accuracy barely moves, a sign of a position prior, so the option order you deploy should be the order you tested. Only labeled data could tell us either thing; from here on we route on the bare names.

We then ask how honest the probabilities are. A fifteen-option question falls into the checkpoint’s choice:11+ temperature bucket, the one clamped to 0.5. The reliability table on the 450 test queries shows the result: 92 percent of answers claim a confidence of at least 0.9, but 91.1 percent of those are right, and the mean confidence of 0.974 sits well above the accuracy of 0.878, an expected calibration error of 0.102. Laya’s training objective uses proper scoring rules, which is what the model card means by calibrated. Still, calibration is a property of a question on a distribution, and you have to measure it on your own labels.

Laya’s calibration module turns labeled examples into records of raw logits and targets, and fits a temperature to them. We build 300 records from the validation split and call agent.fit_temperatures, which fits a choice temperature of 1.258 and installs it, then prints the full temperature table next to what shipped. The fit replaced the entire map: every per-option-count entry is gone, because a bucket needs 2,000 records to keep its own temperature, and the score and yes/no temperatures were reset to 1.0 because there were no records of those types. One fit on a choice question silently changed how every yes/no question in the agent is calibrated. So we restore the shipped values and install only the bucket we measured. On the test set, calibration error falls from 0.102 to 0.059 with accuracy unchanged, since temperature never changes which option wins, and save_calibration writes the result to a JSON file that laya.load can read back.

laya.fit_abstention_thresholds takes the same validation records and, for each option-count bucket, returns the loosest confidence cut that keeps the validation error within a target. For a 5 percent target, it picks 0.602, which keeps 95.7 percent of the validation queries at 4.5 percent error, and Laya applies the same cut itself when it is passed to predict_batch as min_confidence, marking 35 of the 450 test answers as abstained. On the test set, though, that gate keeps 92.2 percent of the queries at 9.2 percent error, nearly twice the budget, and the 2 percent target realizes 5.3 percent. The data explains it: Laya is right on 92.7 percent of the validation queries but only 87.8 percent of the test queries, so an error budget fitted on one sample holds only for traffic that looks like it. The ranking itself is sound, with the most confident half of the test answers 97.8 percent correct, but an error target needs margin and periodic re-fitting on real traffic. The thresholds stay per option-count bucket because one number doesn’t transfer between a two-way and a fifteen-way question.

Production traffic includes requests the router was never built for, so we add 150 out-of-scope CLINC queries and 150 queries from other CLINC domains. The calibrated confidence separates them sharply: banking queries average 0.912, the others about 0.25, and the 5 percent gate stops 89.3 percent of other-domain queries and 93.3 percent of out-of-scope ones while abstaining on 7.8 percent of banking queries. The alternative needs no labeled data: a sixteenth option, not a banking request, catches 80.0 and 90.0 percent of them, at the cost of routing 2.0 percent of banking queries away and lowering banking accuracy from 0.878 to 0.864, because the new option changes what every intent is scored against.

A dedicated yes/no question looks like the natural in-scope check, so we ask whether each message is about the user’s bank account, bills, or payments. It ranks well, with an AUROC of 0.945, but it is biased towards no: banking queries average a probability of only 0.361, so at the default 0.5 cut it recognizes just 28.9 percent of them. Fitting a temperature on 550 validation answers runs to the ceiling of 5.0 and changes nothing at that cut, because dividing two logits by any temperature never changes which is larger. Temperature scaling repairs a scale, not an offset. What works is treating the probability as a score and choosing its cut on labeled data: a cut of 0.09 picked on the validation split gives 92.0 percent recall and 83.0 percent specificity on the test set.

Finally, we wire this into application code through a pydantic schema. laya.decide_batch turns a Literal field into a choice question over the bare names, which step 5 showed is the better wording here, and a bool into a yes/no question, then projects the answers back into a validated model instance. Passing the gate from step 8 as min_confidence turns an uncertain intent into None, which the Optional field accepts, so None becomes an explicit ask-a-human signal; the two out-of-scope queries come back as intent=None. The boolean, however, is cut at a fixed 0.5 inside decide, so a fraud report is marked in_scope=False at a probability of 0.29. Reading the probability from the result details and applying the cut from step 10 gives the right answer.

The summary prints the one-line result of each step and the habits worth carrying over: pin the checkpoint and read its shipped temperatures, test criteria wording and option order on labeled data, fit temperatures on held-out data and install only the bucket you measured, gate per option-count bucket with margin, and choose a yes/no cut on labeled data. It closes with where to go next: fine-tuning, which lives in laya.train on the repository’s main branch but is not yet in the 0.3.27 wheel, the multilingual checkpoint behind laya.Router, and serving.

In conclusion, Laya delivers most of what it promises: one forward pass answers several typed questions with no generated tokens, the bare-name router reached 0.878 on fifteen real banking intents with no training data, which a TF-IDF classifier needs between ten and thirty labeled examples per intent to match, and its calibrated confidence separated in-scope from out-of-scope traffic well enough to stop more than nine in ten out-of-scope queries. What the probabilities are worth, though, depends on work the library leaves to you, and several of its defaults point the wrong way. The shipped temperature for questions with eleven or more options sharpens rather than softens, the one-line calibration call wipes temperatures for question types it never saw, an error budget fitted on validation data held only for traffic that looked like it, and a yes/no question can rank well yet sit on the wrong side of 0.5, which no temperature can fix and which the schema projection then hard-codes. Each of these has a few-line fix shown above, and each would otherwise pass silently. The practical lesson is the same one this library’s own model card states and its defaults make easy to forget: a calibrated decision model is one you have calibrated, on your own labels, for your own questions.

[Sponsored] The web is the one API most agents are missing. Databases, calendars and repos have APIs. The open web mostly doesn’t. The TinyFish MCP server gives any MCP client four tools: TinySearch, TinyFetch (full pages as markdown, JavaScript included), TinyBrowser for logins and forms, and TinyAgent for multi-step jobs. Search and Fetch are free.

Sana Hassan

Sana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.