TensorZero 在 LLM 网关中引入多臂老虎机自适应实验

Summary

Experimentation (A/B testing) with production traffic is the most reliable way to identify the best prompts and models for your task, but traditional approaches have significant limitations: you must either fix the experiment length in advance (risking wasted data or inconclusive results) or repeatedly check for significance (inflating error rates through p-hacking); 11.5KTensorZero now provides adaptive experimentation directly in its open-source LLM gateway. This multi-armed bandit algorithm overcomes the p-hacking problem, running experiments precisely until there’s enough evidence to pick a winner while dynamically allocating LLM inference traffic for maximum efficiency; Across a diverse set of realistic and challenging environments, adaptive experimentation reduced the average time to correctly identify the best LLM variants (prompts, models, etc.) by 37% compared to simple A/B testing

Developing LLM applications involves numerous design decisions: which model to use, how to structure prompts, what temperature and sampling parameters to set, etc. While offline evaluations and synthetic benchmarks can guide these choices, the most reliable way to optimize them is through experimentation (A/B testing) with real production traffic and feedback.

Traditional experimentation involves allocating traffic evenly (or according to fixed proportions) across two or more variants of your application that you want to compare and tracking a metric or outcome of interest. After collecting data, you use a hypothesis test to decide whether you have enough evidence to label one variant the “winner,” meaning that it outperforms the other variants on average. Traditional experimentation has two fundamental limitations:

Fixed stopping rules: You must fix either the sample size or test duration in advance (which risks collecting too little or too much data), or repeatedly check for statistical significance (which inflates your error rates; this is the classic “p-hacking” problem); Suboptimal sample allocation: Traffic allocation does not take variants’ performance into account. This means you continue wasting resources on underperforming variants even after accumulating strong evidence against them. An efficient strategy would typically concentrate sampling on the most promising variants as confidence grows, enabling faster identification of the true winner

In this post, we’ll introduce a multi-armed bandit algorithm that enables adaptive experimentation with dynamic stopping, a more efficient approach that addresses both these limitations. We’ll explain how it works, show you how to implement it for your LLM applications using TensorZero, and demonstrate its advantages with real experimental data.

The mathematical foundation for adaptive experimentation comes from “multi-armed bandits,” a classic framework from reinforcement learning and sequential decision-making.

Imagine a gambler facing a row of slot machines, each with an unknown payout (“reward”) distribution. At each iteration, the gambler has to decide which arm to pull in order to obtain a payout and learn about that machine’s reward distribution.

Multi-armed bandit algorithms use the historical rewards to guide the decisions about which arm to pull in order to achieve a given problem objective.

In the context of LLM experimentation, an “arm” represents a particular LLM configuration: a choice of model, prompt, hyperparameters, inference strategy, and so on.

The two main problem settings are regret minimization (maximizing cumulative reward during the experiment) and best-arm identification (efficiently identifying the best arm with high confidence). We focus on best-arm identification, because practitioners developing LLM applications typically want to identify the best variant and deploy it exclusively going forward, rather than maximize performance during the experimentation phase. Additionally, since serving costs decrease with scale, it’s generally most efficient to allocate all compute to a single LLM.

Best-arm identification (BAI) is sometimes called the pure exploration setting, because the goal is to learn which arm is best without worrying about performance during the experiment itself. In contrast, regret minimization algorithms balance exploration (sampling arms to learn about their performance) and exploitation (sampling the apparent best arm to maximize immediate reward).

Best-arm identification comes in two main varieties1, 2:

Fixed-confidence BAI: Minimize the number of samples needed to identify the best arm with a specified confidence level (probability at least 1−δ1 - \delta1−δ for some δ∈(0,1)\delta \in (0, 1)δ∈(0,1)). TensorZero uses this approach; Fixed-budget BAI: Given a fixed number of samples, minimize the probability of misidentifying the best arm. This is useful when experimental resources are strictly limited

ε-Best-Arm Identification is a more general version of BAI where the algorithm can identify any arm that’s “ε\varepsilonε-close” to the best arm.3 This can significantly reduce the number of samples needed when you only care about finding “approximately optimal” arms rather than identifying the single best one. Setting ε=0\varepsilon = 0ε=0 (TensorZero’s default) means you want to find the exact best arm.

Why not use regret minimization algorithms?

Popular regret minimization algorithms like Thompson Sampling4 and Upper Confidence Bound (UCB)5 are not optimal for best-arm identification.6, 7, 8 They tend to concentrate samples on the apparent best arm to maximize cumulative reward, but don’t explore other arms frequently enough to confidently distinguish between close competitors.7 In fact, no algorithm can simultaneously perform optimally for both objectives.8, 9, 10

The key to efficient best-arm identification is knowing how to allocate samples across variants. For any given experiment, there exists a set of optimal sampling proportions: proportions over the arms that minimize the number of samples needed to confidently identify the best arm.1 These are almost never uniform. For example, arms with higher reward variances may require more samples to confidently estimate their mean rewards. Underperforming arms often require little data to be ruled out, meaning sampling can focus on distinguishing the top variants. Crucially, these probabilities depend on the unknown reward distributions, so they can’t be known in advance. (If we knew the reward distributions up front, there would be no need to conduct an experiment!) They can only be estimated as the experiment unfolds and data accumulates.

In traditional experimentation with fixed sampling probabilities (typically uniform), it’s extremely unlikely that you’d land on these optimal proportions by chance. Even if you got lucky and chose good probabilities up front, you’d still need to stop at precisely the right moment to get the efficiency benefits. But repeatedly checking for statistical significance to decide when to stop using a traditional hypothesis test introduces the p-hacking problem, inflating error rates.

In traditional experimentation, you must choose between two approaches, both of which have serious limitations:

Fix the sample size or test duration in advance: You decide ahead of time to collect exactly nnn samples (or run for exactly ttt days), then check for statistical significance only at the end. The choice of how long to run the experiment might be informed by a power analysis: a determination of what sample size would be required to detect a winner under certain assumptions about the unknown reward distributions. This approach maintains valid statistical guarantees, but you risk:

Under-sampling: If you choose nnn too small, you may not collect enough data to detect a winner, leaving you with an inconclusive result. You cannot simply continue sampling until significance is reached, as that would be p-hacking; Over-sampling: If you choose nnn too large, you waste resources collecting unnecessary data from underperforming variants after you already have sufficient evidence to identify the winner; Check for significance repeatedly during the experiment: You might be tempted to check your test statistic as data arrives and stop as soon as p 0 means the algorithm is content to find any variant whose mean performance is at most ε below the best arm’s mean (including the best arm itself); delta (δ): The maximum probability of incorrectly identifying the best variant (Type I error rate). For example, setting δ = 0.05 (the default) means there’s at most a 5% chance the algorithm declares the wrong variant as the winner

These parameters present a fundamental tradeoff: higher sensitivity and lower error rates require longer experiments.

Decreasing ε enables detection of better arms but requires more samples to distinguish variants that are close in performance; Decreasing δ (e.g. from 0.05 to 0.01) provides stronger statistical guarantees but requires more samples since more evidence is needed to confidently identify the best variant

In practice, you should choose these parameters based on your application’s requirements. If the cost of misidentification is high (e.g. deploying a variant that harms user experience), use a smaller δ. If you only care about finding variants that are substantially better than the status quo, you can use a larger ε to speed up convergence. Note that it’s not possible to set δ = 0, because you can never conclude with 100% confidence that the results you observe are due to true differences rather than random chance—achieving zero error probability would require infinite samples.

The algorithm can handle any number of variants. However, if many variants have similar performance (close to the best one), the algorithm will need more samples to confidently distinguish between them.

Real Example: Optimizing the Prompt and Model for a Data Extraction Task

Now let’s look at real LLM data. We ran an adaptive experiment to optimize variants for a data extraction task: named entity recognition (NER). The goal was to extract person, location, and organization entities from text. We tested four variants, consisting of two models (Claude Haiku 4.5 and Claude Sonnet 4.5) crossed with two prompts: a simple prompt with minimal instructions and a more detailed prompt.

The metric we optimized for was exact_match: whether the model’s output exactly matched the ground truth entities (a boolean metric). The algorithm was configured with the default values of delta = 0.05 (5% error rate) and epsilon = 0.0 (strict best-arm identification). We limited the experiment to 2,000 inferences.

Below we see the state of the experiment approximately halfway through and at the end. (These charts are very similar to the ones you’d find in the open-source TensorZero UI!) You can also find the code to reproduce this experiment on GitHub.

Halfway Through the Experiment (~1,000 samples)

At this point, the algorithm is still exploring all four variants, with sampling probabilities concentrated most heavily on the two best-performing variants, claude-sonnet-4-5-detailed-prompt (39.7%) and claude-sonnet-4-5-simple-prompt (38.4%).

Note that the variant with the highest mean doesn’t always get the highest sampling probability (though it does here). That’s because the optimal proportions depend not just on the means, but also on the variances.

Variant Weights

This chart displays the current sampling probabilities for each variant.

Full Experiment (2,000 samples)

By the end of the experiment, the algorithm has identified a clear winner (claude-sonnet-4-5-detailed-prompt) and allocated 100% of subsequent traffic to it. Looking at the Feedback Count tab, we see that this occurred some time between minute 3 and minute 4, or in other words sometime between inference 959 and inference 1,255, because the counts for all the other variants stopped changing.

Variant Weights

This chart displays the current sampling probabilities for each variant.

How TensorZero’s Algorithm Works

The Track-and-Stop strategy that we implement was originally developed in the academic literature under idealized assumptions: immediate feedback after each inference and deterministic arm selection based on perfect knowledge of all accumulated data.1 Production LLM applications present very different challenges that require careful engineering to preserve the algorithm’s theoretical optimality while making it practical.

Production Challenges

Asynchronous feedback: In real LLM applications, feedback rarely arrives immediately. Users might provide ratings minutes, hours, or days after an inference. Automated feedback (like task completion metrics) may need to be computed asynchronously in background jobs. The algorithm must work with whatever feedback has arrived so far, even as new inferences continue; Parallel requests: Production systems handle many concurrent inference requests from different users or application instances. You can’t deterministically sequence arm selections when multiple requests arrive simultaneously — attempting to do so would create a severe bottleneck; Computational overhead: Computing optimal sampling probabilities requires retrieving and aggregating all accumulated feedback from a database and then using that data to solve a numerical optimization problem. Doing this after every single inference would be prohibitively expensive and slow down your application; Variable throughput: The rate of inferences can vary dramatically — you might process 100 requests one day and 10,000 the next, or experience daily cycles with peak hours. The algorithm needs to adapt gracefully to these variations without making assumptions about traffic patterns

TensorZero’s Solution

Our implementation, written in Rust for performance, addresses these challenges while preserving the optimality of the Track-and-Stop strategy:

Probabilistic sampling: Instead of deterministic arm selection, we maintain sampling probabilities that estimate the optimal proportions. Each inference randomly samples from these probabilities, allowing parallel requests without coordination overhead; Periodic updates: A background task periodically recomputes sampling probabilities at a configurable frequency (typically every few minutes). It computes these probabilities by solving a second-order cone program (SOCP) using the Clarabel optimizer.18 This balances responsiveness to new data with computational efficiency; Learning phase: Before random sampling begins, variants are sampled round-robin until reaching a minimum sample threshold (10 by default). This ensures stable statistics before entering the main optimization, preventing premature concentration on poorly-estimated variants; Regularization: The optimization includes a decaying regularization term that penalizes deviation from uniform sampling. This smooths the transition from initial exploration to optimal allocation, improving robustness when sample sizes are still small

The stopping rule continuously evaluates the generalized likelihood ratio test (GLRT) to detect when sufficient evidence has accumulated, automatically directing all traffic to the winner.

Fun fact: The theoretical analysis that underlies the Track-and-Stop strategy was published in 2016,1 so this approach would not have been possible before then!

Separation of Concerns

TensorZero handles experimentation directly in your LLM gateway. You don’t have to integrate experimentation logic into your application code at every LLM call site like you would with standalone experimentation tools like Statsig.

Concretely, you configure your variants and metrics once in TensorZero’s configuration file, and the gateway automatically manages variant selection and feedback collection. Your application code remains unchanged whether you’re running experiments or not.

Conclusion and Future Directions

TensorZero’s multi-armed bandit algorithm enables you to automatically identify the best LLM configuration with minimal wasted inferences. The algorithm adapts sampling in real time, focuses resources on distinguishing top performers, and stops as soon as there’s sufficient evidence—all while maintaining rigorous statistical guarantees against mislabeling the winner.

Real-world LLM applications present additional complexities beyond a single reward metric: you often care about multiple objectives simultaneously (accuracy, cost, latency), face budget constraints, or need to balance short-term performance with long-term learning. We plan to extend our experimentation framework to handle these multi-objective settings, making automated optimization practical for increasingly complex production scenarios.

More broadly, efficient experimentation is just one piece of the LLM optimization puzzle. Now that we can reliably determine which configurations perform best in production, the next frontier is automatically generating promising new configurations to test—and intelligently deciding which candidates are worth evaluating with real traffic versus filtering out through offline evaluation. We’re building toward a future where LLM applications continuously improve themselves through this closed loop of generation, evaluation, and experimentation.

If you’re interested in trying out TensorZero’s experimentation features, see our docs: How to run adaptive A/B tests

Acknowledgements: Thanks to (in alphabetical order) Gabriel Bianconi, Aaron Hill, Andrew Jesson, Shuyang Li, and Viraj Mehta for helpful feedback.

Garivier, A., & Kaufmann, E. (2016). Optimal Best Arm Identification with Fixed Confidence. Proceedings of Machine Learning Research, 49, 998-1027; Kaufmann, E., Cappé, O., & Garivier, A. (2016). On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models. Journal of Machine Learning Research, 17(1), 1-42; Jourdan, M., Degenne, R., & Kaufmann, E. (2023). An ε-Best-Arm Identification Algorithm for Fixed-Confidence and Beyond. Advances in Neural Information Processing Systems (NeurIPS), 36; Agrawal, S., & Goyal, N. (2013). Further Optimal Regret Bounds for Thompson Sampling. Proceedings of Machine Learning Research, 31, 99-107; Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning, 47, 235-256; Bubeck, S., Munos, R., & Stoltz, G. (2009). Pure Exploration in Multi-armed Bandits Problems. In Algorithmic Learning Theory (ALT 2009), Lecture Notes in Computer Science, vol 5809. Springer; Russo, D. (2016). Simple Bayesian Algorithms for Best Arm Identification. Proceedings of Machine Learning Research, 49, 1417-1418. (Extended version published in Operations Research, 68(6), 1625-1647, 2020.); Zhong, Z., Cheung, W. C., & Tan, V. (2023). Achieving the Pareto Frontier of Regret Minimization and Best Arm Identification in Multi-Armed Bandits. Transactions on Machine Learning Research; Yang, J., Tan, V. Y.F., & Jin, T. (2024). Best Arm Identification with Minimal Regret. arXiv preprint arXiv:2409.18909; Zhang, Q., & Ying, L. (2023). Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithms. Advances in Neural Information Processing Systems (NeurIPS), 36; Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B Tests: Why it matters, and what to do about it. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1517-1525; Howard, S. R., Ramdas, A., McAuliffe, J., & Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2), 1055-1080; Ramdas, A., Grünwald, P., Vovk, V., & Shafer, G. (2023). Game-Theoretic Statistics and Safe Anytime-Valid Inference. Statistical Science, 38(4), 576-601; Kaufmann, E., & Koolen, W. M. (2021). Mixture Martingales Revisited with Applications to Sequential Tests and Confidence Intervals. Journal of Machine Learning Research, 22(246), 1-44; Ramdas, A., Ruf, J., Larsson, M., & Koolen, W. (2020). Admissible anytime-valid sequential inference must rely on nonnegative martingales. arXiv preprint arXiv:2009.03167; Kaufmann, E. (2020). Contributions to the Optimal Solution of Several Bandit Problems. PhD dissertation, Université de Lille; Waudby-Smith, I., & Ramdas, A. (2024). Time-uniform central limit theory and asymptotic confidence sequences. The Annals of Statistics, 52(6), 2613-2640; Goulart, P. J., & Chen, Y. (2024). Clarabel: An interior-point solver for conic programs with quadratic objectives. Version 0.11.1