TensorZero 研究：程序化数据筛选蒸馏让小模型推理成本降低 5-30 倍

Summary

Fine-tuning small models on programmatically curated high-quality outputs from a large model can beat large-model performance while reducing inference costs by up to 30x and inference time by up to 4x.

We benchmark closed-source (OpenAI, Google) and open-source (Qwen) models on data extraction (CoNLL++ NER), multi-turn maze navigation (BabyAI), agentic RAG (Multi-Hop), and agentic tool use (τ-bench); Learn how to reproduce this workflow for your own use case using 11.5KTensorZero and other open-source LLMOps tools, even without a GPU, along with best practices for production deployments

The Performance-Cost Dilemma

Every LLM application builder faces a fundamental tradeoff: large models deliver superior performance at higher costs, while small models are affordable but often underperform. This choice becomes critical when scaling to production workloads where inference costs can quickly explode.

Consider a customer service agent handling thousands of conversations daily. Using GPT-4.1 or Claude Sonnet 4 might provide excellent responses, but at $2-$15 per 1M tokens, costs accumulate rapidly. Switch to a smaller model and costs plummet, but so might customer satisfaction.

What if you could have both?

Our research demonstrates a way to escape this tradeoff. As illustrated below, we find that fine-tuning small models on programmatically curated data can achieve up to 30x cost reduction while improving or maintaining competitive performance across diverse applications. Additionally, these smaller models deliver up to 4x faster response times, which can significantly improve user experience in certain scenarios.

Curated behavior cloning fine-tunes small models using high-quality responses from large models. Beyond cost savings, our experiments reveal that this approach yields small models that can:

Match or exceed large model performance on specific tasks; Maintain stable performance on related out-of-distribution tasks; Deliver significantly faster response times, improving user experience

Unlike inference-time optimization techniques that require costly inference-time compute, fine-tuning shifts the computational burden to a one-time training phase, making every subsequent inference dramatically cheaper.

The Approach

We systematically evaluate fine-tuning across four challenging task categories that represent real-world LLM applications:

For each task type, we:

Collect high-quality conversation data from GPT-4.1 (large model); Fine-tune small models on the successful conversations (= episodes, trajectories, traces):Gemini 2.0 Flash and Flash Lite; GPT-4.1 mini and nano; GPT-4o mini; Qwen3-8B; Evaluate performance, reliability, and cost on held-out test sets:Success rate: pass^1 (chance that a conversation succeeds); Reliability: pass^5 (chance that 5 repeated conversations all succeed); Cost per conversation: total cost of inference ÷ number of conversations

We use multiple fine-tuning providers (OpenAI, Google Vertex AI, Unsloth) and leverage 11.5KTensorZero to seamlessly switch between providers and environments without infrastructure complexity.

Results

Our evaluation reveals significant cost efficiency improvements when using fine-tuned models compared to GPT-4.1. When measuring cost per success (cost per task ÷ success rate), all fine-tuned models demonstrate substantial reductions in the resources required to achieve successful task completion. The following table provides a detailed breakdown of cost-efficiency improvements across each individual task environment.

Table: Cost Savings Factor Relative to GPT-4.1

Values show the reduction factor in cost per success (cost per task ÷ success rate) compared to GPT-4.1. Higher values indicate greater cost efficiency improvements.

Across all evaluation environments — treating τ-bench (airline and retail) as a single environment group — Gemini 2.0 Flash Lite (fine-tuned) achieves the highest cost efficiency with 24.1x lower cost per success than GPT-4o, followed by GPT-4.1 nano (fine-tuned) at 17.9x and GPT-4o mini (fine-tuned) at 13.7x.

1. Data Extraction (CoNLL++ NER)

Task: Extract named entities from the CoNLL++ dataset, identifying persons, locations, organizations, and miscellaneous entities.

Success: Correctly identify and classify all named entities in a given text.

Key Finding:

Data extraction represents an ideal task for fine-tuning, with all model families showing substantial improvements over their zero-shot baselines. Fine-tuned small models can match or exceed the performance of larger zero-shot models at a fraction of the cost, making this approach particularly attractive for production data extraction systems.

Dataset: CoNLL++ dataset for named entity recognition, containing annotated text with four entity types: Person (PER), Location (LOC), Organization (ORG), and Miscellaneous (MISC).

Fine-tuning data: Successful demonstrations from GPT-4o, formatted for entity extraction.

Evaluation data: Test set with diverse text samples requiring entity identification and classification.

2. Multi-Turn Navigation (BabyAI GoTo)

Task: Navigate to target locations in a text-based game environment.

Success: Agent reaches the target within 32 steps.

Key Finding: Fine-tuning eliminates the performance-cost tradeoff for this navigation task by consistently improving both the success rate (pass^1) and reliability (pass^5) across all model families except Qwen3-8B. All fine-tuned models outperform their zero-shot counterparts at significantly lower costs.

BabyAI Evaluation Details Fine-tuning data:

446 successful GPT-4.1 conversations as demonstrations (356 training, 90 validation).

Evaluation data:

100 test tasks × 5 repetitions.

3. Agentic RAG (Multi-Hop)

Task: Retrieve Wikipedia articles to verify a claim using multi-policy (informally, multi-agent) search.

Success: Agent returns the top-5 ground-truth Wikipedia article titles.

Key Finding:

Fine-tuned small models can surpass their large model teachers without policy-level feedback. Fine-tuned Gemini 2.0 Flash, GPT-4o mini, and GPT-4.1 mini exceed GPT-4.1’s success rate, with fine-tuned GPT-4.1 mini and GPT-4o mini also exceeding its reliability. Additionally, fine-tuned models deliver faster response times, with Qwen3-8B (fine-tuned) responding in ~513ms compared to GPT-4.1’s ~1305ms.

Architecture:

Two specialized policies (informally, agents) working together:

Policy 1: Generates search queries given a claim and accumulated notes; Policy 2: Extracts relevant titles and creates notes for the next search

Training Approach:

We fine-tune two separate versions of each small model, one for each policy. Importantly, we only use successful conversation demonstrations from GPT-4.1, not policy-specific feedback.

Fine-tuning data:

671 successful GPT-4.1 conversations as demonstrations (536 training, 135 validation).

Evaluation data:

115 test tasks × 5 repetitions.

4. Agentic Tool Use (τ-bench)

τ-bench benchmarks multi-turn, agentic tool-use agents in both retail and airline customer service environments. We augment the τ-bench system instruction with the Anthropic think-tool prompt in all evaluations.

To explore the generalization performance of fine-tuning across domains (i.e. out of distribution), we fine-tune models using data from τ-bench’s retail domain and additionally evaluate in τ-bench’s airline domain.

Success: Agent completes the user’s request with correct database changes (measured by state comparison) and communicates all required information.

Key Finding: The τ-bench results reveal the practical limits and opportunities of fine-tuning. In the retail domain, fine-tuned models offer compelling cost-performance tradeoffs that could satisfy many production requirements. The airline results show that fine-tuned models generally maintain stable performance on related tasks, with some models showing modest improvements over their base counterparts.

Retail Domain (in-distribution)

Task: Handle multi-turn customer service conversations for order management including cancellations, modifications, returns, exchanges, and address updates while following domain-specific policies and authentication requirements.

Fine-tuning data:

410 successful GPT-4.1 conversations as demonstrations (328 training, 82 validation).

Evaluation data: 115 test tasks × 5 repetitions.

The persistent performance gap between the small models and the large model likely reflects the task’s inherent complexity, requiring multi-step reasoning, business rule adherence, and precise database state management. Our relatively limited training dataset of 332 demonstrations may be too small for such complex agentic tasks. Additionally, we used default fine-tuning settings for each provider, so additional hyperparameter tuning could yield further gains.

Airline Domain (out-of-distribution)

Task: Handle multi-turn customer service conversations for flight reservations including bookings, modifications, cancellations, and customer inquiries while following complex airline policies regarding baggage allowances, cabin restrictions, and payment rules.

Fine-tuning data:

Zero airline-specific data (models fine-tuned only on retail tasks)

Evaluation data:

50 test tasks × 5 repetitions

The airline domain presents an interesting pattern where some fine-tuned models (like Gemini 2.0 Flash and GPT-4.1 nano) show modest improvements over their zero-shot counterparts, while others (GPT-4.1 mini, GPT-4o mini, and Gemini 2.0 Flash-Lite) experience performance regression. This mixed behavior on out-of-distribution tasks represents a compelling area for future research into transfer learning dynamics and domain adaptation strategies.

Does Curation Even Matter?

To assess whether data curation impacts fine-tuning effectiveness, we compare models trained on curated datasets (containing only successful demonstrations) against those trained on uncurated datasets (containing all demonstrations).

Key Finding: The impact of data curation varies significantly by model family and task type. For data extraction and agentic RAG, curation consistently improves performance across all models. On agentic RAG, curation improves fine-tuned performance of GPT-4.1 mini achieves from 43.9% to 46.8% pass^1 and Gemini 2.0 Flash from 35.7% to 43.3% pass^1. Data extraction tasks show similar patterns, with both GPT and Gemini models benefiting from curated training data. However, agentic tool use reveals a more complex picture: while GPT models benefit from curation (especially for reliability), Gemini models actually perform better with uncurated data on this task. One explanation for these divergent results could be the difference in training data quality. GPT-4.1 has a 36% success rate on the agentic RAG training data, while it achieves 82% for agentic tool use. When training data contains a majority of failures, curation may provide clearer learning signals. However, when most demonstrations succeed, the value of naive filtering may diminish. Context-aware credit assignment strategies, like reinforcement fine-tuning, may be the key to unlocking the full potential of curation.

Fine-tuning offers compelling cost savings and potential performance improvements, but requires empirical validation for each use case. The technique is accessible and cost-effective enough to justify experimentation, and tools like 11.5KTensorZero make it straightforward to determine whether it is right for you.

Here are the key benefits fine-tuning can deliver:

1. Dramatic Cost Reduction with Strategic Flexibility

A 5-30x cost reduction can fundamentally transform LLM application economics. You can start with an expensive, high-performance model and seamlessly transition to a less expensive fine-tuned alternative without any infrastructure changes. You can serve more users, handle more queries, and experiment more freely. Our experiments demonstrate success across multiple model families (GPT-4o mini, Gemini, Qwen) and fine-tuning platforms, reducing vendor lock-in risks while providing negotiating leverage and ensuring continuity if any single provider changes pricing or availability.

2. Improved User Experience Through Faster Response Times

Beyond cost savings, fine-tuned small models consistently deliver faster response times, improving user experience and enabling real-time applications. Our measurements show 2-4x faster response times across all tasks, with some models like Qwen3-8B achieving sub-200ms responses on navigation tasks. Faster responses reduce user wait times, enable more interactive applications, and can improve conversion rates in customer-facing scenarios.

3. Cost Effective Alternative to Inference-Time Optimization

While techniques like best-of-n sampling, chain-of-thought prompting, or dynamic in-context learning can improve model performance, they multiply inference costs by requiring multiple model calls, longer contexts, or longer completions. Fine-tuning requires a one-time investment — using OpenAI GPT-4.1 mini, fine-tuning costs ranged from $2.30 (BabyAI) to $32 (τ-bench) — but then delivers performance improvements through single, fast inference calls. This upfront investment can pay for itself in hundreds of conversations or fewer.

4. Flexible Deployment Strategies

Fine-tuned models can enable sophisticated routing architectures:

Handle 90% of queries with fine-tuned small models at a fraction of the cost; Route complex edge cases to large models only when necessary; Implement fallback strategies that maintain service quality during peak loads; Use successful large model demonstrations for further fine-tuning

Practical Implementation Guide

Ideal scenarios for fine-tuning

Well-defined, repeatable tasks with clear success criteria; High-volume operations where cost savings compound significantly; Applications requiring consistent performance and response times; Teams seeking to reduce dependence on expensive frontier models

Key considerations and limitations

While fine-tuning offers compelling advantages, several factors warrant careful consideration. The approach requires initial data collection using expensive large models, but the upfront investment is recoverable through reduced inference expenses. Success depends on clearly defining task-specific success criteria, whether through programmatic metrics or human evaluation. Ongoing quality assurance becomes essential to detect performance drift over time, particularly as input distributions evolve. The benefits may not transfer perfectly to adjacent use cases, requiring robust evaluation and monitoring. Additionally, different model families exhibit varying fine-tuning characteristics, making provider selection an important strategic decision. While providers like OpenAI and Google Vertex AI automatically determine hyperparameters for most use cases, more niche or complex tasks may require manual tuning for optimal results.

Implementation roadmap with TensorZero

11.5KTensorZero provides an open-source stack for building industrial-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluation, and experimentation — everything you need to systematically fine-tune models for production use cases.

Set up an LLM gateway: Integrate the TensorZero Gateway to access every LLM provider through a unified API (<1ms p99 latency); Collect quality data: Configure feedback collection to capture 100-1000 successful conversations from expensive models, automatically stored in your database for easy fine-tuning export; Train and evaluate: Fine-tune multiple small models programmatically or via the UI. Optionally, run evaluations to identify the best performers.Example: Fine-tune Gemini 2.0 Flash-Lite on τ-bench RetailThe script below demonstrates how to fine-tune Gemini 2.0 Flash-Lite on τ-bench Retail using TensorZero’s programmatic API. Alternatively, you can fine-tune LLMs using the TensorZero UI; Deploy with confidence: Leverage the experimentation features to gradually route traffic from expensive to fine-tuned models, starting with a small allocation and scaling based on performance metrics

The Path Forward

Fine-tuning is reshaping the traditional performance-cost tradeoff in LLM applications. By leveraging the knowledge distilled in large models to train specialized small models, we can achieve the best of both worlds: enterprise-grade performance at startup-friendly costs.

As AI becomes the foundation of more software applications, the teams that master cost-effective deployment will build sustainable competitive advantages. The gap between prototype and production-ready AI applications is narrowing, but only for those who optimize intelligently.

Don’t just scale up. Scale smart.