TensorZero 实测 OpenAI RFT：成本是 SFT 的 100-700 倍，值不值？

Summary

We benchmarked OpenAI’s supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT) workflows on three tasks:Data Extraction: RFT improved performance with limited data, though SFT on a larger dataset achieved better results at a still much lower cost; Agentic Coding: RFT improved performance where SFT failed; Customer Service: SFT outperformed RFT. RFT actually degraded performance; RFT costs 100-700x more than SFT on equivalent datasets. SFT on much larger datasets often still costs only a small fraction of a tiny RFT job; We demonstrate how to implement systematic SFT and RFT workflows programmatically using 11.5KTensorZero

OpenAI shipped reinforcement fine-tuning (RFT) for o4-mini earlier this year, and at $100 per hour of training time, we had to ask: is this actually worth it?

The pitch sounds compelling: train your model to get better at your specific task using reinforcement learning. But with supervised fine-tuning (SFT) costing literally pennies in comparison, we wanted to stress test whether RFT delivers enough performance gains to justify the 100-700x cost premium.

So we ran the experiment. We took three real-world tasks — data extraction, agentic coding, and customer service (agentic tool use) — and compared RFT against both base o4-mini and supervised fine-tuned GPT-4.1 mini.

The results were… complicated. And expensive. Here’s what we found.

Evaluation

We picked three tasks that cover diverse production use cases for LLMs:

Data Extraction: CoNLL++ NER; Agentic Coding: terminal-bench; Customer Service: τ-bench retail

For each task, we ran the same experiment:

Collected training data from successful o4-mini completions; Collaborated with Claude Code to write LLM grader instructions; Ran RFT on o4-mini using the LLM grader; Ran SFT on GPT-4.1 mini on the same data for comparison; Measured what actually matters:Pass^1: What is the success rate?; Pass^k: Is it reliable? (all k attempts succeed; gets harder as k grows); Training cost: $100/hour + grading tokens ($1.10 per M input, $4.40 per M output, $0.275 per M cached); Inference cost per conversation

Code to reproduce our results is available in our LLMGym.

Results

1. Data Extraction (CoNLL++ NER)

Task: Extract named entities from the CoNLL++ dataset, identifying persons, locations, organizations, and miscellaneous entities.

Success Criteria: Correctly identify and classify all named entities (e.g. people, organizations) in unstructured text. Success is focused more on understanding the expected format and edge cases rather than actual named entity recognition.

Training Cost: RFT on 10 training examples cost $65.07 ($50 training, $15.07 grading). SFT on the same data cost $0.09 (723x difference). SFT on all available training examples cost $0.41 (159x difference). Estimated RFT cost for the full dataset: $134.35 (328x increase). [details in appendix]

Key Finding: RFT improved performance on data extraction with just 10 training examples. However, SFT on a larger dataset achieved better results at 159x lower optimization cost, 11x cheaper inference, and 3x faster response times.

Fine-tuning data:

10 training inferences using o4-mini (127 inferences in full training set)

2 validation inferences using o4-mini (31 inferences in full validation set)

500 test tasks with 5 repetitions

2. Agentic Coding (terminal-bench)

Task: Complete complex computational tasks in a terminal environment including compiling code repositories, training machine learning models, configuring servers, and performing data analysis through multi-step command sequences.

Success: Agent successfully executes all required terminal commands and produces the correct output as verified by automated test scripts in isolated Docker environments.

Training Cost: RFT on 2 training conversations (17 inferences) cost $168.57 ($131.67 training, $36.90 grading). SFT on the same data cost $0.70 (241x difference). SFT on all available conversations cost $3.29 (51x difference). Estimated RFT cost for the full dataset: $400.95 (122x increase). [details in appendix]

Key Finding: RFT significantly improved performance on agentic coding tasks with just 2 training conversations (17 inferences). SFT showed performance degradation relative to baseline on both equivalent and full training datasets.

Fine-tuning data:

2 training conversations using o4-mini comprising 17 inferences (16 conversations comprising 237 inferences in full training set)

1 validation conversation using o4-mini comprising 19 inferences (3 conversations comprising 48 inferences in full validation set)

39 test tasks with 5 repetitions

3. Customer Service (τ-bench retail)

Task: Handle multi-turn customer service conversations with tool use for order management including cancellations, modifications, returns, exchanges, and address updates while following domain-specific policies and authentication requirements.

Success: Agent completes the user’s request with correct database changes (measured by state comparison) and communicates all required information.

Training Cost: RFT on 2 training conversations (23 inferences) cost $201.85 ($151.67 training, $50.18 grading). SFT on the same data cost $1.80 (112x difference). SFT on all available conversations cost $29.27 (7x difference). Estimated RFT cost for the full dataset: $10,591.23 (362x increase). [details in appendix]

Key Finding: RFT showed performance degradation relative to baseline on customer service tasks with 2 training conversations (23 inferences), whereas SFT showed improvements both on the equivalent and full training dataset.

Fine-tuning data:

2 training conversations using o4-mini comprising 23 inferences (332 conversations comprising 3866 inferences in full training set)

1 validation conversation using o4-mini comprising 15 inferences (82 conversations comprising 921 inferences in full validation set)

115 test tasks with 5 repetitions

Discussion

The results paint a nuanced picture. RFT delivered performance gains for data extraction and agentic coding, but degraded customer service performance relative to baseline. Only in agentic coding did RFT clearly outperform SFT, but at 241x the cost.

It is interesting to see performance gains on agentic coding with a small amount of training data as presumably OpenAI optimized o4-mini specifically for this task. For all tasks, evaluating how downstream performance scales with training data size would provide useful information to LLM application engineers.

The freedom to design grader functions is an attractive feature of RFT. Evaluating the sensitivity of fine-tuned model performance to variations on model grader prompts or the addition of programmatic graders is an interesting future direction.

The nuanced results demand further analysis, but we are limited by two important factors: cost and content moderation.

Bottleneck: Cost

RFT costs present a real barrier for practitioners actively iterating on LLMs.

At hundreds or thousands of dollars per experiment, extensive hyperparameter search becomes prohibitively expensive. The grader configuration alone presents a high dimensional space of factors that could affect results, but systematic exploration would cost tens of thousands of dollars. API-based fine-tuning methods are attractive because users don’t need their own GPU resources, but the cost of RFT undermines this appeal.

Without the ability to iterate and experiment, it’s difficult to know if poor results come from the method itself or suboptimal configuration. For example, why do we see a performance drop on customer service? We’d like to answer such questions by enriching our analysis so that engineers can make informed decisions, but the cost would be prohibitive at scale. More efficient reinforcement learning techniques (like amortizing advantage estimation through learned value functions) could also make reinforcement learning a more viable optimization solution for LLM applications.

Bottleneck: Content Moderation

RFT appears to trigger much stricter content moderation than SFT or inference.

We had to try multiple random dataset splits to find subsets of data that wouldn’t be flagged, despite these being standard public benchmarks that work fine with SFT. Moreover, the datasets that were flagged did not appear to violate OpenAI’s usage policies in any meaningful way. For example, the data extraction dataset only contains text that is typically found in major newspapers.

The presence of examples that trigger policy violations makes it impossible for the full training sets to pass validation. So even if the considerable investment were made, we can’t run RFT using all available examples at present. We have shared example datasets with OpenAI for clarification.

Programmatic SFT and RFT using TensorZero

11.5KTensorZero provides an open-source stack for building industrial-grade LLM applications. It unifies an LLM gateway, observability, optimization, evaluation, and experimentation — everything you need to systematically fine-tune models for production use cases.

Set up an LLM gateway: Integrate the TensorZero Gateway to access every LLM provider through a unified API (<1ms p99 latency); Collect quality data: Configure feedback collection to curate successful conversations, automatically stored in your database. Easily export curated datasets for further analysis or optimization; Train and evaluate: Run SFT and RFT jobs with multiple hyperparameter configurations programmatically. (See our earlier blog post: Distillation Programmatic Data Curation: Smarter LLMs, 5-30x Cheaper Inference) Optionally, run evaluations to identify the best performers.Example: Reinforcement Fine-Tuning for o4-mini on terminal-benchThe script below demonstrates how to fine-tune o4-mini on terminal-bench using TensorZero’s programmatic API; Deploy with confidence: Leverage the experimentation features to gradually route traffic to your fine-tuned models, starting with a small allocation and scaling based on performance metrics

So, is reinforcement fine-tuning worth it?

For most use cases, not yet.

At 100-700x the cost of supervised fine-tuning, we only observed RFT to deliver clear wins on agentic coding tasks. Data extraction performed better for less with SFT, and customer service actually got worse with RFT.

The technology shows promise. It gives engineers flexibility in reward design through flexible grader configurations and the performance gains can be substantial when it works. But between the prohibitive costs, content moderation hurdles, and limited applicability, RFT is a difficult choice to justify. Until costs come down dramatically or best practices are established across task types, supervised fine-tuning remains a pragmatic alternative in most cases for LLM application improvement.

But, if you have a use case that benefits from reasoning (e.g. agentic coding) or sufficient budget, RFT may deliver the performance edge you need.

We estimate the cost of RFT on full training datasets by extrapolating from our small-scale experiments. Since OpenAI’s pricing is time-based ($100/hour) plus token-based grading costs, we needed to estimate both components.

Actual costs from our RFT experiments using minimal training data. Training cost is based on OpenAI's $100/hour rate. Grader costs are $1.10 per M input tokens, $4.40 per M output tokens, and $0.275 per M cached tokens.

Estimated costs for RFT on full datasets.

Estimation Approach

For training cost, we assume training time\text{time}time scales linearly with the number of gradient steps. We know the number of inferences n\mathrm{n}n and OpenAI determines epochs\text{epochs}epochs and batch-size\text{batch-size}batch-size prior to full dataset RFT failure:

We assume that step-time0\text{step-time}_0step-time0 is unchanged and estimate:

For grader cost, we assume each inference requires the same average token usage so that grader usage scales with the number of inferences n∗epochs\mathrm{n} * \text{epochs}n∗epochs:

Acknowledgements: Thanks to Viraj Mehta, Gabriel Bianconi, Alan Mishler, and Shuyang Li for helpful feedback.