Hugging Face 把 Claude Code、Codex 等编码工具变成 RL 训练环境
We turned Claude Code, Codex, Hermes, Pi, @opencode and other coding harnesses into RL environments. No changes to the harnesses, no changes to the training cod
Hugging Face 的 Clément Delangue 宣布把 Claude Code、Codex、Hermes、Pi、opencode 等编码工具变成 RL 环境，不改动工具本身也不改动训练代码，全部开源。做法是用一个 capture proxy 伪装成模型 API，兼容 OpenAI Chat Completions、OpenAI Responses、Anthropic Messages、Gemini 四种格式，转发到 vLLM 并记录 token ID 与 logprobs，再交给 TRL 训练，目前已有 10 个工具接入且均未修改。在 LFM2.5-2.6B 上，单工具训练后 OpenCode 从 34% 提升到 58%，四工具同时训练后四者均提升（42% 到 54%），而用 Qwen3.8-27B 的 3,189 条 rollout 做 SFT 只停在 47.5%。
正文
当前显示原文，中文全文译文尚未就绪。
We turned Claude Code, Codex, Hermes, Pi, @opencode and other coding harnesses into RL environments. No changes to the harnesses, no changes to the training code. Any open model, any task set, fully open source my friends!
Same model, same weights: 62% under Mini-SWE-Agent, 33% under Claude Code. But training inside a real harness normally means reimplementing it as an environment, so most models get trained in a scaffold nobody actually ships.
The fix is a proxy, not a rewrite. The harness thinks it's talking to a model API. It's actually talking to a capture proxy that speaks the 4 formats coding agents use (OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, Gemini), forwards to @vllm_project, records the exact token IDs and logprobs vLLM sampled, and hands TRL sequences it can train on. The harness becomes the environment. 10 harnesses run through it today, none modified.
And because you control the reward, you can shape behavior the harness never asked for. We added a small bonus for solving a task in fewer tool calls: on tasks it already solved, the model now uses 31% fewer calls, in every harness, and about half under Codex.
Tested on LFM2.5-2.6B from @liquidai: → Train in one harness: better mostly in that harness (OpenCode 34% → 58%). → Train in 4 at once: better in all 4 (42% → 54%). → SFT on 3,189 rollouts from Qwen3.8-27B instead: plateaus at 47.5%, below both RL runs.
Everything is open and reproducible: the capture proxy in OpenEnv, the trainer in TRL, the tasks, the SFT data, the training code and all 7 trained models. Bigger models and bigger runs next.