Hugging Face Shares Industry Practices for Training Agents: Environments, Evaluation, and Rewards Are Harder Than the Model
Hugging Face released four talks, a panel discussion, and a Q&A, discussing practical approaches to training Agents in industry, with the core conclusion being that environment, evaluation, and reward design are harder than the model itself. Miles used 64×B300 fully asynchronous training on Terminal Bench 2, and after about 100 steps the reward rose from about 0.44 to 0.56; OpenEnv Arena, after filtering down layer by layer from about 90,000 findable RL environments, had only about 800 relatively clean and usable ones left. The talks also mentioned cheating problems such as Agents finding reference answers in the sandbox and modifying the scoring logic to fake a pass, as well as about 1% of cheating problems contaminating an entire round of training.
Why read this
AI score 60
Four industry-side talks collectively present the real difficulties and lessons learned in environment, evaluation, and reward design for Agent training.
Topics
View original postlast week we closed the Training Agents series IRL at the GitHub offices in SF 4 talks, a panel and audience Q&A. almost 2h on post-training agents with open models > Learnings from enterprise RL (@t2govind Microsoft) > Environments and benchmarks in industry (@VBarres86633 Mercor) > How to train your agents with Miles (@nblintao RadixArk) > Why we built OpenEnv Arena (@xdotli BenchFlow) full recording + slides https://t.co/cIkm20wEdV
View original
last week we closed the Training Agents series IRL at the GitHub offices in SF 4 talks, a panel and audience Q&A. almost 2h on post-training agents with open models > Learnings from enterprise RL (@t2govind Microsoft) > Environments and benchmarks in industry (@VBarres86633 Mercor) > How to train your agents with Miles (@nblintao RadixArk) > Why we built OpenEnv Arena (@xdotli BenchFlow) full recording + slides https://t.co/cIkm20wEdV
Body · AI translation
How Industry Trains Agents: Environments, Evaluation, and Rewards Are Harder Than the Model Itself
HuggingFace's officially released four talks + roundtable + audience Q&A are an in-person continuation of HuggingFace's free course Training Agents (trajectory SFT → distillation → GRPO → building RL environments), with a focus on industry-side implementation.
Slides collection: https://t.co/mvOl6omthQ Course series: https://t.co/jDCMFTlXGn YouTube video: https://t.co/NBRXbikmbL
Gobind (Microsoft / GitHub & Foundry) -- Enterprise RL Traditional large enterprises (industrial, manufacturing, energy) buy business metrics: latency, accuracy, and controllable risk in post-training; they care about the entire chain (tools, data, network, harness, wall-clock), not a single model. Real agents often hang off dozens to hundreds of tools, and many problems must be solved as distributed systems. In practice: first quickly customize an off-the-shelf harness, build a simulation from production trajectories, then deliberately perturb it (stale data, timeouts), paired with functional-correctness and safety validators. Post-training mixes SFT / distillation / DPO / RL; the preference is for modular decision models rather than using a frontier large model for everything. The outlook includes: more hand-crafted rewards and environments in the future; discoverable "gym" resources appearing on GitHub; and physical AI, alignment, and regulation catching up.
Victor (Mercor, formerly Sierra) -- Metrology of Environments and Benchmarks Core distinction: training environments provide reward (how learnable, how hard to cheat), while evaluation environments provide measurement (how stable, whether they can predict production performance). Asking "does it look like the real world" is too coarse; you must ask what it is like relative to. A wind tunnel need not look like the sky, but it must preserve the relationships that truly affect outcomes. What is evaluated should be the Agent = model + harness + operations, and you cannot report only bare-model scores. Reliability caps validity; beyond average scores, you also need pass items, red teaming, and predictable deployment. Benchmarks should be maintained as long-term products, not as one-off question banks.
Tao Lin (RadixArk / Miles, SGLang) -- Asynchronous Agent RL Miles treats multi-step environment trajectories as training samples; it interfaces with OpenEnv's reset / step. Case: a large MoE on Terminal Bench 2, 64×B300 (half for inference and half for training), fully asynchronous, with a new sandbox each episode; after about 100 steps the reward went from about 0.44 to 0.56. Practical pitfalls were fed back upstream: agents found reference answers inside the sandbox and modified the scoring logic to fake a pass; others did it right but were judged wrong because of unstable infrastructure/scorers. Conclusion: scaling is not just throughput, but also isolation and reliable rewards.
Xiangyi Li (BenchFlow) -- OpenEnv Arena OpenEnv Arena is used to screen trainable environments. Many "environments" in the open are unusable: of about 90,000 findable RL environments, after layer-by-layer filtering about 800 relatively clean ones remain; most competition submissions cannot train or are worse than the baseline. About 1% of cheating problems will contaminate the entire subsequent round of training. When scaling to about 320B-level models, if there are not enough steps they cannot climb the target leaderboard; old academic tasks that work well on small models sometimes only teach large models to "use fewer tools and spend fewer tokens." Safety/cyber tasks are noisy, and some frontier models refuse to execute even with authorized phrasing; open models that do not refuse indiscriminately have unique value here.
Roundtable and Q&A The obstacles are often described as: a lack of screened open environments; enterprise "model fatigue," outcome pressure, and budgets being consumed by coding agents; and after the era of large-model APIs, the data and MLOps muscle needed for self-hosting has atrophied. Gobind believes that after coding, operations/decision-making modules have more product space. The closing line is very down-to-earth: you cannot expect that "change the database and the model becomes smarter."