Prime Intellect 在 PRIME-RL 中推出多智能体训练支持

Today, the Prime Intellect RL stack expands from training individual agents to multi-agent systems. You can now program arbitrary interactions between agents, choose which roles learn, and assign credit across the complete interaction.

Two recently introduced abstractions set the stage for this development: verifiers v1 introduced the primitives for running a single agent on a programmable task in a swappable harness and runtime, and prime-rl gained an algorithms layer to make the mapping from rollouts to training signal programmable.

Today, we are bringing both pieces together by introducing abstractions that enable multi-agent training and evaluations first-class. Some interesting examples that can now be expressed:

Agentic Judging — solver traces are graded by a judge; Self-Play — a model playing against itself; User-Sim — a user agent interacting with an assistant

This post introduces the two main abstractions, the Agent and Env, explains the design choices behind them and tours several interaction patterns that we believe open promising new directions for agentic RL.

Agent

verifiers v1 natively supported all the primitives that define a single agent rollout:

Taskset supplies a set of Task objects; Harness defines the program that drives the model; Runtime defines where the program executes

An Agent just becomes the natural home for each of these. The core signature is Agent.run(task: Task) -> Trace — when given a Task the agent produces a Trace, the artifact produced by the rollout.

Behind the scenes, the agent manages the lifecycle of all the hidden machinery which enables you to run any agent in any runtime on any task. By making the abstraction first-class, you can easily script over it. Multi-agent environments are then just one instantiation of a program over agents.

Env

The Env now becomes the home for multi-agent training and evaluation. Its core signature is Env.run(task: Task, agents: Agents) -> None — it is passed an initial task and a list of pre-initialized agents and then programs the full multi-agent control flow; every finished agent run automatically joins the resulting Episode.

Anything that could be expressed in verifiers v1 collapses to a one-line run method in SingleAgentEnv.

We will go through four already implemented multi-agent envs; we think they are both useful in their own right and instructive demonstrations of what these abstractions make possible.

Agentic Judging

Our post on scaling agentic RL ended with a problem that deterministic grading cannot solve: fixed graders are often too narrow. For example, in software engineering, tests may assert a particular implementation detail, making a valid solution incorrectly receive zero reward.

A naive LLM judge would not solve this problem — bounded by a single call, it would be unreasonable to expect an accurate verdict. An agentic judge, on the other hand, is capable of exploring the codebase, looking at the failing tests and overruling the deterministic tests.

The AgenticJudgeEnv defines the sequential interaction between a solver and judge agent.

Illustrative example. Check out the full implementation here.

Self-Play

A central bottleneck in LLM RL is task scarcity. Useful learning signal requires tasks that are close to the agent's current capabilities. Self-play offers an alternative to static tasksets by allowing the model to help generate its own curriculum. As the model improves, it can create harder tasks, be a stronger opponent, or find new failure cases for itself. We believe that self-play will become an important technique to push the frontier.

In this section, we introduce two variants of self-play — Proposer-Solver and Kuhn-Poker.

Proposer-Solver

In ProposerSolverEnv the proposer receives a seed topic and constructs a new task, which is then solved by a group of solvers. Each solver is rewarded for answering correctly while the proposer is rewarded for calibration to the solver population. If every solver succeeds, the task was too easy; if none succeed, it may be too difficult or invalid. The environment's learnability peaks at a 50% solve rate, encouraging the proposer to generate tasks that yield the maximum training signal for RL (inspired by Zhao et al. (2025), Absolute Zero).

The interaction also requires role-aware credit assignment. Solver attempts should be compared with attempts on the same proposed problem, while proposer traces should be compared with other proposals generated from the same seed task. Classic GRPO cannot represent this hierarchy, so we implemented Hierarchical GRPO to preserve these comparison sets without mixing roles or problem difficulties.

Illustrative example. Check out the full implementation here.

Kuhn-Poker

In KuhnPokerEnv two models play Poker. The environment maintains the private cards, public game state, and legal actions. As the policy improves, it also becomes a stronger opponent, producing a moving curriculum without a separate opponent service.

Because different roles can have structurally different reward distributions, our training system supports Role-Conditioned Advantage Estimation (RAE) for this setting. Rather than measuring every agent against one shared baseline, RAE measures each role relative to its own reward history.

User-Sim

The UserSimEnv models users as agents, and the episode is a turn-by-turn conversation between the user and assistant agents.

Both Traces remain visible in the Episode. The simulated user is frozen by default, while the assistant's Trace is scored against the original Task and used for training.

This makes user simulation another composition of the same primitives rather than a separate evaluation path. Different user populations, personae, hidden goals, and interaction policies can be plugged into the same interface, while the assistant can be configured differently.

Illustrative example. Check out the full implementation here.

Agents beyond RL

We expect the agent abstraction to be useful beyond training and evaluation. We are already building powerful synthetic data generation and curation pipelines. Each agent's trace is unified and auditable data artifacts.

To illustrate, here is Laguna-S2.1 in the pool harness finding the latest released verifiers version.

Multi-agent support releases today in verifiers 0.3.0 and prime-rl 0.8.0 — we are excited about empowering researchers to express multi-agent RL and push the frontier for open-source AGI.

Citation

Rewriting Prime Agent in Rust

On the Nature of the Swarm