蚂蚁 inclusionAI 开源单节点 RL 后训练工具包 AReno

inclusionAI; AReno

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork128 (128); Star323 (323)You must be signed in to star a repository

An easy-to-use, fast toolkit to scale up RL post-training on a single node.

Repository files navigation

README; Contributing; Apache-2.0 license

👋 Hi, everyone! AReno is a fast, effortless, and self-contained toolkit that scales RL post-training up locally, initiated by the inclusionAI ASystem team and maintained by the AReno community.

AReno: ASystem Reinforcement Learning Nano

AReno is a local LLM post-training toolkit for RL, SFT/DPO-style training, serving, and agentic RL. It was originally developed by engineers from the ASystem Team at Ant Group.

Built on a self-contained, full-stack design, AReno is optimized to extract maximum performance from a single node, making it well-suited for fast, local post-training with no external training or inference backend in the loop.

AReno's mission is to make LLM RL accessible for a broad community of researchers and developers — so you can go from a base checkpoint to a trained, served model on a single node, without standing up a cluster or wiring together a training framework, an inference server, and a kernel library.

Small but complete, like its name — nano in footprint, full-stack in capability. We hope AReno makes scaling up your ideas locally both fast and delightful. Enjoy!

Highlights

✨ Plug-and-play: various post-training methods are easily accessible via the --algo flag or the same Trainer class from Python, no cluster or launcher to set up; 🪶 Lightweight: one self-contained train/serve stack with CUDA, Ascend NPU, and Apple Silicon MLX backends; 🧰 Agentic RL ready: run an agent function against AReno's local OpenAI-compatible proxy, return explicit trajectories, and train from tokens, logprobs, rewards, and loss masks derived by the trainer; 🎞️ Multimodal: use image, audio, and video content with compatible model processors through the same OpenAI-style message format in serving and agentic training; 🧩 Native LoRA: train TP-aware adapters for Qwen3, Qwen3-MoE, and Bailing-MoE V3, save standard PEFT artifacts, and reload them for training or serving; 🗜️ QLoRA: train Qwen3 dense and Bailing-MoE V3 adapters on CUDA with NF4 weights, double quantization, and paged optimizers; 🧩 Extensible: easily register new algorithms, model adapters, reward functions, and hardware backends without changing the core

Installation

From source

AReno supports Linux (x86_64 or aarch64) with an NVIDIA GPU and CUDA-enabled PyTorch 2.6 or newer, plus Apple Silicon macOS through MLX. Windows users can use WSL2 for the CUDA path. Ascend NPU integration uses CANN and torch_npu. The CLI selects NPU on Linux when torch_npu is installed, otherwise CUDA, and MLX on native arm64 macOS.

CUDA/WSL2 installation:

Apple Silicon installation:

Platform dependency markers install the MLX stack on Apple Silicon without pulling in Linux-only Torch/CUDA packages. See the Apple Silicon and MLX guide for supported algorithms, checkpoints, serving, multimodal behavior, and memory controls.

Docker

Use our official AReno image inclusionai/areno container. Run:

The Dashboard launches and monitors local or Modal training and serving jobs, with agent chat, editable plans, GPU/cost estimates, and live metrics and logs. See the Modal setup guide.

Quick Start

AReno provides core API abstractions for LLM post-training, with AReno SDK, RL loop is a short cycle of Trainer calls. Each step below maps a concept to the SDK call that performs it:

Create the trainer — construct a Trainer on the AReno backend and init() it to load the tokenizer and start workers; Rollout — inside rollout_session(...), rollout_batch(...) generates on-policy completions for each prompt; Score — reward each completion and turn rewards into advantages (your reward function, not AReno's); Train — pack the rollout into TrainSequence objects and call train(batch, loss_fn) to run one optimizer step; Repeat — new weights produce new rollouts; loop until done, then close()

See the documentation for the full Trainer API.

Command Line Interface (CLI)

Operations agent

areno agent is a local operations assistant for AReno train and serve tasks. It uses an OpenAI-compatible model to inspect the current checkout, read command help, run diagnostics, and produce or execute AReno commands for the current machine. Configure the agent once with your endpoint, model, and API key:

Then describe the job in natural language:

From a source checkout, you can run the same agent without installing AReno:

The agent can ask follow-up questions through the terminal when a required parameter is missing, and it streams command output while it works.

Diagnostics

Check whether the current machine is ready to run AReno:

areno check prints OK, WARN, and FAIL statuses with concrete next steps for common setup issues such as missing CUDA, CPU-only PyTorch, missing CUDA_HOME, unavailable nvcc, missing optional runtime dependencies, or a missing areno_accel extension.

For issue reports, collect a descriptive environment report:

The report includes AReno, Python, platform, PyTorch/CUDA, GPU, nvcc, dependency import status, and relevant environment variables.

Training

Tiny training smoke test

Use this command when you only want to check that a machine can run one small official training task end to end:

This is a smoke/sanity task for the CLI, dataset loader, reward function, rollout, and training-step wiring. It is not a quality benchmark. It requires either a CUDA-capable NVIDIA GPU on Linux or Apple Silicon with MLX. A successful run should reach rollout logs and a train_stats=... line without raising an exception. On Apple Silicon, use a checkpoint implemented by mlx-lm and start with --tp-size 1 --world-size 1 --mini-bs 1.

Run GSPO on a GSM8K-style dataset with a reward function:

--ckpt and --dataset-path accept either local paths or remote repo IDs. By default, remote refs use ModelScope. Use --model-hub hf to pull non-local model and dataset refs from Hugging Face. Switch algorithms by changing --algo (e.g. --algo grpo, --algo sft).

For models whose tokenizer chat template supports a thinking-mode switch, add --disable-thinking to pass enable_thinking=False during training prompt rendering. Tokenizers that do not support this argument automatically use their normal chat-template path.

For rollout-based algorithms, add --tune-params when you want AReno to probe rollout and train memory before the real run and fill conservative values for --max-running-prompts, --batch-size, and --mini-bs:

The tuner uses dummy-loaded model weights and synthetic token rows, respects the configured sequence limits and tensor-parallel size, and enables --drop-rollout-state for the tuned run. See docs/cli/training.rst for the full tuning rules.

For Agentic RL, add --agent-fn to supply an agent function. The agent calls the local OpenAI-compatible endpoint, including tools and tool_choice when needed, and returns explicit AgentTrajectoryTurn objects. AReno converts those turns into trainable assistant outputs and masks tool results by default:

For a more realistic multi-turn software-engineering loop, see examples/agentic/coding. It uses a 100-record SWE-bench-style local dataset that ranges from easy to hard with most records marked hard, constrained Codex-style tools (inspect_tree, read_file, rg, apply_patch, run_command, submit), and an explicit trajectory-returning run_agent.py. The records are self-contained local Python tasks and do not require sandbox services, package installation, or network access.

DuelGrid is a larger agentic demo with a browser game UI and multi-action turns. Before GSPO/RLVR post-training, Gemma-E2B-it performs poorly in this game and often moves back and forth without progress. After training, it learns to collect pickups, chase the user, attack when in range, and avoid trap tiles.

See examples/agentic/duelgrid for the rule engine, fixed-path dataset loader, reward function, OpenAI-compatible agent, and browser UI.

For the full list of training options, run areno train --help.

Native LoRA

Enable CUDA-native LoRA by passing a rank. AReno freezes the base model and trains the selected projection adapters through the same rollout and training engine, including agentic RL:

Saved checkpoints contain standard PEFT adapter_config.json and adapter_model.safetensors files. Resume training or serve an adapter by supplying the frozen base checkpoint together with --lora-adapter-path. See the native LoRA guide for supported models and targets, agentic training, save/reload, and serving examples.

QLoRA

For CUDA Qwen3 dense and Bailing-MoE V3, add --qlora --adam-4bit to the training command above to use NF4 weights, double quantization, and paged Adam4bit. --qlora enables rank 8 / alpha 16 LoRA by default; the existing --lora-* options override these settings. Saved adapters automatically restore their quantization mode through --lora-adapter-path.

Serving

Serve a trained checkpoint as an OpenAI-compatible endpoint with continuous batching:

To serve a saved native LoRA adapter without merging it into the base model:

The command selects CUDA, NPU or MLX from the host environment. MLX serving is single-process (--tp-size 1 --world-size 1) and uses the same long-lived continuous-batch request scheduler and HTTP API.

Point any OpenAI client at http://localhost:8000/v1/chat/completions to start generating. For the full list of serving options, run areno serve --help.

Use --disable-thinking with areno serve to pass enable_thinking=False to compatible tokenizer chat templates for request rendering.

Development

If you want to contribute to AReno or customize it for your own needs, read the contribution guide and make a development install:

New algorithms, model adapters, kernels, reward functions, and hardware backends all have first-class extension points, so most contributions land without forking the core.

Citation and Acknowledgement

If you find the project helpful, please cite:

AReno's API design is inspired by Tinker from ThinkingMachines. We would like to express our gratitude for their pioneering work.

License

This repository's source code is available under the Apache 2.0 License.

An easy-to-use, fast toolkit to scale up RL post-training on a single node.